Visitors

Unable to create Tech Support file for UCS Manager – workspace partition is full.

Creation of ‘ucsm’ Tech Support file (Technical support data for the entire UCSM instance) for UCS Manager has failed with the following error message:

‘Failed to create the Technical Support file.
Please make sure that there is enought space on bootflash.
For Example:- You can delete unused firmware images to free up bootflash space’

Failed to create the Tech Support file

I checked the local storage on Fabric Interconnect and realised that there is plenty of space on the bootflash partition but the workspace partition is full.

Cisco UCS FI workspace partition full

Apparently, it is a bug in UCSM firmware v. 2.1(1f). You need to log a call with Cisco Support (or VCE Support if the UCS domain is part of Vblock) and obtain a debug plugin ‘ucs-dplug.5.0.3.N2.2.11f.gbin‘.
Follow this procedure to free up workspace partition and install the fix:

  1. SSH to the Fabric Interconnect UCS Manager cluster IP address and login as admin;
  2. Connect to primary Fabric Interconnect and confirm the workspace partition is full:
     UCS01-A# scope fabric-interconnect a
     UCS01-A /fabric-interconnect # show storage
    Storage on local flash drive of fabric interconnect:
     Partition        Size (MBytes)    Used Percentage
     ---------------- ---------------- ---------------
     bootflash        8785             35
     opt              1916             5
     workspace        277              Full
  3. Connect to local management on primary Fabric Interconnect:
     UCS01-A /fabric-interconnect # connect local-mgmt
     Cisco Nexus Operating System (NX-OS) Software
     {skipped}
  4. Review of the Fabric Interconnect (FI) workspace partition reveals it is completely full:
     UCS01-A(local-mgmt)# cd workspace:/
     UCS01-A(local-mgmt)# ls
    1        16 Aug 13 20:13:51 2011 cores
     2      1024 Oct 17 14:42:50 2011 debug_plugin/
     1        31 Aug 13 20:13:51 2011 diagnostics
     1 272245758 Dec 20 16:58:00 2013 fwm_trace_log
     2      1024 Aug 13 20:11:50 2011 lost+found/
     2      1024 Dec 18 20:34:52 2013 techsupport/
    Usage for workspace://
     290835456 bytes total
     281743360 bytes used
     0 bytes free

    fwm_trace_log fills up the whole partition!

  5. Delete fwm_trace_log file from primary Fabric Interconnect (FI):
     UCS01-A(local-mgmt)# rm workspace:/fwm_trace_log
     UCS01-A(local-mgmt)# ls
    1   16 Aug 13 20:13:51 2011 cores
     2 1024 Oct 17 14:42:50 2011 debug_plugin/
     1   31 Aug 13 20:13:51 2011 diagnostics
     2 1024 Aug 13 20:11:50 2011 lost+found/
     2 1024 Dec 18 20:34:52 2013 techsupport/
    Usage for workspace://
     290835456 bytes total
     8427520 bytes used
     267392000 bytes free
  6. Connect to the local management on subordinate Fabric Interconnect (FI) and delete fwm_trace_log file from there as well:
    UCS01-A(local-mgmt)# connect local-mgmt B
    Cisco Nexus Operating System (NX-OS) Software
    {skipped}
    UCS01-B(local-mgmt)# ls
    1        16 Aug 13 20:09:11 2011 cores
    2      1024 Oct 17 14:42:50 2011 debug_plugin/
    1        31 Aug 13 20:09:11 2011 diagnostics
    1 272521089 Jan 06 17:17:42 2014 fwm_trace_log
    2      1024 Aug 13 20:06:54 2011 lost+found/
    2      1024 Aug 13 20:08:12 2011 techsupport/
    Usage for workspace://
    291089408 bytes total
    282019840 bytes used
    0 bytes free
    UCS01-B(local-mgmt)# rm fwm_trace_log
  7. Upload debug plugin ‘ucs-dplug.5.0.3.N2.2.11f.gbin‘ (2.2MB) to FI using UCS Manager (Equipment -> Equipment -> Firmware management -> Download Firmware)
  8. Connect to local management, navigate to the debug plugin directory:
    UCS01-B /fabric-interconnect # connect local-mgmt A
    UCS01-A(local-mgmt)# cd debug_plugin/
  9. Load the debug plugin:
    UCS01-A(local-mgmt)# load-debug-plugin ucs-dplug.5.0.3.N2.2.11f.gbin
    ###############################################################
    Warning: debug-plugin is for engineering internal use only!
    For security reason, plugin image has been deleted.
    ###############################################################
    Successfully loaded debug-plugin!!!
  10. Create link for fwm_trace_log:
    Linux(debug)# ln -s /volatile/fwm_trace_log /workspace/fwm_trace
    Linux(debug)# exit
    exit
  11. Repeat the procedure on the peer Fabric Interconnect:
    UCS01-A(local-mgmt)# connect local-mgmt B
    {skipped}
    UCS0101-B(local-mgmt)# cd debug_plugin/
    UCS0101-B(local-mgmt)# load-debug-plugin ucs-dplug.5.0.3.N2.2.11f.gbin
    ###############################################################
    Warning: debug-plugin is for engineering internal use only!
    For security reason, plugin image has been deleted.
    ###############################################################
    Successfully loaded debug-plugin!!!
    Linux(debug)# ln -s /volatile/fwm_trace_log /workspace/fwm_trace
    Linux(debug)# exit
    exit

Hope this will help.

2 comments to Unable to create Tech Support file for UCS Manager – workspace partition is full.

  • Ziggy

    Hi Mark,

    Great post as always. It’s worth to mention 2 things:
    – debug plugin can be run by VCE/Cisco only
    – this is a bug CSCuh88231, so whoever will hit this, can mention bug id to VCE/Cisco support – it might speed up a call

    regards
    Ziggy

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>