Unable to create Tech Support file for UCS Manager – workspace partition is full.

Creation of ‘ucsm’ Tech Support file (Technical support data for the entire UCSM instance) for UCS Manager has failed with the following error message:

‘Failed to create the Technical Support file.
Please make sure that there is enought space on bootflash.
For Example:- You can delete unused firmware images to free up bootflash space’

I checked the local storage on Fabric Interconnect and realised that there is plenty of space on the bootflash partition but the workspace partition is full.

Apparently, it is a bug in UCSM firmware v. 2.1(1f). You need to log a call with Cisco Support (or VCE Support if the UCS domain is part of Vblock) and obtain a debug plugin ‘ucs-dplug.5.0.3.N2.2.11f.gbin‘.
Follow this procedure to free up workspace partition and install the fix:

1. SSH to the Fabric Interconnect UCS Manager cluster IP address and login as admin;
2. Connect to primary Fabric Interconnect and confirm the workspace partition is full:

    UCS01-A# scope fabric-interconnect a
    UCS01-A /fabric-interconnect # show storage
   Storage on local flash drive of fabric interconnect:
    Partition        Size (MBytes)    Used Percentage
    ---------------- ---------------- ---------------
    bootflash        8785             35
    opt              1916             5
    workspace        277              Full

3. Connect to local management on primary Fabric Interconnect:

    UCS01-A /fabric-interconnect # connect local-mgmt
    Cisco Nexus Operating System (NX-OS) Software
    {skipped}

4. Review of the Fabric Interconnect (FI) workspace partition reveals it is completely full:

    UCS01-A(local-mgmt)# cd workspace:/
    UCS01-A(local-mgmt)# ls
   1        16 Aug 13 20:13:51 2011 cores
    2      1024 Oct 17 14:42:50 2011 debug_plugin/
    1        31 Aug 13 20:13:51 2011 diagnostics
    1 272245758 Dec 20 16:58:00 2013 fwm_trace_log
    2      1024 Aug 13 20:11:50 2011 lost+found/
    2      1024 Dec 18 20:34:52 2013 techsupport/
   Usage for workspace://
    290835456 bytes total
    281743360 bytes used
    0 bytes free

   fwm_trace_log fills up the whole partition!

5. Delete fwm_trace_log file from primary Fabric Interconnect (FI):

    UCS01-A(local-mgmt)# rm workspace:/fwm_trace_log
    UCS01-A(local-mgmt)# ls
   1   16 Aug 13 20:13:51 2011 cores
    2 1024 Oct 17 14:42:50 2011 debug_plugin/
    1   31 Aug 13 20:13:51 2011 diagnostics
    2 1024 Aug 13 20:11:50 2011 lost+found/
    2 1024 Dec 18 20:34:52 2013 techsupport/
   Usage for workspace://
    290835456 bytes total
    8427520 bytes used
    267392000 bytes free

6. Connect to the local management on subordinate Fabric Interconnect (FI) and delete fwm_trace_log file from there as well:

   UCS01-A(local-mgmt)# connect local-mgmt B
   Cisco Nexus Operating System (NX-OS) Software
   {skipped}
   UCS01-B(local-mgmt)# ls
   1        16 Aug 13 20:09:11 2011 cores
   2      1024 Oct 17 14:42:50 2011 debug_plugin/
   1        31 Aug 13 20:09:11 2011 diagnostics
   1 272521089 Jan 06 17:17:42 2014 fwm_trace_log
   2      1024 Aug 13 20:06:54 2011 lost+found/
   2      1024 Aug 13 20:08:12 2011 techsupport/
   Usage for workspace://
   291089408 bytes total
   282019840 bytes used
   0 bytes free
   UCS01-B(local-mgmt)# rm fwm_trace_log

7. Upload debug plugin ‘ucs-dplug.5.0.3.N2.2.11f.gbin‘ (2.2MB) to FI using UCS Manager (Equipment -> Equipment -> Firmware management -> Download Firmware)
8. Connect to local management, navigate to the debug plugin directory:

   UCS01-B /fabric-interconnect # connect local-mgmt A
   UCS01-A(local-mgmt)# cd debug_plugin/

9. Load the debug plugin:

   UCS01-A(local-mgmt)# load-debug-plugin ucs-dplug.5.0.3.N2.2.11f.gbin
   ###############################################################
   Warning: debug-plugin is for engineering internal use only!
   For security reason, plugin image has been deleted.
   ###############################################################
   Successfully loaded debug-plugin!!!

10. Create link for fwm_trace_log:

    Linux(debug)# ln -s /volatile/fwm_trace_log /workspace/fwm_trace
    Linux(debug)# exit
    exit

11. Repeat the procedure on the peer Fabric Interconnect:

    UCS01-A(local-mgmt)# connect local-mgmt B
    {skipped}
    UCS0101-B(local-mgmt)# cd debug_plugin/
    UCS0101-B(local-mgmt)# load-debug-plugin ucs-dplug.5.0.3.N2.2.11f.gbin
    ###############################################################
    Warning: debug-plugin is for engineering internal use only!
    For security reason, plugin image has been deleted.
    ###############################################################
    Successfully loaded debug-plugin!!!
    Linux(debug)# ln -s /volatile/fwm_trace_log /workspace/fwm_trace
    Linux(debug)# exit
    exit

Hope this will help.