Visitors

EMC VNXe upgrade failed: Extracting the image on peer SP

Array: EMC VNXe 3100
Current version – 2.4.0.20932
Candidate version – 2.4.1.21171

EMC VNXe firmware upgrade failed at 15% with the following error message:

VNXe upgrade failed - Extracting the image on peer SP

“Software Update failed:
Extracting the image on peer SP
The following error(s) caused the software upgrade to fail:

  • The upgrade has failed. Try rebooting the Storage Processor (SPA) and then install the candidate version a second time. If this does not solve the problem, record the error codes and contact the VNXe Online Portal for all support options. [Error Code: No_Response_From_spa]”

I tried all of the following without any luck:

  • Restarted Upgrade :);
  • Rebooted SPA and SPB, re-upload the Candidate Firmware version and run the upgrade again;
  • Restarted Management Software;
  • Reimaged SPA with new image through GUI and CLI.

I also noticed SPA took a lot longer to come back online after restart and if I restart SPB and therefore fail the services over to SPA, Unisphere was not responding and only when SPB came back online, it started to function correctly. In the end, EMC accepted it may be a hardware fault and replaced SPA.

Unfortunately, even after we replaced SPA, VNXe upgrade did not finish successfully but this time it failed at 41% which was a good indication we were on the right track.

Before you start upgrading VNXe, make sure you check the available space on /dev/sda2 file system:

  1. SSH to VNXe Management IP address and login as ‘service‘. The password may be different to the ‘admin‘ account;
  2. Run ‘df -h‘ to check the available space:
    service@(none) spa:~> df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2             5.0G  4.3G  401M  92% / =======> 8% of free space WILL NOT be enough!
    tmpfs                 725M  2.8M  722M   1% /dev/shm
    udev                  469M  900K  468M   1% /dev
    /dev/sda4             4.1G  538M  3.4G  14% /cores
    /dev/tmpfs             32M  9.6M   23M  30% /tmp
    /dev/mirrora4          17G  2.9G   13G  19% /EMC/backend/service
    /dev/c4nasdba2       1019M   30M  938M   4% /EMC/backend/CEM
    /dev/c4nasdba1       1015M   41M  924M   5% /nbsnas
    /dev/c4loga2          653M   22M  599M   4% /EMC/backend/perf_stats
    /dev/c4loga1          3.4G  665M  2.6G  21% /EMC/backend/log_shared
  3. Verify VNXe storage integrity – ‘svc_storage_integritycheck‘:
    service@(none) spa:~> svc_storage_integritycheck
    Storage integrity verified successfully. All storage elements appear to be healthy.
  4. Delete unused files on both SP’s – ‘svc_clean_ssd -c‘:
    service@(none) spa:~> svc_clean_ssd -c
    INFO: Peer SP found - attempting to run on both SPs.
    Now searching for files that could be deleted on SPA ...Done!
    
    [Fri Dec  6 15:46:29 UTC 2013] Filesystem Usage Report (SPA) :
    =============================================
    Local Root filesystem mount point         : /
    Total Size of Drive                       : 5040MB
    Current Total Space Used                  : 4351MB, 86.32% usage
    A total of 1130MB can be saved on this system by removing files.
    Total Space Used (after / cleanup) : 3221MB, 63.90% usage
    {skipped}
    Cleared 1130MB of space off of SPA root filesystem!
    {skipped}
    This script will clear up some space on the root filesystem. It can also
    atempt to find large files on the filesystem and generate a report so the
    user can analyze the findings.
    {skipped}
  5. Run ‘df -h‘ to check available space:
    service@(none) spa:~> df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2             5.0G  3.2G  1.6G  68% / ======> 1.6 GB or 32% is free!
    tmpfs                 725M  2.8M  722M   1% /dev/shm
    udev                  469M  900K  468M   1% /dev
    /dev/sda4             4.1G  538M  3.4G  14% /cores
    /dev/tmpfs             32M  9.6M   23M  30% /tmp
    /dev/mirrora4          17G  2.9G   13G  19% /EMC/backend/service
    /dev/c4nasdba2       1019M   30M  938M   4% /EMC/backend/CEM
    /dev/c4nasdba1       1015M   40M  925M   5% /nbsnas
    /dev/c4loga2          653M   22M  599M   4% /EMC/backend/perf_stats
    /dev/c4loga1          3.4G  665M  2.6G  21% /EMC/backend/log_shared
  6. Connect to the peer SP and check available space:
    service@(none) spa:~> ssh peer
    Last login: Tue Dec  3 20:51:10 2013 from peer
    service@(none) spb:~> df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2             5.0G  2.9G  1.9G  61% / ======> That's much better!
    tmpfs                 725M  3.3M  722M   1% /dev/shm
    udev                  469M  924K  468M   1% /dev
    /dev/sda4             4.1G  2.0G  2.0G  50% /cores
    /dev/tmpfs             32M  9.9M   23M  31% /tmp
    /dev/mirrora4          17G  2.4G   13G  16% /EMC/backend/service
  7. Check if any software upgrade packages are still present in ‘/cores/service‘ and delete them ‘rm -rf VNXe*‘:
    root@(none) spb:~> cd /cores/service
    root@(none) spb:/cores/service> ls
    .bash_history  VNXe-MR4SP1-upgrade-2.4.1.21171-RETAIL.tgz.bin      svc_networkch
    .bashrc        VNXe-MR4SP1-upgrade-2.4.1.21171-RETAIL.tgz.bin.gpg  svc_puhccheck
    .ssh           svc_help                                            svc_puhccheck
    root@(none) spb:/cores/service> rm -rf VNXe*
  8. Run ‘df -h‘ to check available space again:
    root@spa:~>df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2             5.0G  2.7G  2.1G  57% / ======> Even better!
    tmpfs                 725M  2.8M  722M   1% /dev/shm
    udev                  469M  900K  468M   1% /dev
    /dev/sda4             4.1G  538M  3.4G  14% /cores
    /dev/tmpfs             32M  9.6M   23M  30% /tmp
    /dev/mirrora4          17G  2.9G   13G  19% /EMC/backend/service
    /dev/c4nasdba2       1019M   30M  938M   4% /EMC/backend/CEM
    /dev/c4nasdba1       1015M   40M  925M   5% /nbsnas
    /dev/c4loga2          653M   22M  599M   4% /EMC/backend/perf_stats
    /dev/c4loga1          3.4G  665M  2.6G  21% /EMC/backend/log_shared
  9. Run ‘svc_clean_ssd -c‘ again:
    root@spa:~>svc_clean_ssd -c
    INFO: Peer SP found - attempting to run on both SPs.
    Now searching for files that could be deleted on SPA ...Done!
    [Fri Dec  6 16:14:04 UTC 2013] Filesystem Usage Report (SPA) :
    =============================================
    Local Root filesystem mount point         : /
    Total Size of Drive                       : 5040MB
    Current Total Space Used                  : 2683MB, 53.23% usage
    A total of 9MB can be saved on this system by removing files.
    Total Space Used (after / cleanup) : 2674MB, 53.05% usage
  10. Go back to VNXe Unisphere GUI and upload Software Upgrade / Candidate Version;
  11. Make sure you have 20-25% of free space on /dev/sda2 file system:
    service@(none) spa:~> df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda2             5.0G  3.4G  1.3G  73% / ======> 20-25% of free space is OK
    tmpfs                 725M  2.8M  722M   1% /dev/shm
    udev                  469M  900K  468M   1% /dev
    /dev/sda4             4.1G  538M  3.4G  14% /cores
    /dev/tmpfs             32M  9.6M   23M  30% /tmp
    /dev/mirrora4          17G  2.9G   13G  19% /EMC/backend/service
    /dev/c4nasdba2       1019M   30M  938M   4% /EMC/backend/CEM
    /dev/c4nasdba1       1015M   35M  930M   4% /nbsnas
    /dev/c4loga2          653M   22M  599M   4% /EMC/backend/perf_stats
    /dev/c4loga1          3.4G  665M  2.6G  21% /EMC/backend/log_shared
  12. Proceed with the upgrade.

Hope this will help.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>