Array: EMC VNXe 3100
Current version – 2.4.0.20932
Candidate version – 2.4.1.21171
EMC VNXe firmware upgrade failed at 15% with the following error message:
“Software Update failed:
Extracting the image on peer SP
The following error(s) caused the software upgrade to fail:
- The upgrade has failed. Try rebooting the Storage Processor (SPA) and then install the candidate version a second time. If this does not solve the problem, record the error codes and contact the VNXe Online Portal for all support options. [Error Code: No_Response_From_spa]”
I tried all of the following without any luck:
- Restarted Upgrade :);
- Rebooted SPA and SPB, re-upload the Candidate Firmware version and run the upgrade again;
- Restarted Management Software;
- Reimaged SPA with new image through GUI and CLI.
I also noticed SPA took a lot longer to come back online after restart and if I restart SPB and therefore fail the services over to SPA, Unisphere was not responding and only when SPB came back online, it started to function correctly. In the end, EMC accepted it may be a hardware fault and replaced SPA.
Unfortunately, even after we replaced SPA, VNXe upgrade did not finish successfully but this time it failed at 41% which was a good indication we were on the right track.
Before you start upgrading VNXe, make sure you check the available space on /dev/sda2 file system:
- SSH to VNXe Management IP address and login as ‘
service‘. The password may be different to the ‘admin‘ account; - Run ‘
df -h‘ to check the available space:service@(none) spa:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 5.0G 4.3G 401M 92% / =======> 8% of free space WILL NOT be enough! tmpfs 725M 2.8M 722M 1% /dev/shm udev 469M 900K 468M 1% /dev /dev/sda4 4.1G 538M 3.4G 14% /cores /dev/tmpfs 32M 9.6M 23M 30% /tmp /dev/mirrora4 17G 2.9G 13G 19% /EMC/backend/service /dev/c4nasdba2 1019M 30M 938M 4% /EMC/backend/CEM /dev/c4nasdba1 1015M 41M 924M 5% /nbsnas /dev/c4loga2 653M 22M 599M 4% /EMC/backend/perf_stats /dev/c4loga1 3.4G 665M 2.6G 21% /EMC/backend/log_shared
- Verify VNXe storage integrity – ‘
svc_storage_integritycheck‘:service@(none) spa:~> svc_storage_integritycheck Storage integrity verified successfully. All storage elements appear to be healthy.
- Delete unused files on both SP’s – ‘
svc_clean_ssd -c‘:service@(none) spa:~> svc_clean_ssd -c INFO: Peer SP found - attempting to run on both SPs. Now searching for files that could be deleted on SPA ...Done! [Fri Dec 6 15:46:29 UTC 2013] Filesystem Usage Report (SPA) : ============================================= Local Root filesystem mount point : / Total Size of Drive : 5040MB Current Total Space Used : 4351MB, 86.32% usage A total of 1130MB can be saved on this system by removing files. Total Space Used (after / cleanup) : 3221MB, 63.90% usage {skipped} Cleared 1130MB of space off of SPA root filesystem! {skipped} This script will clear up some space on the root filesystem. It can also atempt to find large files on the filesystem and generate a report so the user can analyze the findings. {skipped} - Run ‘
df -h‘ to check available space:service@(none) spa:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 5.0G 3.2G 1.6G 68% / ======> 1.6 GB or 32% is free! tmpfs 725M 2.8M 722M 1% /dev/shm udev 469M 900K 468M 1% /dev /dev/sda4 4.1G 538M 3.4G 14% /cores /dev/tmpfs 32M 9.6M 23M 30% /tmp /dev/mirrora4 17G 2.9G 13G 19% /EMC/backend/service /dev/c4nasdba2 1019M 30M 938M 4% /EMC/backend/CEM /dev/c4nasdba1 1015M 40M 925M 5% /nbsnas /dev/c4loga2 653M 22M 599M 4% /EMC/backend/perf_stats /dev/c4loga1 3.4G 665M 2.6G 21% /EMC/backend/log_shared
- Connect to the peer SP and check available space:
service@(none) spa:~> ssh peer Last login: Tue Dec 3 20:51:10 2013 from peer service@(none) spb:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 5.0G 2.9G 1.9G 61% / ======> That's much better! tmpfs 725M 3.3M 722M 1% /dev/shm udev 469M 924K 468M 1% /dev /dev/sda4 4.1G 2.0G 2.0G 50% /cores /dev/tmpfs 32M 9.9M 23M 31% /tmp /dev/mirrora4 17G 2.4G 13G 16% /EMC/backend/service
- Check if any software upgrade packages are still present in ‘
/cores/service‘ and delete them ‘rm -rf VNXe*‘:root@(none) spb:~> cd /cores/service root@(none) spb:/cores/service> ls .bash_history VNXe-MR4SP1-upgrade-2.4.1.21171-RETAIL.tgz.bin svc_networkch .bashrc VNXe-MR4SP1-upgrade-2.4.1.21171-RETAIL.tgz.bin.gpg svc_puhccheck .ssh svc_help svc_puhccheck root@(none) spb:/cores/service> rm -rf VNXe*
- Run ‘
df -h‘ to check available space again:root@spa:~>df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 5.0G 2.7G 2.1G 57% / ======> Even better! tmpfs 725M 2.8M 722M 1% /dev/shm udev 469M 900K 468M 1% /dev /dev/sda4 4.1G 538M 3.4G 14% /cores /dev/tmpfs 32M 9.6M 23M 30% /tmp /dev/mirrora4 17G 2.9G 13G 19% /EMC/backend/service /dev/c4nasdba2 1019M 30M 938M 4% /EMC/backend/CEM /dev/c4nasdba1 1015M 40M 925M 5% /nbsnas /dev/c4loga2 653M 22M 599M 4% /EMC/backend/perf_stats /dev/c4loga1 3.4G 665M 2.6G 21% /EMC/backend/log_shared
- Run ‘
svc_clean_ssd -c‘ again:root@spa:~>svc_clean_ssd -c INFO: Peer SP found - attempting to run on both SPs. Now searching for files that could be deleted on SPA ...Done! [Fri Dec 6 16:14:04 UTC 2013] Filesystem Usage Report (SPA) : ============================================= Local Root filesystem mount point : / Total Size of Drive : 5040MB Current Total Space Used : 2683MB, 53.23% usage A total of 9MB can be saved on this system by removing files. Total Space Used (after / cleanup) : 2674MB, 53.05% usage
- Go back to VNXe Unisphere GUI and upload Software Upgrade / Candidate Version;
- Make sure you have 20-25% of free space on
/dev/sda2file system:service@(none) spa:~> df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 5.0G 3.4G 1.3G 73% / ======> 20-25% of free space is OK tmpfs 725M 2.8M 722M 1% /dev/shm udev 469M 900K 468M 1% /dev /dev/sda4 4.1G 538M 3.4G 14% /cores /dev/tmpfs 32M 9.6M 23M 30% /tmp /dev/mirrora4 17G 2.9G 13G 19% /EMC/backend/service /dev/c4nasdba2 1019M 30M 938M 4% /EMC/backend/CEM /dev/c4nasdba1 1015M 35M 930M 4% /nbsnas /dev/c4loga2 653M 22M 599M 4% /EMC/backend/perf_stats /dev/c4loga1 3.4G 665M 2.6G 21% /EMC/backend/log_shared
- Proceed with the upgrade.
Hope this will help.
Recent Comments