Trend Micro Deep Security: Prepare ESX operation failed – The installation transaction failed

Environment:

• VMware vSphere ESXi 5.5 (VMware ESXi, 5.5.0, Build 1746974)
• Trend Micro Deep Security Manager 9.0 SP1 Patch 3 (Build 9.0.6500)

Prepare ESX operation failed with the “The installation transaction failed” error.

Trend Micro Knowledge Base Solution ID 1102068.

Root Cause:
VMware has confirmed that this issue could happen when downloading and installing the Deep Security Filter Driver, or other vendors’ VIB files, on ESXi 5.5.
This issue does not happen on ESXi 5.1 and older versions, because it is caused by a newly-added logic on ESXi 5.5. This new logic processes multiple partner CRLs in /usr/share/certs/vmparter.crl, but it does not clear the “PEM_R_NO_START_LINE” error in the openSSL error queue. This causes inaccurate communication during VIB payload downloading.

Fix availability:
VMware will include the fix in ESXi 5.5 Update 2.

Recommended Action:
While waiting for the fix from VMware, do the following workaround:

1. On the DSM, do the following:
   • For Windows: Open the Windows command line and change the directory path to C:Program FilesTrend MicroDeep Security Manager. Execute the following command:
     dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

     dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true
     Stopping Trend Micro Deep Security Manager...
     System Setting: settings.configuration.filterDriverNoSigCheck Value: true saved
     Starting Trend Micro Deep Security Manager...
     Complete

     Don’t be fooled by the Complete statement. You may need to manually start the Trend Micro Deep Security Manager service up… :)

   • For Linux: Log in via SSH and run the following command:
     /opt/dsm/dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

   The DSM service will stop and start again during the process.

2. Go to the DSM console and prepare the ESX again.
   The Filter Driver installation should now be successful.