Trend Micro Deep Security: Prepare ESX operation failed – The installation transaction failed


  • VMware vSphere ESXi 5.5 (VMware ESXi, 5.5.0, Build 1746974)
  • Trend Micro Deep Security Manager 9.0 SP1 Patch 3 (Build 9.0.6500)

Prepare ESX operation failed with the “The installation transaction failed” error.

Trend Micro Knowledge Base Solution ID 1102068.

Root Cause:
VMware has confirmed that this issue could happen when downloading and installing the Deep Security Filter Driver, or other vendors’ VIB files, on ESXi 5.5.
This issue does not happen on ESXi 5.1 and older versions, because it is caused by a newly-added logic on ESXi 5.5. This new logic processes multiple partner CRLs in /usr/share/certs/vmparter.crl, but it does not clear the “PEM_R_NO_START_LINE” error in the openSSL error queue. This causes inaccurate communication during VIB payload downloading.

Fix availability:
VMware will include the fix in ESXi 5.5 Update 2.

Recommended Action:
While waiting for the fix from VMware, do the following workaround:

  1. On the DSM, do the following:
    • For Windows: Open the Windows command line and change the directory path to C:Program FilesTrend MicroDeep Security Manager. Execute the following command:
      dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

      dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true
      Stopping Trend Micro Deep Security Manager...
      System Setting: settings.configuration.filterDriverNoSigCheck Value: true saved
      Starting Trend Micro Deep Security Manager...

      Don’t be fooled by the Complete statement. You may need to manually start the Trend Micro Deep Security Manager service up… :)

    • For Linux: Log in via SSH and run the following command:
      /opt/dsm/dsm_c -action changesetting -name "settings.configuration.filterDriverNoSigCheck" -value true

    The DSM service will stop and start again during the process.

  2. Go to the DSM console and prepare the ESX again.
    The Filter Driver installation should now be successful.

1 comment to Trend Micro Deep Security: Prepare ESX operation failed – The installation transaction failed

  • John

    This was extremely helpful! thank you very much…I was pulling my hair out trying to deploy the filter driver to a few hosts. It was also causing problems when attempting a manual install.

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>




This site uses Akismet to reduce spam. Learn how your comment data is processed.