Visitors

Trend Micro Deep Security: Anti-Malware Engine Offline

In my ongoing battle with Trend Micro Deep Security I came across another problem / strange behavior. Virtual machines running  on a particular host had Anti-Malware Engine Offline status.

DSVA was Managed (Online) and did not present any other issues. Going through the usual sequence <Clear Warnings/Errors>  – <Check Status> did not clear the error message. I checked if vShield driver is installed and running on the VM and it was the case. After vMotioning virtual machine to another host the status changed to Managed (Online). When vMotioned back to the original host is changes back to Anti-Malware Engine Offline. Clearly, it was the case of some DSVA / vShiled issue on that host.

After pocking around DSVA / vShiled Endpoint driver it has been identified that UserVar.VshieldEndpointSolutionsConfiguration had a duplicate entry:

esxcfg-advcfg --get /UserVars/VshieldEndpointSolutionsConfiguration

Result:

~ # esxcfg-advcfg --get /UserVars/VshieldEndpointSolutionsConfiguration
 Value of VshieldEndpointSolutionsConfiguration is <id:7498352642083520512;ip:169.254.1.39;port:48651;><id:7498352642083520512;ip:169.254.1.39;port:48651;>

You can also check it via vSphere Client:

Open vSphere Client, connect to vCenter or directly to the host, click on Configuration tab, under Software click on Advanced Settings, scroll to the bottom and select UserVars.

… and PowerCLI command:

Get-VMHostAdvancedConfiguration -VMHost 'HostName' -Name 'UserVars.VshieldEndpointSolutionsConfiguration'  | Format-Table -AutoSize

Result:

Name                                           Value
 ----                                           -----
 UserVars.VshieldEndpointSolutionsConfiguration <id:7498352642083520512;ip:169.254.1.39;port:48651;><id:7498352642083520512;ip:169.254.1.39;port:48651;>

SOLUTION:

Remove duplicate entry and restart vShield Endpoint service

Restart vShield Endpoint service:

/etc/init.d/vShield-Endpoint-Mux restart

Result:

~ # /etc/init.d/vShield-Endpoint-Mux restart
 vShield-Endpoint-Mux stopped
 vShield-Endpoint-Mux started

Check if vShield Endpoint driver is running:

ps | grep vShield-Endpoint-Mux

Result:

~ # ps | grep vShield-Endpoint-Mux
 4024587 4024587 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux
 4008208 4024587 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux

The issue has been resolved and VM status changed to Managed (Online)

Hope this will help.

16 comments to Trend Micro Deep Security: Anti-Malware Engine Offline

  • Nguyen Quoc Dung

    Thanks very much for your information. It’s exactly my sistuation and now problem solved.
    I just wonder why this param has many duplicate entry.
    Before that, our system running well, just after a power suddenly off and make whole system down, error “Anti-Malware Engine Offline” appeared in many VMs.

    • Hi Nguyen,
      Unfortunately, we did not get to the bottom of it… Neither VMware nor Trend could tell us the reason this happened. We had to install, re-install vShiled, DSVAs and everything else so many time that we lost the track of what is what. Will keep you posted if we discover the root cause though.

  • Congratulations for the post!!!

  • Nguyen Quoc Dung

    Hi Mark,
    I tried to replicate this error in my lab: suddently power off all systems. Error happens again. I think maybe problem because ESX hosts or vCenter started before VSM started (all VMs include VSM, DSM, vCenter… in 2 ESXi Hosts).

    • Hi Nguyen,
      I upgraded vSphere to ESXi 5.0 Update 1 build 768111, VSM from 5.0 Build 473791 to 5.0.2 Build 791471, vShield Endpoint from 5.0.0-447150 to 5.0.1-638861 re-activated DSVA on several hosts in one datacenter last weekend. Two of the host I upgraded had duplicate UserVar.VshieldEndpointSolutionsConfiguration. As there were a lot of updates at the same time, I could not figure out which upgrade caused hosts to have duplicate vShield Endpoint entries. I will be upgrading hosts in a different datacenter in the next couple of weeks and will try to catch what causes this problem to reappear. Will keep you posted.

    • Nguyen, I upgraded vShield Manager and Endpoint last weekend and followed the procedure described in this post: http://www.vstrong.info/2012/10/22/how-to-upgrade-vshield-manager-vshield-endpoint-and-dont-break-trend-micro-dsva/ The upgrade did not incur duplicate values. It is likely that we did not deactivate DSVAs prior to vShiled upgrade and that is what caused duplicated values in the Endpoint configuration.

  • Thanks for these posts, Mark – I’ve been struggling with a failed Deep Security 8 environment for several weeks now (46 DSVAs, 1300 VMs) and it’s a bit comforting to see a counterpoint to all of the “but we haven’t heard of these problems with Deep Security!” I’ve gotten from both VMware and Trend.

    We’ve had to implement a few undocumented tweaks to DSVAs, to the DSM, and VMware has had to manually delete duplicate entries in the vShield Manager db in order to clear up some of the issues we’ve run into. We’re now at the point of reinstalling the entire infrastructure from soup to nuts with Trend’s support. I will be sure to keep your posts in mind should we run into these specific issues in the immediate future.

    • Thank you Mike. Yep, that is exactly what we did: Installed Trend Micro Deep Security, struggled for six months with all the ‘issues’, blamed DSVA for all all connectivity problems, packet drops etc etc and then deleted everything and re-installed TM DS. It is working OK at the moment but it will take time to re-gain trust in the product!… Would you be able to share your experience or HOW TOs? Thank you.

      • Absolutely! We’re currently working with Trend on-site based on how unstable the environment has been. We were looking to roll this out to a much larger footprint (hundreds of hosts, thousands of VMs) but have had to delay that until we can narrow down what’s been happening.

        As of the moment, most instability seems to be centered around the DSM itself. Communication on port 4120 is failing when appliances need to communicate back to the DSM – test by logging into the DSVA (Alt-F2) and telnet to the DSM IP or FQDN on port 4120. If successful you’ll get a quick string of garbled characters in the console. But what we’re finding is that that communication will start to take longer and longer, eventually timing out and then outright failing. A reboot of the DSM restores connectivity.

  • Ravi

    how we can found the same way to windows vm’s whether Vsepflt services running on multiple machine

  • Good day gentleman,

    We are having the same problem as you describe, with a small twist though:
    We have some VM’s on the same host which are running without any problems, but on the same appliance and esx host, have problems with some VM’s.

    • Hi Tijmen,
      There may be an issue with VMware Tools on those VMs. Endpoint driver may not be installed or a “deactivate/activate” procedure is required.

      • Hi Mark,

        Thanks for the fast reply!
        According to vSphere, the VMWare tools are up to date, am currently busy removing them all together, then reinstalling.
        I have tried deactivate on 1 VM in the DS manager, but when activating, getting errors about missing appliance. But in the DSMgr it does see the appliance :S

        Please keep in mind that i am a total noob in this :) Only just moved over to my sys admin function ;)

        • Hi Tijmen,
          When you install VMware Tools (for Windows VMs), make sure you include Endpoint Drivers (you need to either run FULL or Custom install or use the script).
          For Linux VMs you still need to install the driver.
          Make sure that the host is prepared, DSVA is installed and activated, all VMs have policies configured correctly and activated.

          • Hosts are prepared, DSVA is installed and activated.
            The problem is that we have VM’s which are working on ESX host 1, DSVA 1. We have also VM’s which are on ESX1 and DSVA1 which dont work.. that’s the strange thing..
            The VM where i just reinstalled the VMTools on, says it has no applianice. How do i assign it to one? :S

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>