Visitors

HOW TO: Start, Stop, Restart services on VMware vSphere ESX server via PowerCLI

The service you will need to start or stop simultaneously on multiple servers is likely to be the SSH (TSM-SSH) service.

Here is how you do it with PowerShell / PowerCLI:

$vCenter = 'vCenter.vstrong.info' ; Name or IP address of your vCenter Server
$Cluster = 'Vblock cluster 1'; Name of the cluster
Connect-VIServer $vCenter
$Scope = Get-Cluster $Cluster | Get-VMHost
foreach ($ESXhost in $Scope){
Get-VMHost -name $ESXhost | Get-VMHostService | where {$_.key -eq 'TSM-SSH'} | Start-VMHostService -Confirm:$false
# Change the line above with the following:
# Start-VMHostService
# Stop-VMHostService
# Restart-VMHostService
}
Disconnect-VIServer -Server $vCenter -Confirm:$false

Should you need to start / stop or restart other services, just change the service key.

Here is the list of services:

Get-VMHost -name Name_or_IP_address_of_a_host | Get-VMHostService | Format-Table -AutoSize

Result:

Key            Label                                                           Policy Running Required
---            -----                                                           ------ ------- --------
DCUI           Direct Console UI                                               on     True    False   
TSM            ESXi Shell                                                      off    False   False   
TSM-SSH        SSH                                                             off    False   False   
lbtd           lbtd                                                            on     True    False   
lsassd         Local Security Authentication Server (Active Directory Service) off    False   False   
lwiod          I/O Redirector (Active Directory Service)                       off    False   False   
netlogond      Network Login Server (Active Directory Service)                 off    False   False   
ntpd           NTP Daemon                                                      on     True    False   
sfcbd-watchdog CIM Server                                                      on     True    False   
vmware-fdm     vSphere High Availability Agent                                 on     True    False   
vpxa           vpxa                                                            on     True    False

HOW TO: Find HBA WWN number on VMware vSphere ESX server

There are several ways to get HBA WWNs on VMware vSphere ESX/ESXi host:

  1. vSphere Client;
  2. Using ESXi Shell;
  3. Using Powershell / PowerCLI script.

1. Connect to a server or vCenter, open server Configuration tab, under Hardware select Storage Adapters:

You can also copy WWNN (World Wide Node Name) and WWPN (World Wide Port Name)

2. How to find HBA WWN via ESXi Shell / CLI:

VMware vSphere ESXi 5.0+:

~ # esxcli storage core adapter list
HBA Name  Driver        Link State  UID                                   Description
--------  ------------  ----------  ------------------------------------  ------------------------------------------------------------
vmhba0    megaraid_sas  link-n/a    unknown.vmhba0                        (0:1:0.0) LSI / Symbios Logic MegaRAID SAS SKINNY Controller
vmhba1    fnic          link-up     fc.20000025b5020110:20000025b502a121  (0:8:0.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba2    fnic          link-up     fc.20000025b5020110:20000025b502a120  (0:9:0.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver

VMware ESX/ESXi 2.1.0 – 4.1.x:

~ # esxcfg-scsidevs -a
vmhba0  megaraid_sas      link-n/a  unknown.vmhba0                          (0:1:0.0) LSI / Symbios Logic MegaRAID SAS SKINNY Controller
vmhba1  fnic              link-up   fc.20000025b5020110:20000025b502a121    (0:8:0.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver
vmhba2  fnic              link-up   fc.20000025b5020110:20000025b502a120    (0:9:0.0) Cisco Systems Inc Cisco VIC FCoE HBA Driver

OR

  • Connect to ESXi shell either via putty/SSH or DCUI (Direct Console User Interface) / server console
  • Run ‘ls /proc/scsi/‘ and check the folder names:
    ~ # ls /proc/scsi/
     mptsas   qla2xxx
  • Look for a folder like ‘qla2xxx‘ – QLogic HBA, ‘lpfc820‘ – Emulex HBA, ‘bnx2i” – Brocade HBA;
  • Run ‘ls /proc/scsi/qla2xxx’. You will get a list of files, named by a number. Each file contains information about one HBA;
    ~ # ls /proc/scsi/qla2xxx/
    6  7
  • Now run ‘cat /proc/scsi/qla2xxx/6‘ to get full info on the HBA. Alternatively, run the following commands:
    • Run ” cat /proc/scsi/qla2xxx/6 | grep -A3 ‘SCSI Device Information:’  ” to get WWNN and WWPNs:
      ~ # cat /proc/scsi/qla2xxx/6 | grep -A3 'SCSI Device Information:'
      SCSI Device Information:
      scsi-qla0-adapter-node=20000024ff31f0c8:000000:0;
      scsi-qla0-adapter-port=21000024ff31f0c8:000000:0;
    • Run ” cat /proc/scsi/qla2xxx/6 | grep ‘Host Device Name’ ” to get vmhba number:
      ~ # cat /proc/scsi/qla2xxx/6 | grep 'Host Device Name'
      Host Device Name vmhba3

3. Powershell script to list host name, vmhba number, HBA model / driver and World Wide Port Name (WWN):
$scope = Get-VMHost     # All hosts connected in vCenter
#$scope = Get-Cluster -Name 'MyCluster' | Get-VMHost # All hosts in a specific cluster
foreach ($esx in $scope){
Write-Host "Host:", $esx
$hbas = Get-VMHostHba -VMHost $esx -Type FibreChannel
foreach ($hba in $hbas){
$wwpn = "{0:x}" -f $hba.PortWorldWideName
Write-Host `t $hba.Device, "|", $hba.model, "|", "World Wide Port Name:" $wwpn
}}

Result:

Host: ESXi5-001.vstrong.info
vmhba1 | Cisco VIC FCoE HBA Driver | World Wide Port Name: 20000025b502a101
vmhba2 | Cisco VIC FCoE HBA Driver | World Wide Port Name: 20000025b502a100

Trend Micro Deep Security: Anti-Malware Engine Offline

In my ongoing battle with Trend Micro Deep Security I came across another problem / strange behavior. Virtual machines running  on a particular host had Anti-Malware Engine Offline status.

DSVA was Managed (Online) and did not present any other issues. Going through the usual sequence <Clear Warnings/Errors>  – <Check Status> did not clear the error message. I checked if vShield driver is installed and running on the VM and it was the case. After vMotioning virtual machine to another host the status changed to Managed (Online). When vMotioned back to the original host is changes back to Anti-Malware Engine Offline. Clearly, it was the case of some DSVA / vShiled issue on that host.

After pocking around DSVA / vShiled Endpoint driver it has been identified that UserVar.VshieldEndpointSolutionsConfiguration had a duplicate entry:

esxcfg-advcfg --get /UserVars/VshieldEndpointSolutionsConfiguration

Result:

~ # esxcfg-advcfg --get /UserVars/VshieldEndpointSolutionsConfiguration
 Value of VshieldEndpointSolutionsConfiguration is <id:7498352642083520512;ip:169.254.1.39;port:48651;><id:7498352642083520512;ip:169.254.1.39;port:48651;>

You can also check it via vSphere Client:

Open vSphere Client, connect to vCenter or directly to the host, click on Configuration tab, under Software click on Advanced Settings, scroll to the bottom and select UserVars.

… and PowerCLI command:

Get-VMHostAdvancedConfiguration -VMHost 'HostName' -Name 'UserVars.VshieldEndpointSolutionsConfiguration'  | Format-Table -AutoSize

Result:

Name                                           Value
 ----                                           -----
 UserVars.VshieldEndpointSolutionsConfiguration <id:7498352642083520512;ip:169.254.1.39;port:48651;><id:7498352642083520512;ip:169.254.1.39;port:48651;>

SOLUTION:

Remove duplicate entry and restart vShield Endpoint service

Restart vShield Endpoint service:

/etc/init.d/vShield-Endpoint-Mux restart

Result:

~ # /etc/init.d/vShield-Endpoint-Mux restart
 vShield-Endpoint-Mux stopped
 vShield-Endpoint-Mux started

Check if vShield Endpoint driver is running:

ps | grep vShield-Endpoint-Mux

Result:

~ # ps | grep vShield-Endpoint-Mux
 4024587 4024587 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux
 4008208 4024587 vShield-Endpoint-Mux /usr/lib/vmware/vShield-Endpoint-Mux

The issue has been resolved and VM status changed to Managed (Online)

Hope this will help.

HOW TO: Enable or Disable Lockdown Mode on VMware vSphere ESXi host

Q: What is Lockdown Mode?
A: Lockdown Mode prevents users from logging directly to the host. The host will only be accessible through local console or vCenter Server. None of remote management options e.g. vCLI, PowerCLI script, SSH will work. When it is enabled, only vpxuser () has authentication permissions and can connect to the host remotely.

How can you enable/disable Lockdown Mode:

  1. From the Direct Console User Interface (DCUI);
  2. From vSphere Client;
  3. Using ESXi Shell;
  4. Using PowerCLI script.


1. Enable/Disable Lockdown Mode from DCUI:

  1. Open server console;
  2. Press F2 to Customize System/View Logs;
  3. Open Configure Lockdown Mode;
  4. Press SPACE to enable or disable lockdown mode;
  5. Press ENTER to save the changes.


2. Enable/Disable Lockdown Mode in vSphere Client:

  1. Open vSphere Client and connect to the host / vCenter Server;
  2. Select the host and click on Configuration tab;
  3. Under Software select Security Profile;
  4. Under Lockdown Mode click Edit;
  5. Tick / Untick Enable Lockdown Mode;
  6. Click OK.


3. Enable/Disable Lockdown Mode in ESXi Shell:

Check if Lockdown mode is enabled: vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
Enable Lockdown mode: vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter
Disable Lockdown mode: vim-cmd -U dcui vimsvc/auth/lockdown_mode_exit

Example:

~ # vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
false
~ # vim-cmd -U dcui vimsvc/auth/lockdown_mode_enter
~ # vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
true
~ # vim-cmd -U dcui vimsvc/auth/lockdown_mode_exit
~ # vim-cmd -U dcui vimsvc/auth/lockdown_is_enabled
false


4. Enable/Disable Lockdown Mode using PowerCLI script:

$vCenter = 'vCenterServer_Name_or_IP_address'
Connect-VIServer $vCenter
 $Scope = Get-VMHost #This will change the Lockdown Mode on all hosts managed by vCenter, amend this if you need to
 foreach ($ESXhost in $Scope) {
 (get-vmhost $ESXhost | get-view).ExitLockdownMode() # To DISABLE Lockdown Mode
 # (get-vmhost $ESXhost | get-view).EnterLockdownMode() # To ENABLE Lockdown Mode
 }
Disconnect-VIServer -Server $vCenter -Confirm:$false

HOW TO: Identify what vNICs correspond to physical NICs

How to identify what vNICs correspond to physical NICs on HP ProLiant BL680 G5 full-height blade with two quad-port mezzanine cards:

This diagram is for a blade in Device Bay 1:

VMware NICs:

vmnic0 Embedded Ethernet Port 1 Bay 1 Port 9
vmnic1 Embedded Ethernet Port 2 Bay 2 Port 9
vmnic2 Embedded Ethernet Port 3 Bay 1 Port 1
vmnic3 Embedded Ethernet Port 4 Bay 2 Port 1
vmnic4 Mezzanine Slot 2 Port 1 Bay 5 Port 1
vmnic5 Mezzanine Slot 2 Port 2 Bay 6 Port 1
vmnic6 Mezzanine Slot 2 Port 3 Bay 7 Port 1
vmnic7 Mezzanine Slot 2 Port 4 Bay 8 Port 1
vmnic8 Mezzanine Slot 3 Port 1 Bay 7 Port 9
vmnic9 Mezzanine Slot 3 Port 2 Bay 8 Port 9
vmnic10 Mezzanine Slot 3 Port 3 Bay 5 Port 9
vmnic11 Mezzanine Slot 3 Port 4 Bay 6 Port 9

HOW TO: Create a new user and add it to a role on ESXi host.

This script creates a new user and adds the user to a role on the ESX host:

$NewUser = '_New_user_name_'
$NewUserPassword = '_New_user_Password_'
$NewUserDesc = '_New_user_description_'
$HOSTCredentials = Get-Credential -Credential root
$ESXhost = '_My_ESX_HOST_'

Connect-VIServer $ESXhost -Credential $HOSTCredentials
New-VMHostAccount -Id $NewUser -Password $NewUserPassword -Description $NewUserDesc -UserAccount -Server $ESXhost -AssignGroups users
$AuthMgr = Get-View (Get-View ServiceInstance).Content.AuthorizationManager
$Entity = Get-Folder ha-folder-root | Get-View
$Perm = New-Object VMware.Vim.Permission
$Perm.entity = $Entity.MoRef
$Perm.group = $false
$Perm.principal = $NewUser
$Perm.propagate = $true
# You can either specify roleID or use the line below if you know the role name.
# $Perm.roleId = ($AuthMgr.RoleList | where {$_.Name -eq "ReadOnly"}).RoleId
$Perm.roleId = "-2"
$AuthMgr.SetEntityPermissions($Entity.MoRef,$Perm)

Disconnect-VIServer -Server $ESXhost -Confirm:$false

Here is the list of common roles / role IDs:

RoleName                     Label                                    RoleId
--------                     -----                                    ------
NoAccess                     No access                                -5
Anonymous                    Anonymous                                -4
View                         View                                     -3
ReadOnly                     Read-only                                -2
Admin                        Administrator                            -1
VirtualMachinePowerUser      Virtual machine power user (sample)      4
VirtualMachineUser           Virtual machine user (sample)            5
ResourcePoolAdministrator    Resource pool administrator (sample)     6
VMwareConsolidatedBackupUser VMware Consolidated Backup user (sample) 7
DatastoreConsumer            Datastore consumer (sample)              8
NetworkConsumer              Network consumer (sample)                9

For the full list of Roles you have in your environment please use this script:

$report =@()
$authMgr = Get-View AuthorizationManager
foreach($role in $authMgr.RoleList){
$row = "" | Select RoleName, Label, RoleId
$row.RoleName = $role.Name
$row.Label = $role.Info.Label
$row.RoleId = $role.RoleId
$report += $row
}
$report

HOW TO: Check if VMware vShield driver is installed and running

  1. Login to the server
  2. Run msinfo32.exe
  3. Navigate to “Software Components“, “System Drivers
  4. Check if the ‘vsepflt‘ / ‘VFileFilter‘ system driver is in the list and the state is ‘Running’

VMware products firewall configuration

What ports do I need to open for VMware vSphere ESXi 5.x and vCenter Server / Update Manager?

You can study vSphere Installation and Setup guide and VMware vSphere 5.x Security Guide. I also found VMware KB 1012382 article: TCP and UDP Ports required to access vCenter Server, ESX hosts, and other network components and, for your convenience, saved the firewall port tables as Excell spreadsheet which can be filtered by product.

HOW TO: Upgrade old VMware Tools

In my last project the customer had a large number of virtual machines with a really old VMware Tools installed, some with version 3.1.0000. Neither Automatic nor Interactive Tool Upgrade worked. I tried to uninstall VMware Tools via Add/Remove Programs but it did work not either as Windows Installer could not find VMware Tools.msi.

The question is “Where can I download previous / old version of VMware Tools from?”
Here is the link:
http://packages.vmware.com/tools/esx

If you cannot find the source code / ISO of exact version of VMware Tools installed on the virtual machine, then just find the version closest to the version installed and run the upgrade.

For example, when I could not find the source files for VMware Tools version 3.1.0000, I upgraded VMware Tools to v. 3.5.0 – 110268 and then upgraded the tools to the latest version through Automatic/Interactive Tools Upgrade.

HOW TO: Configure EMC VNX Control Station and DataMovers to use a NTP Server

Open EMC Unisphere and login with your sysadmin account.

Navigate to Home, from drop down menu select ‘All Systems’ and then click on ‘Domains’:

Click on ‘Configure NTP’
Click OK on ‘NTP time synchronization will override manual clock settings.
NTP is applied to the Local Domain only.’

Tick ‘Enable NTP’ and select time synchronization interval from the drop down menu. The options are: 30 mins (default), 1 hour, 4 hours, 12 hours, 1 day, 7 days, 30 days.

Type IP address(es) of the NTP servers. You can configure up to ten NTP servers maximum.

Click Apply. Click OK

Make sure EMC Control Station can ping NTP server IP addresses! To check if it can please SSH to Control Station and run ‘ping *NTP_SERVER_IP_ADDRDSS*’

If you have a single VNX you will not have the Domains options and therefore need to login to the Control Station as ‘root’, select your array from the systems list, click System menu and select ‘Control Station Properties’ under ‘System Management’.

Click OK to save settings.