NTP Compliancy: disable NTP sync with host completely (PowerCLI)

Last week I was identifying a root cause for a P1 outage my customer had suffered from. This customer has a business-critical application running 24/7 which is very important for all activities within the company. They had suffered an outage for at least three times last month, and therefore they requested my employer ITQ to perform a health check including a root cause analysis for the latest P1 incidents.

It will not be a big surprise if I told you that the root cause was NTP? Although incorrectly configured NTP settings often lead to issues, unfortunately this is not yet configured consistently and well enough in a lot of environments.

The issues that the customer was facing was as follows. The ESXi hypervisor was pointing to an NTP server which was not existing anymore. So the time between the hypervisor and the domain controller was completely out of sync. After a vMotion of a VM with Server 2012 R2 as Guest OS (domain member), the VMware tools inside the Guest OS initiated a sync with the hypervisor what resulted in a 5 minutes time difference. Since the business-critical application was very time-sensitive, this lead into a service disruption within the application stack. The customer said to me; but the ‘Synchronize guest time with host’ is disabled on every VM in the environment, so is this a bug? The answer is No! and it is well explained in this VMware Blogpost.

I know there already a few blog post and articles regarding this topic, but I had a specific use case for the script I’ve written. And wanted to share it. Maybe someone will ever come into the same situation, in which this script can be very useful.

The script has been written to execute tasks on a per-host basis, since this was a requirement stated by the customer. Of course it can be easily rewritten to a per-cluster basis script. In this specific use-case environment DRS was disabled, the business-critical application which consist of 48 clustered VMs is not supported on a DRS enabled cluster.

What does the script do?

#1 – Export all the existing VMs on the ESXi host to a CSV file. Including a separate CSV file for the powered on VMs only. 
#2 – Power down (Shutdown Guest OS) all the VMs on the ESXi host, otherwise certain advanced configuration settings for the VMs cannot be set. A fail-safe has been built-in in case a VM doesn’t have VMware tools installed. Then the hypervisor can’t shut down the VM correctly.  The script will not continue until you manual turn off the remaining VMs. 
#3 – Setting the appropriate NTP servers on the ESXi host including the ‘Start and stop with host’ Policy.
#4 – Disable the VMware tools time sync for vMotion, Snapshots and so on.. by setting the advanced settings. (can only be done on a powered off VM)
#5 – Set the VMware tools time sync with host option per VM to $false
#6 – Export the advanced config for all the adjusted VMs to an CSV file
#7 – Power on all the VMs which were running when starting this script, all the powered off VMs will not be started.
#8 – Performing a latest check on the ‘VMware Tools timesync with host option’. If everything in the script executed without any errors the export file: vms_with_enabled_hostsync.csv should be empty.

The script:

#####################################################################
# Purpose: Adjust NTP and make it compliant and consistent within in the whole environment 
# Date: 3/26/2018 
# By: Wesley Geelhoed – ITQ 
#####################################################################

Get-Module-ListAvailable VMware*|Import-Module

##Infra-Info##
$NTP1=’x.x.x.x’
$NTP2 = ‘x.x.x.x’
$vcenterserver = ‘vcenter@fqdn.local’
$vcenteradmin = “administrator@fqdn.local”
$vcenteradminpw = “P@$$w0rd”
 
##Connect to vCenter##
write-host Connecting to vCenter Server instance $vcenterserver -ForegroundColor Yellow
Connect-VIServer $vcenterserver -User $vcenteradmin -Password $vcenteradminpw -Force

$esxiserver = Read-Host ‘Enter the FQDN of the ESXi server which need to be NTP compliant’

## –#1– –#2– Power Down All VMs##
$poweredonVMs = get-vmhost -name $esxiserver | Get-VM | Where-object {$_.powerstate -eq “poweredon”}
$poweredonVMs_exportcsv = get-vmhost -name $esxiserver | Get-VM | Where-object {$_.powerstate -eq “poweredon”} | Export-Csv C:\poweredon_vms_$esxiserver.csv
$allVMs = get-vmhost -name $esxiserver|Get-VM
$allVMs_exportcsv = get-vmhost -name $esxiserver | Get-VM | Export-Csv C:\all_vms_$esxiserver.csv -NoTypeInformation
Write-Host Shutdown guest VMs when applicable, you may see an error if there are no VMs with the Powered On status -ForegroundColor Yellow
$poweredononesx = Import-Csv C:\poweredon_vms_$esxiserver.csv

foreach ($item in $poweredononesx){
$povm = $item.Name
If (Get-vm $povm | Shutdown-VMGuest -Confirm:$false -WarningAction SilentlyContinue -Erroraction SilentlyContinue) {
write-host Shutting down all the VMs -ForegroundColor Green
write-host Waiting for shutdown to complete -ForegroundColor Yellow

do

{
$povm = Get-VM -Name $item.Name
$status =
$povm.Powerstate
write-host $povm Powerstate is: -ForegroundColor Yellow
$status
sleep 5
}until ($povm.Powerstate -eq “poweredoff”)
write-host $povm powered off successfully -ForegroundColor Green

} else {

do
{
write-host Not all VMs could be shutdown correctly, please bring the remaining VMs manually down -ForegroundColor Red
$povm = Get-VM-Name $item.Name
$status =
$povm.Powerstate
write-hos t$povm Powerstate is: -ForegroundColor Yellow
$status
sleep 5
}until ($povm.Powerstate -eq “poweredoff”)
write-host $povm powered off successfully -ForegroundColor Green
}

}

## –#3– Set NTP Server##
write-host Remove old NTP server on $esxiserver -ForegroundColor Yellow
$oldNTP = Get-VMHostNtpServer -VMHost $esxiserver
Get-VMHost -Name $esxiserver | Remove-VMHostNtpServer -NtpServer $oldNTP -Confirm:$false
write-host Configuring NTP server and enable NTP service on $esxiserver -ForegroundColor Yellow
Get-VMhost -Name $esxiserver | Add-VMHostNtpServer -NtpServer $NTP1, $NTP2
Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”} | Set-VMHostService -Policy “on”

Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”} | Start-VMHostService

## –#4– –#5– Disable VMtools time sync completely##
$vmconfigspec = New-Object VMware.Vim.VirtualMachineConfigSpec
$value1 = New-Object VMware.Vim.OptionValue
$value1.Key = ‘tools.syncTime’
$value1.Value = 0
$vmconfigspec.ExtraConfig += $value1
$value2 = New-Object VMware.Vim.OptionValue
$value2.Key = ‘time.synchronize.continue’
$value2.Value = 0
$vmconfigspec.ExtraConfig += $value2
$value3 = New-Object VMware.Vim.OptionValue
$value3.Key = ‘time.synchronize.restore’
$value3.Value = 0
$vmconfigspec.ExtraConfig += $value3
$value4 = New-Object VMware.Vim.OptionValue
$value4.Key = ‘time.synchronize.resume.disk’
$value4.Value = 0
$vmconfigspec.ExtraConfig += $value4
$value5 = New-Object VMware.Vim.OptionValue
$value5.Key = ‘time.synchronize.shrink’
$value5.Value = 0
$vmconfigspec.ExtraConfig += $value5
$value6 = New-Object VMware.Vim.OptionValue
$value6.Key = ‘time.synchronize.tools.startup’
$value6.Value = 0
$vmconfigspec.ExtraConfig += $value6
$value7 = New-Object VMware.Vim.OptionValue
$value7.Key = ‘time.synchronize.tools.enable’
$value7.Value = 0
$vmconfigspec.ExtraConfig += $value7
$value8 = New-Object VMware.Vim.OptionValue
$value8.Key = ‘time.synchronize.resume.host’
$value8.Value = 0
$vmconfigspec.ExtraConfig += $value8

$allvmsonesx = Import-Csv C:\all_vms_$esxiserver.csv
foreach($item in $allvmsonesx){
$vmname = $item.Name
write-host Disable VMware Tools timesync for $vmname -ForegroundColor Green
$vm = Get-VM -Name $vmname
$vm.ExtensionData.ReconfigVM($vmconfigspec)
$spectools = New-Object VMware.Vim.VirtualMachineConfigSpec
$spectools.tools= New-Object VMware.Vim.ToolsConfigInfo
$spectools.tools.SyncTimeWithHost=$false
$vm = Get-view-id $vm.Id
$vm.ReconfigVM_task($spectools)
}

## –#6– Export adjusted advanced configuration for VMs##
foreach ($item in $allvmsonesx){
$shut_vm = $item.Name
write-host Exporting adjusted advanced configurationto CSV file for $shut_vm -ForegroundColor Green
“Virtual Machine is $shut_vm” | Out-File -Encoding Ascii -append C:\adjusted_advancedconfig_vms_on_$esxiserver.csv
Get-AdvancedSetting -Entity $shut_vm | where {$_.Name -like “time*”} | select name, value | Out-File -Encoding Ascii -append C:\adjusted_advancedconfig_vms_on_$esxiserver.csv
}

## –#7– Power on VMs again##
foreach ($item in $poweredononesx){
$shut_vm = $item.Name
write-host Power on $shut_vm -ForegroundColor Green
Start-vm -vm $shut_vm -Confirm:$false

}

## –#8– Exporting VMs with timesync with host enabled to CSV##
foreach ($item in $allvmsonesx){
$shut_vm = $item.Name
write-host Check timesync with host parameter for $shut_vm -ForegroundColor Green
get-view -viewtype virtualmachine -Filter @{‘Config.Tools.SyncTimeWithHost’=’True’} | Where-object {$_.Name-eq $shut_vm} |select name |Out-File-Encoding Ascii -append C:\vms_with_enabled_hostsync_in_$vcenterserver.csv|Out-Null
}

##Finishing Script##
write-host
write-host Script has finished for $esxiserver the NTP settings are valid, and all VMs on this host are compliant! -ForegroundColor Yellow
write-host
write-host Current NTP setting for $esxiserver
Get-VMHostNtpServer -VMHost $esxiserver | fl | Write-Output

Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”}

 

#####################################################################

If you have any questions, find it useful or have any other remarks on this blog post please let me know!

Enjoy!

Sources: vmware.com, communities.vmware.com

Video:

 

4 comments

    1. Mischa,

      Thanks!

      No, this setting in the GUI differs from the advanced configuration settings. In my script I turned off both. But you can turn off time sync in event of an (Storage) vMotion, Snapshot, Power Cycle and leave the Time Sync with host enabled.

      The difference between the two is that the VMware tools sync settings which are in the advanced config only syncing when an event occurs and the ‘Time Sync with host’ option (if enabled) is providing a periodic time synchronization.

      Wesley

      Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s