NTP Compliancy: disable NTP sync with host completely (PowerCLI)

Last week I was identifying a root cause for a P1 outage my customer had suffered from. This customer has a business-critical application running 24/7 which is very important for all activities within the company. They had suffered an outage for at least three times last month, and therefore they requested my employer ITQ to perform a health check including a root cause analysis for the latest P1 incidents.

It will not be a big surprise if I told you that the root cause was NTP? Although incorrectly configured NTP settings often lead to issues, unfortunately this is not yet configured consistently and well enough in a lot of environments.

The issues that the customer was facing was as follows. The ESXi hypervisor was pointing to an NTP server which was not existing anymore. So the time between the hypervisor and the domain controller was completely out of sync. After a vMotion of a VM with Server 2012 R2 as Guest OS (domain member), the VMware tools inside the Guest OS initiated a sync with the hypervisor what resulted in a 5 minutes time difference. Since the business-critical application was very time-sensitive, this lead into a service disruption within the application stack. The customer said to me; but the ‘Synchronize guest time with host’ is disabled on every VM in the environment, so is this a bug? The answer is No! and it is well explained in this VMware Blogpost.

I know there already a few blog post and articles regarding this topic, but I had a specific use case for the script I’ve written. And wanted to share it. Maybe someone will ever come into the same situation, in which this script can be very useful.

The script has been written to execute tasks on a per-host basis, since this was a requirement stated by the customer. Of course it can be easily rewritten to a per-cluster basis script. In this specific use-case environment DRS was disabled, the business-critical application which consist of 48 clustered VMs is not supported on a DRS enabled cluster.

What does the script do?

#1 – Export all the existing VMs on the ESXi host to a CSV file. Including a separate CSV file for the powered on VMs only.
#2 – Power down (Shutdown Guest OS) all the VMs on the ESXi host, otherwise certain advanced configuration settings for the VMs cannot be set. A fail-safe has been built-in in case a VM doesn’t have VMware tools installed. Then the hypervisor can’t shut down the VM correctly. The script will not continue until you manual turn off the remaining VMs.
#3 – Setting the appropriate NTP servers on the ESXi host including the ‘Start and stop with host’ Policy.
#4 – Disable the VMware tools time sync for vMotion, Snapshots and so on.. by setting the advanced settings. (can only be done on a powered off VM)
#5 – Set the VMware tools time sync with host option per VM to $false
#6 – Export the advanced config for all the adjusted VMs to an CSV file
#7 – Power on all the VMs which were running when starting this script, all the powered off VMs will not be started.
#8 – Performing a latest check on the ‘VMware Tools timesync with host option’. If everything in the script executed without any errors the export file: vms_with_enabled_hostsync.csv should be empty.

The script:

#####################################################################

# Purpose: Adjust NTP and make it compliant and consistent within in the whole environment

# Date: 3/26/2018

# By: Wesley Geelhoed – ITQ

#####################################################################

Get-Module-ListAvailable VMware*|Import-Module

##Infra-Info##

$NTP1=’x.x.x.x’

$NTP2 = ‘x.x.x.x’

$vcenterserver = ‘vcenter@fqdn.local’

$vcenteradmin = “administrator@fqdn.local”

$vcenteradminpw = “P@$$w0rd”

##Connect to vCenter##

write-host Connecting to vCenter Server instance $vcenterserver -ForegroundColor Yellow

Connect-VIServer $vcenterserver -User $vcenteradmin -Password $vcenteradminpw -Force

$esxiserver = Read-Host ‘Enter the FQDN of the ESXi server which need to be NTP compliant’

## –#1– –#2– Power Down All VMs##

$poweredonVMs = get-vmhost -name $esxiserver | Get-VM | Where-object {$_.powerstate -eq “poweredon”}

$poweredonVMs_exportcsv = get-vmhost -name $esxiserver | Get-VM | Where-object {$_.powerstate -eq “poweredon”} | Export-Csv C:\poweredon_vms_$esxiserver.csv

$allVMs = get-vmhost -name $esxiserver|Get-VM

$allVMs_exportcsv = get-vmhost -name $esxiserver | Get-VM | Export-Csv C:\all_vms_$esxiserver.csv -NoTypeInformation

Write-Host Shutdown guest VMs when applicable, you may see an error if there are no VMs with the Powered On status -ForegroundColor Yellow

$poweredononesx = Import-Csv C:\poweredon_vms_$esxiserver.csv

foreach ($item in $poweredononesx){

$povm = $item.Name

If (Get-vm $povm | Shutdown-VMGuest -Confirm:$false -WarningAction SilentlyContinue -Erroraction SilentlyContinue) {

write-host Shutting down all the VMs -ForegroundColor Green

write-host Waiting for shutdown to complete -ForegroundColor Yellow

{

$povm = Get-VM -Name $item.Name

$status =

$povm.Powerstate

write-host $povm Powerstate is: -ForegroundColor Yellow

$status

sleep 5

}until ($povm.Powerstate -eq “poweredoff”)

write-host $povm powered off successfully -ForegroundColor Green

} else {

{

write-host Not all VMs could be shutdown correctly, please bring the remaining VMs manually down -ForegroundColor Red

$povm = Get-VM-Name $item.Name

$status =

$povm.Powerstate

write-hos t$povm Powerstate is: -ForegroundColor Yellow

$status

sleep 5

}until ($povm.Powerstate -eq “poweredoff”)

write-host $povm powered off successfully -ForegroundColor Green

}

## –#3– Set NTP Server##

write-host Remove old NTP server on $esxiserver -ForegroundColor Yellow

$oldNTP = Get-VMHostNtpServer -VMHost $esxiserver

Get-VMHost -Name $esxiserver | Remove-VMHostNtpServer -NtpServer $oldNTP -Confirm:$false

write-host Configuring NTP server and enable NTP service on $esxiserver -ForegroundColor Yellow

Get-VMhost -Name $esxiserver | Add-VMHostNtpServer -NtpServer $NTP1, $NTP2

Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”} | Set-VMHostService -Policy “on”

Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”} | Start-VMHostService

## –#4– –#5– Disable VMtools time sync completely##

$vmconfigspec = New-Object VMware.Vim.VirtualMachineConfigSpec

$value1 = New-Object VMware.Vim.OptionValue

$value1.Key = ‘tools.syncTime’

$value1.Value = 0

$vmconfigspec.ExtraConfig += $value1

$value2 = New-Object VMware.Vim.OptionValue

$value2.Key = ‘time.synchronize.continue’

$value2.Value = 0

$vmconfigspec.ExtraConfig += $value2

$value3 = New-Object VMware.Vim.OptionValue

$value3.Key = ‘time.synchronize.restore’

$value3.Value = 0

$vmconfigspec.ExtraConfig += $value3

$value4 = New-Object VMware.Vim.OptionValue

$value4.Key = ‘time.synchronize.resume.disk’

$value4.Value = 0

$vmconfigspec.ExtraConfig += $value4

$value5 = New-Object VMware.Vim.OptionValue

$value5.Key = ‘time.synchronize.shrink’

$value5.Value = 0

$vmconfigspec.ExtraConfig += $value5

$value6 = New-Object VMware.Vim.OptionValue

$value6.Key = ‘time.synchronize.tools.startup’

$value6.Value = 0

$vmconfigspec.ExtraConfig += $value6

$value7 = New-Object VMware.Vim.OptionValue

$value7.Key = ‘time.synchronize.tools.enable’

$value7.Value = 0

$vmconfigspec.ExtraConfig += $value7

$value8 = New-Object VMware.Vim.OptionValue

$value8.Key = ‘time.synchronize.resume.host’

$value8.Value = 0

$vmconfigspec.ExtraConfig += $value8

$allvmsonesx = Import-Csv C:\all_vms_$esxiserver.csv

foreach($item in $allvmsonesx){

$vmname = $item.Name

write-host Disable VMware Tools timesync for $vmname -ForegroundColor Green

$vm = Get-VM -Name $vmname

$vm.ExtensionData.ReconfigVM($vmconfigspec)

$spectools = New-Object VMware.Vim.VirtualMachineConfigSpec

$spectools.tools= New-Object VMware.Vim.ToolsConfigInfo

$spectools.tools.SyncTimeWithHost=$false

$vm = Get-view-id $vm.Id

$vm.ReconfigVM_task($spectools)

}

## –#6– Export adjusted advanced configuration for VMs##

foreach ($item in $allvmsonesx){

$shut_vm = $item.Name

write-host Exporting adjusted advanced configurationto CSV file for $shut_vm -ForegroundColor Green

“Virtual Machine is $shut_vm” | Out-File -Encoding Ascii -append C:\adjusted_advancedconfig_vms_on_$esxiserver.csv

Get-AdvancedSetting -Entity $shut_vm | where {$_.Name -like “time*”} | select name, value | Out-File -Encoding Ascii -append C:\adjusted_advancedconfig_vms_on_$esxiserver.csv

}

## –#7– Power on VMs again##

foreach ($item in $poweredononesx){

$shut_vm = $item.Name

write-host Power on $shut_vm -ForegroundColor Green

Start-vm -vm $shut_vm -Confirm:$false

}

## –#8– Exporting VMs with timesync with host enabled to CSV##

foreach ($item in $allvmsonesx){

$shut_vm = $item.Name

write-host Check timesync with host parameter for $shut_vm -ForegroundColor Green

get-view -viewtype virtualmachine -Filter @{‘Config.Tools.SyncTimeWithHost’=’True’} | Where-object {$_.Name-eq $shut_vm} |select name |Out-File-Encoding Ascii -append C:\vms_with_enabled_hostsync_in_$vcenterserver.csv|Out-Null

}

##Finishing Script##

write-host

write-host Script has finished for $esxiserver the NTP settings are valid, and all VMs on this host are compliant! -ForegroundColor Yellow

write-host

write-host Current NTP setting for $esxiserver

Get-VMHostNtpServer -VMHost $esxiserver | fl | Write-Output

Get-VMhost -Name $esxiserver | Get-VMHostService | Where-Object {$_.key -eq “ntpd”}

#####################################################################

If you have any questions, find it useful or have any other remarks on this blog post please let me know!

Enjoy!

Sources: vmware.com, communities.vmware.com

Video:

4 comments

M. Buijs - Be-Virtual.net says:

April 3, 2018 at 13:25

Hey Wesley,

Nice detailed blog post.

Got a question about it. I wrote a article about changing time synchronization from within the guest OS… Do you know if this is disabled by disabling the VMware Tools settings in the advanced configuration.

Link:
https://www.be-virtual.net/changing-guest-time-synchronization-setting-from-within-guest-os/

Thanks 🙂

Mischa

LikeLike

1. Wesley Geelhoed says:
  
  April 3, 2018 at 13:43
  
  Mischa,
  
  Thanks!
  
  No, this setting in the GUI differs from the advanced configuration settings. In my script I turned off both. But you can turn off time sync in event of an (Storage) vMotion, Snapshot, Power Cycle and leave the Time Sync with host enabled.
  
  The difference between the two is that the VMware tools sync settings which are in the advanced config only syncing when an event occurs and the ‘Time Sync with host’ option (if enabled) is providing a periodic time synchronization.
  
  Wesley
  
  LikeLiked by 1 person
  
  1. M. Buijs - Be-Virtual.net says:
    
    April 3, 2018 at 15:04
    
    Thanks for the clarification! 🙂
    
    LikeLike
Frans says:

April 4, 2018 at 13:11

Nice one Wes, just what I needed.

LikeLiked by 1 person