NTP Compliancy: disable NTP sync with host completely (PowerCLI)
Last week I was identifying a root cause for a P1 outage my customer had suffered from. This customer has a business-critical application running 24/7 which is very important for all activities within the company. They had suffered an outage for at least three times last month, and therefore they requested my employer ITQ to perform a health check including a root cause analysis for the latest P1 incidents.
It will not be a big surprise if I told you that the root cause was NTP? Although incorrectly configured NTP settings often lead to issues, unfortunately this is not yet configured consistently and well enough in a lot of environments.
The issues that the customer was facing was as follows. The ESXi hypervisor was pointing to an NTP server which was not existing anymore. So the time between the hypervisor and the domain controller was completely out of sync. After a vMotion of a VM with Server 2012 R2 as Guest OS (domain member), the VMware tools inside the Guest OS initiated a sync with the hypervisor what resulted in a 5 minutes time difference. Since the business-critical application was very time-sensitive, this lead into a service disruption within the application stack. The customer said to me; but the ‘Synchronize guest time with host’ is disabled on every VM in the environment, so is this a bug? The answer is No! and it is well explained in this VMware Blogpost.
I know there already a few blog post and articles regarding this topic, but I had a specific use case for the script I’ve written. And wanted to share it. Maybe someone will ever come into the same situation, in which this script can be very useful.
The script has been written to execute tasks on a per-host basis, since this was a requirement stated by the customer. Of course it can be easily rewritten to a per-cluster basis script. In this specific use-case environment DRS was disabled, the business-critical application which consist of 48 clustered VMs is not supported on a DRS enabled cluster.
What does the script do?
#1 – Export all the existing VMs on the ESXi host to a CSV file. Including a separate CSV file for the powered on VMs only. #2 – Power down (Shutdown Guest OS) all the VMs on the ESXi host, otherwise certain advanced configuration settings for the VMs cannot be set. A fail-safe has been built-in in case a VM doesn’t have VMware tools installed. Then the hypervisor can’t shut down the VM correctly. The script will not continue until you manual turn off the remaining VMs. #3 – Setting the appropriate NTP servers on the ESXi host including the ‘Start and stop with host’ Policy. #4 – Disable the VMware tools time sync for vMotion, Snapshots and so on.. by setting the advanced settings. (can only be done on a powered off VM) #5 – Set the VMware tools time sync with host option per VM to $false #6 – Export the advanced config for all the adjusted VMs to an CSV file #7 – Power on all the VMs which were running when starting this script, all the powered off VMs will not be started. #8 – Performing a latest check on the ‘VMware Tools timesync with host option’. If everything in the script executed without any errors the export file: vms_with_enabled_hostsync.csv should be empty.
Nice detailed blog post.
Got a question about it. I wrote a article about changing time synchronization from within the guest OS… Do you know if this is disabled by disabling the VMware Tools settings in the advanced configuration.
No, this setting in the GUI differs from the advanced configuration settings. In my script I turned off both. But you can turn off time sync in event of an (Storage) vMotion, Snapshot, Power Cycle and leave the Time Sync with host enabled.
The difference between the two is that the VMware tools sync settings which are in the advanced config only syncing when an event occurs and the ‘Time Sync with host’ option (if enabled) is providing a periodic time synchronization.
LikeLiked by 1 person
Thanks for the clarification! 🙂
Nice one Wes, just what I needed.
LikeLiked by 1 person