Issue with replacing certificate for a new Load Balanced PSC setup (vSphere 6.5 & NSX 6.4)

Last week I ran into an issue while I was configuring a Load Balanced Platform Service Controller setup. My initial configuration was as follows: VCSA01 is pointed to the PSC01, and the VCSA02 is pointed to the PSC02 (within the same SSO domain). My goal was to load balance my existing Platform Service Controllers via a VIP on a NSX Load Balancer for the two vCenters and one NSX Manager (Connected to VCSA01).

Since my already existing FQDNs for the PSCs were PSC01.wesleygeelhoed.local en PSC02.wesleygeelhoed.local i chose psc.wesleygeelhoed.local as the VIP FQDN. My PSCs have custom certificates installed from my internal PKI (Microsoft CA) and don’t have the psc.wesleygeelhoed.local FQDN as Subject Alternative Name in the certificate yet. So I had to replace the certificates. After generating the CSR, issuing the certificate I was ready to replace them on both of my PSCs. During the replacement of my former custom certificate the VMCA within the PSC02 generated the following error: Previous MACHINE_SSL_CERT Subject Alternative Name does not match new MACHINE_SSL_CERTIFICATE Subject Alternative Name. According to the following VMware knowledge base article: KB2150267 there is no way replace the SSL Machine certificates for new certs with different common name. In the past I deployed a faulty certificate to this PSC02 and I’m not able to recover from that (don’t have that issue on my PSC01 which is deployed with the right common name in the certificate). The only thing I need to do for my PSC01 is regenerating the certificates via the VMCA (explained later in this post) and provide multiple Subject Alternative Names.

In order to get my PSC02 working in the load balanced setup I had to redeploy it. The steps that need to be taken are as following:

Setup the NSX load balancer (explained in detail by my ITQ co-worker Pascal van de Bor)
Repoint the VCSA01, VCSA02 to the PSC02
Replace the certificate for the PSC01 with the VIP FQDN included as Subject Alternative Name in the certificate
Configure PSC01 appliance for High Availability
Repoint the VCSA01, VCSA02 and the NSX Manager to the PSC VIP FQDN on the NSX Load Balancer
Decommision PSC02
Redeploy PSC02
Generate custom certificate for PSC02 and configure HA.
Test Load Balancing

*at this moment I already have my NSX load balancer configuration in place!

PSC01
Prior to the certificate replacement I repointed both of my VCSAs to the PSC02 platform service controller by running this command on VCSA01 and VCSA02: “cmsso-util repoint –repoint-psc psc02.wesleygeelhoed.local”

Then logged on with root on the PSC01 (Putty/SSH) and typed ”shell” to gain the appropriate access. To be able to use winSCP to remote connect to the PSC, I executed the following command “chsh -s /bin/bash root”. Now I started the VMware Certificate Manager by typing: “/usr/lib/vmware-vmca/bin/certificate-manager”. Chose option 1 to replace the machine SSL certificate for a custom one. Filled in the administrator account and password and again chose option 1 to generate the CSR. Filled in the required parameters:

Output Directory Path: /tmp
Country: NL
Name: psc01.wesleygeelhoed.local
Organization: IT
Org. Unit: VMW
State: ZH
Locality: AADR
Ipaddress: *BLANK*
email: infoatwesleygeelhoeddotnl
hostname (FQDN): psc01.wesleygeelhoed.local, psc.wesleygeelhoed.local (Filled in all of the required Subject Alternative Names here for the certifcate)
VMCA name: WGE CA

The vmca_issued_csr.csr and the vmca_issued_key.key are placed in the /tmp folder on the PSC. I used winSCP to connect via SCP (port 22) to the platform service controller. Now I was able to extract the CSR and request the certificate at my internal Microsoft CA and uploaded the generated certificate along with Root certificate to the /tmp folder on the PSC. Back in the putty session I chose 1 to continue importing the custom certificates.

CSR generated at: /tmp/vmca_issued_csr.csr
1. Continue to importing Custom certificate(s) and key(s) for Machine SSL certificate

2. Exit certificate-manager

Option [1 or 2]: 1

Please provide valid custom certificate for Machine SSL.
File : /tmp/pnew.cer

Please provide valid custom key for Machine SSL.
File : /tmp/vmca_issued_key.key

Please provide the signing certificate of the Machine SSL certificate
File : /tmp/Root.cer

You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? :Y

After this process was completed, the certificate was replaced successfully. Now I need to configure the PSC for High Availability. Browsed to the VMware SSO bin directory with this command: “cd /usr/lib/vmware-sso/bin” and executed the following command “python updateSSOConfig.py –lb-fqdn=psc.wesleygeelhoed.local” to enable HA for the PSC01. We also need to update the Service Registration Endpoints in the VMdirectory, this is done by running this command only once on one platform service controller: “python UpdateLsEndpoint.py –lb-fqdn=psc.wesleygeelhoed.local –user=administrator@vwees.local”

After completion of the above tasks I disabled the PSC02 in the NSX Edge Load Balancing config (NSX Edge-> Load Balancer-> Pools-> PSC_POOL-> Edit -> Disable PSC02 Member). And repointed both of my vCenter Server Appliance to the Load Balanced FQDN PSC VIP: “cmsso-util repoint –repoint-psc psc.wesleygeelhoed.local“. This command was executed from both VCSAs.

Tested successfully the SSO access to VCSA01 and VCSA02 via the vSphere Web Client and logged on to the NSX Manager Virtual Appliance WebGUI management (https://nsx-manager01.wesleygeelhoed.local/login.jsp) to repoint the SSO lookup URL. I did go to Manage vCenter Registration and hit Edit on the Lookup Service URL.

Provided the proper Load Balanced PSC VIP FQDN with the SSO administrator password. And hit OK.

As you can see the NSX Manager is now connecting to the platform service controller(s) via a NSX load balancer.

PSC02
Now it’s time to decommission the existing PSC02 since the both VCSAs and the NSX Manager are connected on the Load Balanced VIP FQDN already. I shutdown the PSC02 VM and logon to the PSC01 as root. To unregister the POWERED OFF Platform Service Controller run the following command:

cmsso-util unregister –node-pnid psc02.wesleygeelhoed.local –username administrator@vwees.local

Now I can safely remove the old PSC02 appliance and redeploy it. It will obviously do no harm to double check if the PSC is removed from the system configuration. Easiest way (in my opinion) to do this is go to Administration -> System Configuration in the vSphere Web Client and check if the node is gone.

During the re-deployment of the PSC02 I chose the same IP/DNS/NTP/SYSLOG addresses and pointed to PSC01 to be able to choose the same SSO domain. Now we can proceed with the custom certificate for PSC02, for detailed prerequisites please follow this feature walk through from VMware.

Then logged on with root on the PSC02 (Putty/SSH) and typed ”shell” to gain the appropriate access. To be able to use winSCP to remote connect to the PSC, I executed the following command “chsh -s /bin/bash root”. Now I started the VMware Certificate Manager by typing: “/usr/lib/vmware-vmca/bin/certificate-manager”. Chose option 1 to replace the machine SSL certificate for a custom one. Filled in the administrator account and password and again chose option 1 to generate the CSR. Filled in the required parameters:

Output Directory Path: /tmp
Country: NL
Name: psc02.wesleygeelhoed.local
Organization: IT
Org. Unit: VMW
State: ZH
Locality: AADR
Ipaddress: *BLANK*
email: infoatwesleygeelhoeddotnl
hostname (FQDN): psc02.wesleygeelhoed.local, psc.wesleygeelhoed.local (Filled in all of the required Subject Alternative Names here for the certifcate)
VMCA name: WGE CA

CSR generated at: /tmp/vmca_issued_csr.csr
1. Continue to importing Custom certificate(s) and key(s) for Machine SSL certificate

2. Exit certificate-manager

Option [1 or 2]: 1

Please provide valid custom certificate for Machine SSL.
File : /tmp/pnew.cer

Please provide valid custom key for Machine SSL.
File : /tmp/vmca_issued_key.key

Please provide the signing certificate of the Machine SSL certificate
File : /tmp/Root.cer

You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? :Y

After this process was completed, the certificate was replaced successfully. The only thing that was left to do was enable the PSC for High Availability. Browsed to the VMware SSO bin directory with this command: “cd /usr/lib/vmware-sso/bin” and executed the following command “python updateSSOConfig.py –lb-fqdn=psc.wesleygeelhoed.local” to enable the PSC02 for HA.

Now the PSC02 is ready for production. In the NSX load balancer configuration I enabled the PSC02 again (NSX Edge-> Load Balancer-> Pools-> PSC_POOL-> Edit -> Enable PSC02 Member). And tested the load balancing with wget for windows. As you can see in the printscreen below, the connections are evenly load balanced between the two Platform Service Controllers by the ROUND-ROBIN algorithm.