ESXi Certificate Management for VMware Cloud Foundation

Because you cannot replace ESXi host certificates in SDDC Manager, you perform this operation manually on each ESXi host or in automated way across a workload domain by using Windows Powershell commands.

SDDC Manager does not manage certificates for ESXi hosts. Instead, the VMware Certificate Authority (VMCA) on vCenter Server provisions each new ESXi host with a signed certificate where VMCA is the root certificate authority (CA) by default. To comply with the policy of your organization, you must manually replace the host’s certificate.

You can replace host certificates step-by-step by using product user interface or in an automated way, as code-based alternatives to certain UI-based procedures, by calling the cmdlets in the VMware.CloudFoundation.CertificateManagement module in PowerShell Gallery.

If you want to read the documentation, provide feedback, report an issue with automation, or contribute to the VMware.CloudFoundation.CertificateManagement module, go to the VMware.CloudFoundation.CertificateManagement open-source project in Github.

Content Scope

The ESXi certificate management guidance in this section covers following scenarios:

• Replacing an ESXi host certificate signed by VMCA with a certificate signed by an external certificate authority in an already deployed SDDC workload domain.

• Replacing an ESXi host certificate signed by an external certificate authority with another certificate signed by an external certificate authority in an already deployed SDDC workload domain.

Prerequisites

To perform the configuration associated with ESXi certificate management, verify that your system fulfills the following prerequisites.

Category	Prerequisite
Environment	Verify that your VMware Cloud Foundation instance is healthy and fully operational.
Infrastructure-as-code	To use the infrastructure-as-code method for managing ESXi host certificates, verify that your system fulfills the prerequisites, described in the documentation of the VMware.CloudFoundation.CertificateManagement open-source project in Github.

Change the Certificate Mode to Custom for the ESXi Hosts in a Workload Domain

You change certificate mode for the hosts in a workload domain in VMware Cloud Foundation to custom so that VMCA no longer automatically provisions the ESXi hosts with VMCA certificates.

Prerequisites

• The workload domain vCenter Server must be in a scheduled maintenance window. You must restart vCenter Server service to finish the process.

UI Procedure

1. Log in to the workload domain vCenter Server that manages the target hosts at https://<vcenter-server_fqdn>/ui as [email protected].

2. Change the certificate mode for the managed ESXi hosts.

   • In the Hosts and Clusters inventory, select the vCenter Server instance.

   • On the Configure tab, under Settings , click Advanced Settings and click Edit Settings.

   • In the Name filter text box, enter vpxd.certmgmt.mode.

   • Change the value of vpxd.certmgmt.mode to custom, and click Save.

3. Restart the vCenter Server instance for apply the changes.

   1. From the vSphere Client Menu, select Administration.

   2. In the Deployment section, select System configuration.

   3. Select the vCenter Server instance from the list and click Reboot node.

   4. Enter Change ESXi certificate mode to custom CA mode. as a reason for the restart and click Reboot.

4. Verify that the vCenter Server instance has successfully restarted and that you are able to log in to the vSphere Client.

5. Acknowledge the vSphere vCenter Host Certificate Management Mode alarm.

6. In the vSphere Client, verify that for the workload domain vCenter Server the value of the vpxd.certmgmt.mode property is custom.

PowerShell Procedure

1. Start PowerShell.

2. Replace the values in the sample code and run the commands in the PowerShell console.

   $sddcManagerFqdn = "sfo-vcf01.sfo.rainpole.io" 
   $sddcManagerUser = "[email protected]" 
   $sddcManagerPass = "VMw@re1!"
   $workloadDomain = "sfo-m01"
   $mode = "custom"

3. Set the ESXi certificate mode to custom by running the following command in the PowerShell console.

   Set-EsxiCertificateMode -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain -mode $mode

4. Restart the vCenter Server instance manually to apply the changes as described in the UI-assisted procedure.

5. Retrieve and verify the ESXi certificate mode has been set to custom by entering the following command.

   Get-EsxiCertificateMode -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain

Generate a Certificate Signing Request for the ESXi Hosts in a Workload Domain

By using the ESXi Shell, you generate Certificate Signing Request (CSR) files for each ESXi host in the workload domain. You can send the CSR files to a third-party certificate authority and receive CA-signed certificates for the hosts.

The certificate chain of the third-party certificate authority must be present in the trust store of SDDC Manager and the workload domain vCenter Server.

UI Procedure

1. Log in to the ESXi Shell by using the DCUI or an SSH client as a user with administrator privileges.

2. In the /etc/vmware/ssl directory, create a file by running the following command.

   vi ESXi.cfg

3. Place the following content in the ESXi.cfg file

   [ req ]
   days = certificate-expiration-period (730)
   default_md = sha512
   default_bits = 2048
   distinguished_name = req_distinguished_name
   prompt = no
   [ req_distinguished_name ]
   commonName = esxi-fqdn
   countryName = your-country
   stateOrProvinceName = your-state
   localityName = your-locality
   0.organizationName = your-organization
   organizationalUnitName = your-unit

4. Run the following command to generate a private key and a CSR file, replacing the values in the example command.

   openssl req -new -nodes -out esxi-fqdn.csr -keyout esxi-fqdn.key -config ESXi.cfg

5. Export the esxi-hostname.csr file and optionally esxi-hostname.key from the ESXi host, for example, by using an SCP utility, such as WinSCP.

6. Repeat the steps for the remaining ESXi hosts in the workload domain.

PowerShell Procedure

1. Start PowerShell.

2. Replace the values in the sample code and run the commands in the PowerShell console.

   $sddcManagerFqdn = "sfo-vcf01.sfo.rainpole.io" 
   $sddcManagerUser = "[email protected]" 
   $sddcManagerPass = "VMw@re1!"
   $workloadDomain = "sfo-m01"
   $cluster = "sfo-m01-cl01"
   $country = "US"
   $locality = "San Francisco"
   $organization = "Rainpole"
   $organizationUnit = "IT"
   $stateOrProvince = "California"
   $outputDirectory = "F:\CSR\"

3. Perform the configuration by entering the following command in the PowerShell console.

   Request-VCFCsr -esxi -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain -cluster $cluster -Country $country -Locality $locality -Organization $organization -OrganizationUnit $organizationUnit -StateOrProvince $stateOrProvince -outputDirectory $outputDirectory

4. Repeat the steps for each cluster in the workload domain.

What to do next

Send the CSR files to the third-party certificate authority to receive the CA-signed host certificates.

Verify That the Certificate Authority Is Trusted by vCenter Server and SDDC Manager

For a trusted communication, the certificate authority that signed the certificates of the ESXi hosts must be in the trust store of by both the workload domain vCenter Server and SDDC Manager.

UI Procedure

1. Verify that the certificate of the CA is in the trust store of SDDC Manager.

   1. Log in to SDDC Manager at https://<sddc_manager_fqdn> with a user assigned the Admin role.
   2. In the navigation pane, click Inventory > Workload Domains.

   3. On the Workload Domains page, click the target workload domain.

   4. On the workload domain summary page, click the Certificates tab.

   5. Verify the certificate issuer for the vcenter resource type is the same as the issuer of the CA-signed host certificates.

2. Verify that the certificate of the CA is in the trust store of the workload domain vCenter Server.

   1. Log in to vCenter Server at https://<vcenter_server_fqdn>/ui as [email protected].

   2. From the vSphere Client Menu, select Administration.

   3. Under Certificates, select Certificate Management.

   4. On the upper right hand corner, select the vCenter Server instance from the drop-down menu.

   5. Verify that the Trusted Root Certificates store contains the certificate of the certificate authority with the same issuer name as the host certificates.

   6. Verify that the subject and serial number are the same as in the issuer part of the signed host certificates.

PowerShell Procedure

1. Start PowerShell.

2. Replace the values in the sample code and run the commands in the PowerShell console.

   $sddcManagerFqdn = "sfo-vcf01.sfo.rainpole.io"
   $sddcManagerUser = "[email protected]"
   $sddcManagerPass = "VMw@re1!"
   $workloadDomain = "sfo-m01"
   $rootCertificate = "F:\Certificate\rainpoleRoot64.cer"
   $issuer = "rainpole"

3. Verify if the CA's certificate is trusted by running the following command.

   Confirm-CAInvCenterServer -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain -issuer $issuer -signedCertificate $rootCertificate

4. Verify that the command output returns true.

Replace the ESXi Host Certificates with CA-Signed Certificate in a Workload Domain

Replace each ESXi host certificate with a CA-signed certificate obtained from a certificate authority to meet the security requirements of your organization.

Note:

Perform the replacement of the host certificates in the workload domain in a scheduled maintenance window. This operation requires more time to complete than is usually allocated in a regular operational mode.

UI Procedures

1. Log in to the workload domain vCenter Server that manages the target hosts at https://<vcenter-server_fqdn>/ui as [email protected].

2. If you are using vSAN as principal storage, verify the vSAN health.

   1. In the Hosts and Clusters inventory, select the cluster that contains the ESXi hosts and click the Monitor tab.

   2. In the left pane, under vSAN > Skyline health, click Retest and confirm the operation.

   3. Verify no red alerts appear in the Overview pane.

   4. In the left pane, under vSAN > Resyncing objects, verify that all synchronization tasks are complete.

3. Place the ESXi host in maintenance mode.

   1. In the Hosts and Clusters inventory, expand the workload domain vCenter Server and navigate to the first ESXi host in the first cluster.

   2. Right click on the ESXi host and select Maintenance Mode > Enter Maintenance mode.

   3. In the Enter maintenance mode dialog box, from the vSAN data migration drop-down menu, select Full data migration, and click OK.

4. Deactivate ESXi lockdown mode.

   1. Click the Configure tab for the ESXI host.

   2. Under System, select Security Profile.

   3. In the Lockdown Mode pane, click Edit.

   4. In the Lockdown Mode dialog box, on the Lockdown Mode page, select Disabled.

   5. Click OK.

5. Disconnect the ESXi host from vCenter Server.

   1. In the Hosts and Clusters inventory, right-click the ESXi host and select Connection > Disconnect.

   2. In confirmation dialog box, click OK.

6. Back up the current ESXi host certificate.

   1. Log in to the ESXi Shell by using the DCUI or an SSH client as a user with administrator privileges.

   2. In the /etc/vmware/ssl directory, rename the existing certificates by running the following commands.

      cp rui.crt esxi-hostname-orig.rui.crt
      cp rui.key esxi-hostname-orig.rui.key

   3. Export the esxi-hostname-orig.rui.crt and esxi-hostname-orig.rui.key files from the ESXi host, for example, by using an SCP utility, such as WinSCP.

7. Replace the ESXi host certificate.

   1. Copy the esxi-hostname.cer and esxi-hostname.key certificate files, received from the certificate authority, to the /etc/vmware/ssl directory on the hosts, for example, by using an SCP utility, such as WinSCP.

   2. In the ESXi Shell, in the /etc/vmware/ssl folder, rename both esxi-hostname.cer and esxi-hostname.key to rui.crt and rui.key, respectively, by running the following commands.

      Confirm any file replacement.

      mv /etc/vmware/ssl/esxi-hostname.cer rui.crt
      mv /etc/vmware/ssl/esxi-hostname.key rui.key

   3. Restart the ESXi host by running the following command.

      reboot

8. In the vSphere Client, in the Hosts and Clusters inventory, right-click the ESXi host and select Connection > Connect.

9. Verify the ESXi host certificate has been replaced successfully after the restart.

   1. Click the Configure tab for the ESXI host.

   2. Under System, select Certificate.

   3. Verify the issuer and valid dates of the certificate match to CA-signed certificate.

   4. Verify that the Status is Good.

10. If you are using vSAN as principal storage, verify the vSAN health.

11. Verify ESXi host connectivity to SDDC Manager.

    1. Log in to SDDC Manager at https://<sddc_manager_fqdn> with a user assigned the Admin role.

    2. In the navigation pane, select Inventory > Hosts.

    3. Verify the Configuration Status of the host is Active.

12. Verify ESXi host connectivity to the NSX Manager instance for the workload domain.

    1. In the NSX Manager UI at https://<nsx_manager_fqdn>, click the System tab.

    2. In the navigation pane, select Fabric > Hosts.

    3. On the Clusters tab, verify that NSX configuration is Success and Node Status is Up for the ESXi host.

13. In the vSphere Client, reactivate the lockdown mode of the ESXi host.

14. Exit the maintenance mode of the ESXi host.

    1. In the vSphere Client, expand the workload domain vCenter Server and navigate to the ESXi host.

    2. Right-click the ESXi host and select Maintenance Mode > Exit Maintenance mode.

15. Repeat the procedure for the remaining ESXi hosts in the cluster.

16. Repeat the procedure for the hosts in the remaining clusters in the workload domain.

PowerShell Procedure

1. Start PowerShell.

2. Replace the values in the sample code and run the commands in the PowerShell console.

   $sddcManagerFqdn = "sfo-vcf01.sfo.rainpole.io" 
   $sddcManagerUser = "[email protected]" 
   $sddcManagerPass = "VMw@re1!"
   $workloadDomain = "sfo-m01"
   $cluster = "sfo-m01-cl01" 
   $certificateDirectory = "F:\Certificate"
   $certificateFileExt = ".cer"

3. If you are using vSAN as principal storage, to verify the vSAN health status, run the following command and verify that the output contains no red alerts.

   Get-vSANHealthSummary -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain -cluster $cluster 

4. Back up the current certificates of the hosts in the cluster by using the ESXi Shell.

   Caution:

   The Install-VCFCertificate command that you run to replace the host certificates does not back up the original ESXi certificate and private key.

5. Replace the ESXi host certificate by running the following command.

   Install-VCFCertificate -esxi -server $sddcManagerFqdn -user $sddcManagerUser -pass $sddcManagerPass -domain $workloadDomain -cluster $cluster -certificateDirectory $certificateDirectory  -certificateFileExt $certificateFileExt

   The command output shows successfully completed certificate replacement.

6. If you are using vSAN as principal storage, verify the vSAN health status again.

7. Repeat the procedure for the remaining clusters in the workload domain.

What to do next

Because the ESXi certificate mode is set to custom, before adding ESXi hosts to the workload domain, replace their certificates with CA-signed ones.

Restore the Certificate of an ESXi Host in a Workload Domain from a Backup

If you run into issues during ESXi certificate replacement, you can restore the original certificate setup of a host from the backup you took before the replacement operation.

Procedure

1. Log in to the workload domain vCenter Server that manages the target host at https://<vcenter-server_fqdn>/ui as [email protected].

2. Place the ESXi host in maintenance mode.

   1. In the Hosts and Clusters inventory, expand the workload domain vCenter Server and navigate to the first ESXi host in the first cluster.

   2. Right click on the ESXi host and select Maintenance mode > Enter maintenance mode.

   3. For a vSAN cluster, in the Enter maintenance mode dialog box, from the vSAN data migration drop-down menu, select Full data migration, and click OK.

3. Deactivate ESXi Lockdown mode.

   1. Click the Configure tab for the ESXI host.

   2. Under System, select Security Profile.

   3. In the Lockdown Mode pane, click Edit.

   4. In the Lockdown Mode dialog box, on the Lockdown Mode page, select Disabled.

   5. Click OK.

4. Deactivate ESXi lockdown mode by using the VMware Host Client.

   The ESXi host might be disconnected from the workload domain vCenter Server because of authentication issues caused by the unsuccessful certificate replacement on the host.

   1. Log in to the ESXi host at https://<esxi_host_fqdn>/ui as root.

   2. In the navigation pane, select Manage > Security & users.

   3. Select Lockdown Mode and click Edit settings.

   4. Select Disabled and click Change.

5. Disconnect the ESXi host from vCenter Server.

   1. In the vSphere Client, in the Hosts and Clusters inventory, right-click the ESXi host and select Connection > Disconnect.

   2. In confirmation dialog box, click OK.

6. Restore the original ESXi host certificate.

   1. Copy the original certificate files esxi-hostname-orig.rui.crt and esxi-hostname-orig.rui.key to the /etc/vmware/ssl directory on the host, for example, by using an SCP utility, such as WinSCP..

   2. Log in to the ESXi Shell, using either the DCUI or an SSH client, as a user with administrator privileges.

   3. In the /etc/vmware/ssl directory, rename esxi-hostname-orig.rui.crt and esxi-hostname-orig.rui.key to rui.crt and rui.key, respectively, by running the following commands.

      mv /etc/vmware/ssl/esxi-hostname-orig.rui.crt rui.crt
      mv /etc/vmware/ssl/esxi-hostname-orig.rui.key rui.key

   4. Restart the ESXi host by running the following command.

      reboot

7. In the vSphere Client, in the Hosts and Clusters inventory, right-click the ESXi host and select Connection > Connect.

8. Verify the ESXi host certificate has been replaced successfully after the restart.

   1. Click the Configure tab for the ESXI host.

   2. Under System, select Certificate.

   3. Verify the issuer and valid dates of the certificate match to CA-signed certificate.

   4. Verify that the Status is Good.

9. If you are using vSAN as principal storage, verify the vSAN health.

   1. In the Hosts and Clusters inventory, select the cluster that contains the ESXi hosts and click the Monitor tab.

   2. In the left pane, under vSAN > Skyline health, click Retest and confirm the operation.

   3. Verify no red alerts appear in the Overview pane.

   4. In the left pane, under vSAN > Resyncing objects, verify that all synchronization tasks are complete.

10. Verify ESXi host connectivity to SDDC Manager.

    1. Log in to SDDC Manager at https://<sddc_manager_fqdn> with a user assigned the Admin role.

    2. In the navigation pane, select Inventory > Hosts.

    3. Verify the Configuration Status of the host is Active.

11. Verify ESXi host connectivity to the NSX Manager instance for the workload domain.

    1. In the NSX Manager UI at https://<nsx_manager_fqdn>, click the System tab.

    2. In the navigation pane, select Fabric > Hosts.

    3. On the Clusters tab, verify that NSX configuration is Success and Node Status is Up for the ESXi host.

12. Exit the maintenance mode of the ESXi host.

    1. In the vSphere Client, expand the workload domain vCenter Server and navigate to the ESXi host.

    2. Right-click the ESXi host and select Maintenance mode > Exit maintenance mode.