If you want to use an enterprise or third-party CA-signed certificate, you have to send a Certificate Signing Request (CSR) to the CA.

Use a CSR with these characteristics:
  • Key size: 2048 bits or more (PEM encoded)
  • PEM format. VMware supports PKCS8 and PKCS1 (RSA keys). When keys are added to VECS, they are converted to PKCS8.
  • x509 version 3
  • For root certificates, the CA extension must be set to true, and the cert sign must be in the list of requirements.
  • SubjectAltName must contain DNS Name=<machine_FQDN>.
  • CRT format
  • Contains the following Key Usages: Digital Signature, Non Repudiation, Key Encipherment
  • Start time of one day before the current time.
  • CN (and SubjectAltName) set to the host name (or IP address) that the ESXi host has in the vCenter Server inventory.

For information about generating the CSR, see the VMware knowledge base article at https://kb.vmware.com/s/article/2113926.