If you want to use a third-party CA-signed certificate, either with VMCA as a subordinate authority or with a custom certificate authority, you have to send a Certificate Signing Request (CSR) to the CA.
Use a CSR with these characteristics:
- 2048 bits
- PKCS1
- No wildcards
- Start time of one day before the current time
- CN (and SubjectAltName) set to the host name (or IP address) that the ESXi host has in the vCenter Server inventory.