Note: The preferred method for rotating the Services TLS CA and its leaf certificates in Ops Manager v2.9 and later is to use CredHub Maestro. If all of the service tiles in your deployment are compatible with Maestro, then use the procedure documented in Advanced Certificate Rotation with CredHub Maestro instead of this procedure. To determine whether your tiles are compatible with Maestro, see CredHub Maestro Tile Compatibility.
This topic provides an overview of the Services TLS Certificate Authority (CA) and a procedure for rotating the Services TLS CA certificate and its leaf certificates.
In a VMware Tanzu Operations Manager (Ops Manager) and VMware Tanzu Application Service for VMs (TAS for VMs) deployment, the Services TLS CA signs the certificate that service brokers use to sign leaf certificates for TLS-enabled service instances.
Service instances include instances created by on-demand services such as MySQL for VMware Tanzu, VMware Tanzu GemFire, Redis for VMware Tanzu, and RabbitMQ for VMware Tanzu [VMs]. You may have one or more of these service tiles installed in your deployment. All deployed service tiles in a foundation use the same Services TLS CA certificate, which is stored in BOSH CredHub with the name /services/tls_ca.
When an operator creates an on-demand TLS enabled service instance or updates an existing non-TLS on-demand service instance to use TLS, the following takes place:
BOSH generates a leaf certificate signed by the Services TLS CA.
BOSH then stores this new leaf certificate in BOSH CredHub with the format /p-bosh/service-instance_SERVICE-GUID/unique-name-per-service-type.
In the following procedure, BOSH rotates leaf certificates when you select the upgrade all service instances errand for an on-demand service tile and apply changes in Ops Manager.
Before you begin this rotation procedure, ensure that you have the following:
VCAP_SERVICES for TLS, you will experience app downtime if you perform the certificate rotation described in this procedure. To avoid app downtime, follow the procedure in this Knowledgebase Article instead of the procedure in this topic. Only follow the procedure in this topic if all of your MySQL apps use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt. This section provides the procedure to rotate the Services TLS CA certificate and update all its leaf certificates.
Before starting the rotation, check that the Services TLS CA certificate has not already expired.
To check the expiration date:
Log in to CredHub by following the procedure in Accessing BOSH CredHub with the CredHub CLI.
Run:
credhub get -n /services/tls_ca -j | jq -r .value.ca \
| openssl x509 -text -noout | grep -A 2 "Validity"
For example:
Validity
Not Before: Jan 23 19:04:58 2019 GMT
Not After : Jan 23 19:04:58 2020 GMT
The above command retrieves the certificate from CredHub, decrypts it, parses its JSON structure, and finds its Validity properties. The expiration date for the certificate is the value of Not After.
To obtain or generate a new Services TLS CA certificate:
Check if CredHub has a new temporary certificate from a previous rotation attempt. Run:
credhub get -n /services/new_caNote: This procedure assumes that /services/new_ca is the CredHub location used when generating the temporary certificate. If you used a different location, specify that location instead.
If an older temporary certificate already exists, delete that certificate by running:
credhub delete -n /services/new_caDo one of the following:
If you use self-signed certificates: Generate a new self-signed certificate with a new name or path in CredHub:
credhub generate \
--name="/services/new_ca" \
--type="certificate" \
--no-overwrite \
--is-ca \
--duration=1825 \
--common-name="opsmgr-services-tls-ca"
Where:
name is the CredHub path or name of the new certificate. You can use /services/new_ca or substitute a different name as long as you use the value consistently in the rest of the procedure.common-name is the common name of the generated certificate. You can use opsmgr-services-tls-ca or substitute a different common name as long as you use the value consistently in the rest of the procedure.duration is set to 1825, which is a recommended value of five years. If you do not specify a duration, the default value is 365 or one year.If you use an intermediate certificate signed by a root CA in CredHub: Generate a new certificate signed by the CredHub root CA by running:
credhub generate \
--name="/services/new_ca" \
--type="certificate" \
--no-overwrite \
--is-ca \
--duration=1825 \
--common-name="opsmgr-services-tls-ca" \
--ca=/PATH-TO-ROOT-CA
Where:
name is the CredHub path or name of the new certificate. You can use /services/new_ca or substitute a different name as long as you use the value consistently in the rest of the procedure.common-name is the common name of the generated certificate. You can use opsmgr-services-tls-ca or substitute a different common name as long as you use the value consistently in the rest of the procedure.duration is set to 1825, which is a recommended value of five years. If you do not specify a duration, the default value is 365 or one year.ca is the path to the root CA for the new certificate. Replace PATH-TO-ROOT-CA with the appropriate path for the root CA.To add your new and current Services TLS CA certificate to Ops Manager:
Log in to CredHub.
Get the current Services TLS CA certificate by running:
credhub get --name=/services/tls_ca -k ca
Get the new certificate, either from a pre-existing file, or from your new CredHub location by running:
credhub get --name=/services/new_ca -k ca
Caution: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.
To apply these changes:
Verify which on-demand service tiles are using the Services TLS CA certificate. Run:
credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca
Note: The name of the upgrade all service instances errand may differ slightly between services.
To set the new Services TLS CA certificate:
Do one of the following:
If you have an existing certificate: Obtain the CA certificate and private key file corresponding to the CA that you applied in the previous step. Then, run:
credhub set \
--name="/services/tls_ca" \
--type="certificate" \
--certificate=PEM-PATH/root.pem \
--private=CERT-KEY
Where:
certificate is the location of the root.pem file for the certificate. Replace PEM-PATH with the path of your certificate’s root.pem file.private is the private key for the new certificate. Replace CERT-KEY with the value of the private key for your certificate.If you have created a new self-signed or intermediary certificate: Set the /services/new_ca that you generated in the previous step as the Services TLS CA certificate:
credhub get -n /services/new_ca -k ca > new_ca.ca
credhub get -n /services/new_ca -k certificate > new_ca.certificate
credhub get -n /services/new_ca -k private_key > new_ca.private_key
credhub set -n /services/tls_ca \
--type=certificate \
--root=new_ca.ca \
--certificate=new_ca.certificate \
--private=new_ca.private_key
Caution: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.
In this step, you apply changes to ensure that all on-demand service instances generate new leaf certificates from the new Services TLS CA certificate.
To apply these changes:
Note: The name of the upgrade all service instances errand may differ slightly between services.
After your apps have reconnected to service instances with new certificates, remove the old CA certificate:
Navigate to the Ops Manager Installation Dashboard in Ops Manager and click the BOSH Director tile.
Click Security.
Delete the old CA certificate from the Trusted Certificates field.
Click Save.
Navigate to the Ops Manager Installation Dashboard in Ops Manager and click the TAS for VMs tile.
Click Networking.
Delete the old CA certificate in the Certificate Authorities trusted by the Gorouter and Certificate Authorities trusted by the HAProxy fields.
Click Save.
As a security best practice, you should remove outdated certificates as soon as possible from your deployment. You can schedule this step at a convenient time. Most deployments will not lose any functionality if you do not perform this step immediately.
For some deployments, you may encounter an error if you create a service instance in a deployment that contains an expired Services TLS CA certificate. For more information, see this Knowledgebase Article.
Caution: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.
To apply these changes:
Note: The names of upgrade all service instances errand may differ slightly between services.
