Advanced certificate rotation with CredHub Maestro

You can rotate certificates and certificate authorities (CAs) in BOSH CredHub using CredHub Maestro. This topic only covers rotating certificates that are managed by CredHub.

For more information about setting up and using CredHub Maestro, see Getting started with CredHub Maestro.

When to use CredHub Maestro for certificate rotation

VMware recommends that you use the VMware Tanzu Operations Manager API to perform bulk rotation of certificates. For more information, see Overview of Certificate rotation.

However, there are some cases when you cannot use the Tanzu Operations Manager API to do bulk certificate rotation. In these cases, you can use CredHub Maestro to rotate certificates.

The following describes when to use CredHub Maestro instead of Tanzu Operations Manager API for certificate rotation:

You need to rotate one or more specific CredHub managed certificates as part of a certificate rotation repair process due to errors returned by the standard procedure.
If you use VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), rotate your certificates using the instructions in the TKGI documentation. Use the following instructions for only the certificates that are not rotated by the TKGI procedures.

Prerequisites

Before you rotate CredHub-managed certificates with CredHub Maestro:

To ensure that there is no app availability downtime during a rotation, VMware recommends that you scale up all instance groups and use an external blobstore.
Check which tiles in your installation support certificate rotation with CredHub Maestro. Not every tile currently supports certificate rotation, so there is a potential for downtime. For more information about which tiles support certificate rotation, see CredHub Maestro tile compatibility.

Check expiration dates and certificate types

This section describes how to manually check the expiration dates of the CAs and leaf certificates managed by CredHub.

After you identify the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

Retrieve a list of certificates expiring within a given timeframe by running:
```
maestro list --expires-within TIME-PERIOD
```
Where TIME-PERIOD is the unit for which you want to check the expiry window. Valid units are d for days, w for weeks, m for months, and y for years.
After you identify the list of certificates that expire soon, follow the procedure in Rotate certificates.

Rotate certificates

This section explains how to rotate leaf certificates and CAs stored in CredHub. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation.

To rotate certificates, use one of the following procedures:

If multiple CAs are expiring soon, see Rotate all CAs and Leaf Certificates.
If only leaf certificates are expiring, see Rotate all Leaf Certificates.
To rotate only a subset of CAs, see Rotate a single CA and the Leaf Certificates.
To rotate the Services TLS CA, see Rotate the Services TLS CA and the Leaf Certificates.
To rotate CredHub-set CAs, see Rotate CredHub-set CAs.
To rotate CredHub-set leaf certificates, see Rotate CredHub-set Leaf Certificates.

All CredHub Maestro rotation commands have a dry-run flag you can use to preview the certificates to be modified.

Safety checks

CredHub Maestro performs basic safety checks when rotating certificates to prevent unsafe operations.

To skip CredHub Maestro safety checks, you can pass the --skip-safety-check flag to CredHub Maestro certificate rotation commands.

For more information, see Troubleshooting CredHub Maestro safety violations during certificate rotation.

Rotate all CAs and leaf certificates

This section describes the procedure for bulk rotating all CAs and leaf certificates using CredHub Maestro.

VMware recommends that you use the Tanzu Operations Manager API to perform bulk rotation of certificates unless you meet the conditions described in [When to use CredHub Maestro for Certificate Rotation](#when-to-use). If your deployment includes Tanzu Kubernetes Grid Integrated Edition, using the --all option can damage your deployment. You can rotate individual CA certificates and their leaf certs instead. For TKGi-specific certificates, follow the certificate rotation instructions at Rotating Certificates for Tanzu Kubernetes Grid Integrated Edition.

If any of your deployments use the Services TLS CA:

Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /services/tls_ca.
Run maestro regenerate leaf with --exclude-signed-by /services/tls_ca.

The process for rotating the Services TLS CA requires additional steps to ensure that there is no downtime. To rotate the Services TLS CA, see Rotate the Services TLS CA and its Leaf Certificates.

If you have enabled the Automatic rotation of BOSH DNS certificates feature:

Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /opsmgr/bosh_dns/tls_ca.

To rotate all CAs and leaf certificates in CredHub:

Regenerate all CAs. This creates a new version of every CA and marks these new CA versions as transitional. The transitional status indicates that these new CA versions do not yet sign any leaf certificates, although they are trusted by clients. Run:
```
maestro regenerate ca --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes pane, select the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because, when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Mark the signing version of each CA as transitional. Run:
```
maestro update-transitional signing --all
```
Regenerate all leaf certificates. Run:
```
maestro regenerate leaf --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes pane, select the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Remove the transitional flag on all certificate versions. This action removes the old, inactive CA versions on the next deploy. Run:
```
maestro update-transitional remove --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes pane, select the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotating all leaf certificates

This section describes the procedure for bulk rotating all leaf certificates using CredHub Maestro.

VMware recommends that you use the Tanzu Operations Manager API to perform bulk rotation of certificates if you don’t meet the conditions described in When to use CredHub Maestro for Certificate Rotation.

If your deployment includes Tanzu Kubernetes Grid Integrated Edition, using the --all option can damage your deployment. You can rotate leaf certs signed by individual CAs instead. For TKGi-specific certificates, follow the certificate rotation instructions at Rotating Certificates for Tanzu Kubernetes Grid Integrated Edition.

To rotate all leaf certificates in CredHub:

Regenerate all leaf certificates. Run:
```
maestro regenerate leaf --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes pane, select the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because, when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotate a single CA and its leaf certificates

This section describes the procedure for rotating a single CA and its leaf certificates.

To rotate a single CA and its leaf certificates:

Regenerate the CA you want to rotate. This creates a new version of this CA. Run:
```
maestro regenerate ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to rotate. The CA name you provide must be the full path to the credential.
Redeploy all deployments that use the CA.
Mark the signing version of the CA as transitional. Run:
```
maestro update-transitional signing --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Regenerate all leaf certificates signed by the CA. Run:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
Where CA-NAME is the name of the CA for which you want to regenerate leaf certificates.
Redeploy all deployments that use the CA.
Remove the transitional flag from the CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to remove.
Redeploy all deployments that use the CA.

Rotate the Services TLS CA and its leaf certificates

You can set a value to override the duration for certificates in the BOSH Director tile. VMware recommends that you set the value before starting a certificate rotation. For more information, see Overriding duration for certificates.

This section describes the procedure for rotating the Services TLS CA and its leaf certificates.

Before you begin this rotation procedure, ensure that you have the following:

Only one Tanzu Operations Manager foundation and not a multi-site or Wide Area Network (WAN) setup. For a multi-site or WAN setup, you must perform the Apply Changes steps across all your sites or WAN.
All certificates must include Subject Alternate Name (SAN) and Common Name (CN) entries.
Your services are configured for high availability to minimize potential app impact during CA rotation. Rotating the Services TLS CA requires upgrading all on-demand service instances, which can cause service downtime.
You must know whether your deployment contains non-Java MySQL apps that rely on VCAP_SERVICES for TLS. You must rebind and restage these apps during the procedure, which can require additional time and coordination for the rotation. You do not need to rebind and restage MySQL apps that use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt. For more information about determining which apps are affected, see Rotating /services/tls_ca certificate using CredHub transitional certificate rotation feature.

To rotate the Services TLS CA and its leaf certificates for Gemfire for TAS, WAN configuration only

Follow the procedure in Rotating Certificates (Gemfire for TAS WAN Setups only).

To rotate the Services TLS CA and its leaf certificates (all other configurations)

Check the CredHub Maestro tile compatibility document to ensure that all service tile versions currently deployed are compatible with Maestro.
If you have any service tiles that are not compatible with CredHub Maestro, follow their rotation procedure. Do not follow this procedure. If you do, you might experience extended downtime or data loss.
Regenerate the Services TLS CA and mark the latest version of the CA as transitional. This creates a new version of the Services TLS CA and indicates that the latest version is inactive, so that all leaf certificates trust the new CA.
1. On the Services TLS CA, see if your Services TLS CA was CredHub-set. Run:
```
maestro list
```
  After you run this command, you see one Services TLS CA listed per deployment, but it is the same certificate.
2. If the Services TLS CA was CredHub-set, run:
```
maestro regenerate ca --name "/services/tls_ca" --force
```
3. For deployments that use MySQL for VMware Tanzu, see the MySQL for VMware Tanzu documentation to add the new Services TLS CA to the Trusted Certificates of the BOSH Director.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  
  The name of the **Upgrade all service instances** errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.
The bg-restage plug-in enables zero-downtime blue-green restaging without access to the app source code. For more information, see bg-restage on GitHub.
Mark the signing version of the Services TLS CA as transitional. Run:
```
maestro update-transitional signing --name "/services/tls_ca"
```
Regenerate all service instance certificates signed by the Services TLS CA. Run:
```
maestro regenerate leaf --signed-by "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the following apps:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non Java MySQL apps that rely on VCAP_SERVICES for TLS.
Remove the transitional flag from the Services TLS CA. This removes the old, inactive version of the Services TLS CA on the next deployment. Run:
```
maestro update-transitional remove --name "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
For the following apps:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non Java MySQL apps that rely on VCAP_SERVICES for TLS.

Rotate CredHub-set CAs

This section describes the procedure for rotating CAs that are manually set in CredHub.

To rotate a CredHub-set CA:

Create a new CA using the same parameters you used for the original CA.
Use the CredHub CLI or CredHub API to set the new CA at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.
Find out which tiles or service instances are using the rotated certificate.
1. Get the list of deployment names using one of the following methods:
  - To get a list of deployments using the CA or its leaf certificates, run the following command:
```
maestro --json ls | jq -r '.metadata[] | select(.name == "/PATH/TO/CA/IN/CREDHUB" or .signed_by == "/PATH/TO/CA/IN/CREDHUB") | .deployment_name' | sort | uniq
```
  - To verify the deployments manually:
    1. Use Maestro to get a list of all certificates. Run:
```
maestro ls
```
    2. Locate the output entries whose name or signed_by field matches the CredHub path of the CA that was rotated. Certificates might be listed multiple times if they are associated with multiple deployments.
    3. For each output entry, look at the deployment_name key. This corresponds to a tile that you must redeploy.
2. Entries with a deployment name like service-instance-* are certificates associated with a service instance created by a service tile. You must redeploy the corresponding service tile and enable its Upgrade all service instances errand.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, select the check box for each of the products using the rotated certificates. If you are unsure, select the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because, when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Mark the signing version of the single CA as transitional. Run:
```
maestro update-transitional signing --name "CA-NAME"
```
Regenerate all leaf certificates that are signed by the CA you are rotating. Run:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
You might need to use the --force flag when regenerating leaf certificates if they were set by CredHub. Otherwise, you must provide a new leaf certificate at the same path.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, select the check box for all products using the rotated certificates. If you are unsure, select the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because, when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Remove the transitional flag from the single CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, select the check box for all products using the rotated certificates. If you are unsure, select the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because, when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotate CredHub-set leaf certificates

This section describes the procedure for rotating leaf certificates that are manually set in CredHub.

To rotate a CredHub-set leaf certificate:

Create a new leaf certificate using the same parameters you used for the original leaf certificate.
Use the CredHub CLI or CredHub API to set the new leaf certificate at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.
Verify which tiles or service instances are using the rotated certificate.
1. Get the list of deployment names using one of the following methods:
  - To get a list of deployments using the CA or its leaf certificates, run:
```
maestro --json ls | jq -r '.metadata[] | select(.name == "/PATH/TO/CERTIFICATE/IN/CREDHUB") | .deployment_name' | sort | uniq
```
  - To verify the deployments manually:
    1. Use Maestro to get a list of all certificates by running:
```
maestro ls
```
    2. Locate the output entries whose name matches the CredHub path of the certificate that was rotated. Note that a certificate might be listed multiple times if it is associated with multiple deployments.
2. For each output entry, look at the deployment_name key. This corresponds to a tile that you must redeploy. Entries with a deployment name like service-instance-* are certificates associated with a service instance created by a service tile. You must redeploy the corresponding service tile and enable its Upgrade all service instances errand.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, select the check box for all products using the rotated certificate. If you are unsure, select the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is unselected. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Delete inactive certificate versions

Over the course of many rotations, it is possible for the CredHub database to fill up. In order to prevent this, you can periodically use CredHub Maestro’s garbage collection functionality to delete old, unused certificate versions.

You can run garbage collect on leaf certificates and CAs. Generally, deleting leaf certificates is enough to free up space in the database.

To see if you need to run garbage collect, do one of the following:

Manually verify the BOSH Director’s VM vitals through your IaaS provider.
SSH into the BOSH Director and run df -h /var/vcap/store.

To enhance the safety of the garbage collect command, CredHub Maestro only deletes versions of a certificate older than the currently active version. To delete all inactive versions of CAs or leaf certificates, add the –force flag to the garbage collect command. VMware does not recommend using the –force flag during a rotation.

Delete all leaf certificates

To delete all leaf certificates:

Determine which certificate versions you want to delete by running:
```
maestro garbage-collect leaf --all --dry-run
```
Delete the certificate versions by running:
```
maestro garbage-collect leaf --all
```

Delete all CAs

To delete all CAs:

Determine which CA versions you want to delete by running:
```
maestro garbage-collect ca --all --dry-run
```
Delete the CA versions by running:
```
maestro garbage-collect ca --all
```

Delete a single leaf certificate

To delete a single leaf certificate:

Determine which certificate versions you want to delete by running:

maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run

Delete the certificate versions by running:

maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"

Delete a single CA

To delete a single CA:

Determine which CA you want to delete by running:
```
maestro garbage-collect ca --name "CA-NAME" --dry-run
```
Where CA-NAME is the name of the CA you want to delete.
Delete the CA by running:
```
maestro garbage-collect ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to delete.