Advanced certificate rotation with CredHub Maestro

This topic describes how you can rotate certificates and certificate authorities (CAs) in BOSH CredHub using CredHub Maestro. The topic only covers rotating certificates that are managed by CredHub.

For more information about setting up and using CredHub Maestro, see Getting started with CredHub Maestro.

When to use CredHub Maestro for certificate rotation

VMware recommends that you use the VMware Tanzu Operations Manager API to perform bulk rotation of certificates. For more information, see Overview of Certificate rotation.

However, there are some cases when you cannot use the Tanzu Operations Manager API to do bulk certificate rotation. In these cases, you can use CredHub Maestro to rotate certificates.

The following information describes when to use CredHub Maestro instead of Tanzu Operations Manager API for certificate rotation:

You need to rotate one or more specific CredHub managed certificates as part of a certificate rotation repair process due to errors returned by the standard procedure.
If you use VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), rotate your certificates using the instructions in the TKGI documentation. Use the following instructions for only the certificates that are not rotated by the TKGI procedures.

Prerequisites

Before you rotate CredHub-managed certificates with CredHub Maestro:

Before rotating certificates in Tanzu Operations Manager v2.10, you must reset any certificates that were manually set in CredHub on Tanzu Operations Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, follow the procedure in Reviewing Manually Set Certificates in CredHub. For more information, see Reset manually set certificates in CredHub before rotating certificates in _Tanzu Operations Manager v2.10 Release Notes.
To ensure that there is no app availability downtime during a rotation, VMware recommends that you scale up all instance groups and use an external blobstore.
Check which tiles in your installation support certificate rotation with CredHub Maestro. Not every tile currently supports certificate rotation, so there is a potential for downtime. For more information about which tiles support certificate rotation, see CredHub Maestro tile compatibility.

Check expiration dates and certificate types

This section describes how to manually check the expiration dates of the CAs and leaf certificates managed by CredHub.

After you identify the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

Retrieve a list of certificates expiring within a given timeframe by running:
```
maestro list --expires-within TIME-PERIOD
```
Where TIME-PERIOD is the unit for which you want to check the expiry window. Valid units are d for days, w for weeks, m for months, and y for years.
After you identify the list of certificates that expire soon, follow the procedure in Rotate certificates.

Rotate certificates

This section explains how to rotate leaf certificates and CAs stored in CredHub. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated.

To rotate certificates, follow one of the following procedures:

If multiple CAs are expiring soon, see Rotate all CAs and Leaf Certificates.
If the only certificates expiring are leaf certificates, see Rotate all Leaf Certificates.
To only rotate a subset of CAs, see Rotate a single CA and the Leaf Certificates.
To rotate the Services TLS CA, see Rotate the Services TLS CA and the Leaf Certificates.
To rotate CredHub-set CAs, see Rotate CredHub-set CAs below.
To rotate CredHub-set leaf certificates, see Rotate CredHub-set Leaf Certificates below.

All CredHub Maestro rotation commands have a dry run flag that can be used to preview the certificates to be modified.

Safety checks

CredHub Maestro performs basic safety checks when rotating certificates to prevent unsafe operations.

To skip CredHub Maestro safety checks, you can pass the --skip-safety-check flag to CredHub Maestro certificate rotation commands.

For more information, see Troubleshooting CredHub Maestro safety violations during certificate rotation.

Rotate all CAs and leaf certificates

This section describes the procedure for bulk rotating all CAs and leaf certificates using CredHub Maestro.

VMware recommends that you use the Tanzu Operations Manager API to perform bulk rotation of certificates unless you meet the conditions described in When to use CredHub Maestro for Certificate Rotation. If your deployment includes Tanzu Kubernetes Grid Integrated Edition, using the –all option can damage your deployment. You can rotate individual CA certificates and their leaf certs instead. For TKGi-specific certificates, please follow the certificate rotation instructions at Rotating Certificates for Tanzu Kubernetes Grid Integrated Edition.

If any of your deployments use the Services TLS CA:

Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /services/tls_ca.
Run maestro regenerate leaf with --exclude-signed-by /services/tls_ca.

The process for rotating the Services TLS CA requires additional steps to ensure that there is no downtime. To rotate the Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates below.

To rotate all CAs and leaf certificates in CredHub:

Regenerate all CAs. This creates a new version of every CA and marks these new CA versions as transitional. The transitional status indicates that these new CA versions will not yet sign any leaf certificates, though they will be trusted by clients. Run:
```
maestro regenerate ca --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Mark the signing version of each CA as transitional by running:
```
maestro update-transitional signing --all
```
Regenerate all leaf certificates by running:
```
maestro regenerate leaf --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Remove the transitional flag on all certificate versions. This removes the old, inactive CA versions on the next deploy. Run:
```
maestro update-transitional remove --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotating all leaf certificates

This section describes the procedure for bulk rotating all leaf certificates using CredHub Maestro.

VMware recommends that you use the Tanzu Operations Manager API to perform bulk rotation of certificates unless you meet the conditions described in When to Use CredHub Maestro for Certificate Rotation above.

If your deployment includes Tanzu Kubernetes Grid Integrated Edition, using the --all option can damage your deployment. You can rotate leaf certs signed by individual CAs instead. For TKGi-specific certificates, please follow the certificate rotation instructions at Rotating Certificates for Tanzu Kubernetes Grid Integrated Edition.

To rotate all leaf certificates in CredHub:

Regenerate all leaf certificates by running:
```
maestro regenerate leaf --all
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the Select All Products check box.
4. For each service tile you have deployed:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotate a single CA and its leaf certificates

This section describes the procedure for rotating a single CA and its leaf certificates.

To rotate a single CA and its leaf certificates:

Regenerate the CA you want to rotate. This creates a new version of this CA. Run:
```
maestro regenerate ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to rotate. The CA name you provide must be the full path to the credential.
Redeploy all deployments that use the CA.
Mark the signing version of the CA as transitional by running:
```
maestro update-transitional signing --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Regenerate all leaf certificates signed by the CA by running:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
Where CA-NAME is the name of the CA for which you want to regenerate leaf certificates.
Redeploy all deployments that use the CA.
Remove the transitional flag from the CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to remove.
Redeploy all deployments that use the CA.

Rotate the Services TLS CA and its leaf certificates

You can set a value to override the duration for certificates within the BOSH Director tile. VMware recommends that you set the value before starting a certificate rotation. For more information, see Overriding duration for certificates.

This section describes the procedure for rotating the Services TLS CA and its leaf certificates.

Before you begin this rotation procedure, ensure that you have the following:

Only one Tanzu Operations Manager foundation and not a multi-site or Wide Area Network (WAN) setup. For a multi-site or WAN setup, you must perform the Apply Changes steps across all your sites or WAN.
All certificates must include Subject Alternate Name (SAN) and Common Name (CN) entries.
Your services are configured for high availability to minimize potential app impact during CA rotation. Rotating the Services TLS CA requires an upgrade all on-demand service instances, which can cause service downtime.
You must know whether your deployment contains non-Java MySQL apps that rely on VCAP_SERVICES for TLS. You must rebind and restage these apps during the procedure, which can require additional time and coordination for the rotation. You do not need to rebind and restage MySQL apps that use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt. For more information about determining which apps are affected, see Rotating /services/tls_ca certificate using CredHub transitional certificate rotation feature.

To rotate the Services TLS CA and its leaf certificates:

Check the CredHub Maestro tile compatibility document to ensure that all service tile versions currently deployed are compatible with Maestro.

If you have any service tiles that are not compatible with CredHub Maestro, follow their rotation procedure. Do not follow this procedure. If you do, you might experience extended downtime or data loss.
Regenerate the Services TLS CA and mark the latest version of the CA as transitional. This creates a new version of the Services TLS CA and indicates that the latest version is inactive, so that all leaf certificates trust the new CA.
1. On the Services TLS CA, see if your Services TLS CA was CredHub-set by running:
```
maestro list
```
  After you run this command, you see one Services TLS CA listed per deployment, but it is the same certificate.
2. If the Services TLS CA was CredHub-set, run:
```
maestro regenerate ca --name "/services/tls_ca" --force
```
3. For deployments that use MySQL for VMware Tanzu, follow these instructions to add the new Services TLS CA to the Trusted Certificates of the BOSH Director.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the following apps:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.
The bg-restage plug-in enables zero-downtime blue-green restaging without access to the app source code. For more information, see bg-restage on GitHub.
Mark the signing version of the Services TLS CA as transitional by running:
```
maestro update-transitional signing --name "/services/tls_ca"
```
Regenerate all service instance certificates signed by the Services TLS CA by running:
```
maestro regenerate leaf --signed-by "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the following apps:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non Java MySQL apps that rely on VCAP_SERVICES for TLS.
Remove the transitional flag from the Services TLS CA. This removes the old, inactive version of the Services TLS CA on the next deployment. Run:
```
maestro update-transitional remove --name "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the following apps:
- For Tanzu GemFire for VMs apps, rebind and restage the apps.
- For deployments that use MySQL for VMware Tanzu, rebind and restage all non Java MySQL apps that rely on VCAP_SERVICES for TLS.

Rotate CredHub-set CAs

This section describes the procedure for rotating CAs that are manually set in CredHub.

To rotate a CredHub-set CA:

Create a new CA using the same parameters you used for the original CA.
Use the CredHub CLI or CredHub API to set the new CA at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.
Verify which tiles or service instances are using the rotated certificate.
1. Get the list of deployment names using one of the following methods:
  - To get a list of deployments using the CA or its leaf certificates, run the following command:
```
maestro --json ls | jq -r '.metadata[] | select(.name == "/PATH/TO/CA/IN/CREDHUB" or .signed_by == "/PATH/TO/CA/IN/CREDHUB") | .deployment_name' | sort | uniq
```
  - To verify the deployments manually:
    1. Use Maestro to get a list of all certificates by running:
```
maestro ls
```
    2. Locate the output entries whose name or signed_by field matches the CredHub path of the CA that was rotated. Certificates might be listed multiple times if they are associated with multiple deployments.
    3. For each output entry, look at the deployment_name key. This corresponds to a tile that you must re-deploy.
2. Entries with a deployment name like service-instance-* are certificates associated with a service instance created by a service tile. You must re-deploy the corresponding service tile and enable its Upgrade all service instances errand.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the check box for all products using the rotated certificates. If unsure, enable the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Mark the signing version of the single CA as transitional by running:
```
maestro update-transitional signing --name "CA-NAME"
```
Regenerate all leaf certificates that are signed by the CA you are rotating. Run:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
You might need to use the --force flag when regenerating leaf certificates if they were set by CredHub. Otherwise, you must provide a new leaf certificate at the same path.
Redeploy:
1. Return to Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the check box for all products using the rotated certificates. If unsure, enable the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.
Remove the transitional flag from the single CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the check box for all products using the rotated certificates. If unsure, enable the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Rotate CredHub-set leaf certificates

This section describes the procedure for rotating leaf certificates that are manually set in CredHub.

To rotate a CredHub-set leaf certificate:

Create a new leaf certificate using the same parameters you used for the original leaf certificate.
Use the CredHub CLI or CredHub API to set the new leaf certificate at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.
Verify which tiles or service instances are using the rotated certificate.
1. Get the list of deployment names using one of the following methods:
  - To get a list of deployments using the CA or its leaf certificates, run the following command:
```
maestro --json ls | jq -r '.metadata[] | select(.name == "/PATH/TO/CERTIFICATE/IN/CREDHUB") | .deployment_name' | sort | uniq
```
  - To verify the deployments manually:
    1. Use Maestro to get a list of all certificates by running:
```
maestro ls
```
    2. Locate the output entries whose name matches the CredHub path of the certificate that was rotated. Note that a certificate may be listed multiple times if it is associated with multiple deployments.
2. For each output entry, look at the deployment_name key. This corresponds to a tile that you must re-deploy. Entries with a deployment name like service-instance-* are certificates associated with a service instance created by a service tile. You must re-deploy the corresponding service tile and enable its Upgrade all service instances errand.
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. Click Review Pending Changes.
3. On the Review Pending Changes page, enable the check box for all products using the rotated certificate. If unsure, enable the Select All Products check box.
4. For each service tile you are deploying:
  1. Verify that the Recreate all service instances errand is deactivated. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
  2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
  The names of the Recreate all service instances and the Upgrade all service instances errands might be slightly different between services.
5. Click Apply Changes.

Delete inactive certificate versions

Over the course of many rotations, it is possible for the CredHub database to fill up. In order to prevent this, you can periodically use CredHub Maestro’s garbage collect functionality to delete old, unused certificate versions.

You can run garbage collect on leaf certificates and CAs. Generally, deleting leaf certificates is enough to free up space in the database.

To see if you need to run garbage collect, do one of the following:

Manually check the BOSH Director’s VM vitals through your IaaS provider.
SSH into the BOSH Director and run df -h /var/vcap/store.

To enhance the safety of the garbage collect command, CredHub Maestro only deletes versions of a certificate older than the currently active version. To delete all inactive versions of CAs or leaf certificates, add the --force flag to the garbage collect command. VMware does not recommend using the --force flag during a rotation.

Delete all leaf certificates

To delete all leaf certificates:

Determine which certificate versions you want to delete by running:
```
maestro garbage-collect leaf --all --dry-run
```
Delete the certificate versions by running:
```
maestro garbage-collect leaf --all
```

Delete all CAs

To delete all CAs:

Determine which CA versions you want to delete by running:
```
maestro garbage-collect ca --all --dry-run
```
Delete the CA versions by running:
```
maestro garbage-collect ca --all
```

Delete a single leaf certificate

To delete a single leaf certificate:

Determine which certificate versions you want to delete by running:

maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run

Delete the certificate versions by running:

maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"

Delete a single CA

To delete a single CA:

Determine which CA you want to delete by running:
```
maestro garbage-collect ca --name "CA-NAME" --dry-run
```
Where CA-NAME is the name of the CA you want to delete.
Delete the CA by running:
```
maestro garbage-collect ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to delete.