Rotating CAs and leaf certificates using the Tanzu Operations Manager API

This topic describes how you can rotate certificate authorities (CAs) and leaf certificates using the VMware Tanzu Operations Manager API. To rotate only leaf certificates, see one of the leaf certificate rotation topics listed in Certificate Rotation Procedures in Overview of Certificate Rotation.

This procedure uses the Tanzu Operations Manager API to rotate all CA and leaf certificates generated by Tanzu Operations Manager and CredHub.

This procedure includes three redeployments. The first deployment regenerates the new CA certificates used by Tanzu Operations Manager components. The second deploy replaces the old leaf certificates with the new leaf certificates. The third deploy removes the old CA certificates, leaving only the new CA certificates.

Certificates rotated by the procedure

This procedure rotates all certificates listed on the Certificates page in Tanzu Operations Manager, except those with specialized rotation procedures.

The following certificates types will be rotated:

• CA certificates
• Non-configurable leaf certificates
• Configurable leaf certificates stored in Tanzu Operations Manager

Some certificates require a different procedure to rotate. For example, the Services TLS CA, /services/tls_ca, cannot be rotated using this procedure. For more information, see Certificates with specialized procedures.

The Tanzu Operations Manager API also excludes from rotation the CAs for certain versions of the VMware Tanzu SQL [MySQL for VMs], VMware Tanzu for Valkey on Cloud Foundry (formerly Redis for VMware Tanzu), VMware Tanzu RabbitMQ, VMware Tanzu GemFire tiles, and IPsec for VMware Tanzu. For information about rotating these CAs, see the documentation for those products.

Prerequisites

Before you rotate certificates:

• You must have Tanzu Operations Manager v3.0.

• You must have Jammy stemcells at version 1.8 or later, Xenial stemcells at version 621.171 or later, and be running Windows 2019.41 or later.

  If you have any VMs deployed with any of the following stemcells, you will have to recreate the VMs at every Apply Changes step:

  • Xenial stemcells earlier than release 621.171
  • Windows stemcells earlier than release 2019.41
  • Jammy stemcells earlier than release 1.8

  If you are still using these stemcell versions and you cannot immediately upgrade, you must enable the Recreate all service instances errand, if the service tile provides it.
  Running this errand pushes BOSH Agent certificate updates to service instances.
  
  If the service tile does not include the Recreate all service instances errand, see the procedure Running with old Xenial or Windows stemcells on Rotating CAs and leaf certificates using the Tanzu Operations Manager API for instructions.

• You must reset any certificates that were manually set in CredHub on Tanzu Operations Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, see Reviewing manually set certificates in CredHub.

If you use VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), rotate your certificates using the instructions in the TKGI documentation. For certificates that are not rotated by the TKGI procedures, see Advanced certificate rotation with CredHub Maestro.

• Your product tile must have one of the following versions installed to rotate CredHub-managed certificates with the Tanzu Operations Manager API:

  • TAS for VMs v2.7.21 or later.
  • TAS for VMs v2.8.2 or later.
  • Isolation Segment tile v2.7.21 or later.
  • Isolation Segment tile v2.8.2 or later.
  • Small Footprint TAS for VMs v2.7.21 or later.
  • Small Footprint TAS for VMs v2.8.2 or later.
  • TAS for VMs [Windows] v2.7.17 or later.
  • TAS for VMs [Windows] v2.8.2 or later.

If you have an earlier version of a product tile that was previously listed, use this procedure to rotate certificates managed only by Tanzu Operations Manager. Then, to rotate certificates managed in CredHub, see Advanced certificate rotation with CredHub Maestro.

Procedure

You must complete these steps in the exact order specified. Otherwise, you can damage your deployment.

You can set a value to override the duration for certificates in the BOSH Director tile. VMware recommends that you set the value before starting a certificate rotation. For more information, see Overriding duration for certificates.

To rotate CAs, configurable leaf certificates, and non-configurable leaf certificates:

1. Regenerate CAs. This step regenerates CAs for all CAs visible to the Tanzu Operations Manager API. For more information, see Regenerate CAs.

2. Activate the new CAs. This step activates the new CAs regenerated in the previous step. See Activate the CAs.

3. Regenerate configurable leaf certificates. This step regenerates configurable certs to use the new CAs. See Regenerate configurable Leaf Certificates.

4. Regenerate non-configurable leaf certificates. This step regenerates non-configurable certs to use the new CAs. See Regenerate non-configurable Leaf Certificates.

5. Delete the old CAs. This step deletes the old unused CAs, leaving only the regenerated new CAs. See Delete the old CAs.

Step 1: Regenerate CAs

This section describes how to regenerate the CA certificates used by Tanzu Operations Manager, tiles, and service instances.

You can provide your own custom CA for the Tanzu Operations Manager root CA instead of using one generated by Tanzu Operations Manager, if you want to. The custom CA will be used to sign leaf certificates that are managed by Tanzu Operations Manager. Leaf certificates managed by CredHub are not signed using the Tanzu Operations Manager root CA and will be not be signed by the provided custom CA.

To regenerate CAs:

1. If you have not already done so, follow the procedure in Using Tanzu Operations Manager API to target and authenticate with the Tanzu Operations Manager UAA server. Record your Tanzu Operations Manager access token, and use it for UAA-ACCESS-TOKEN in the following steps.
   When you record your Tanzu Operations Manager access token, remove any newline characters such as \n.

2. Use Tanzu Operations Manager to generate a new CA, or add your own custom CA:

   • To use your own custom CA: Call the Tanzu Operations Manager API certificate_authorities endpoint by running:

     curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
         -X POST \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -H "Content-Type: application/json" \
         -d '{"cert_pem": "-----BEGIN CERTIFICATE-----CUSTOM-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----RSA-KEY"}' \
         -i
     

     Where:

     • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Tanzu Operations Manager deployment.
     • CUSTOM-CERTIFICATE is your custom CA.
     • RSA-KEY is your RSA key. Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in Tanzu Operations Manager.
     • UAA-ACCESS-TOKEN is your UAA access token.

     If the command succeeds, the API returns a response that includes the new CA. For example:

     HTTP/1.1 200 OK
     {
           "certificate_authorities": [
             {
               "guid": "f7bc18f34f2a7a9403c3",
               "issuer": "CUSTOM-CERTIFICATE",
               "created_on": "2017-01-09",
               "expires_on": "2021-01-09",
               "active": true,
               "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
             }
           ]
     }
     

   • To use a Tanzu Operations Manager-generated CA: Call the Tanzu Operations Manager API generate endpoint by running:

     curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/generate" \
           -X POST \
           -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
           -H "Content-Type: application/json" \
           -d '{}' \
           -i
     

     Where:

     • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
     • UAA-ACCESS-TOKEN is your UAA access token.

     If the command succeeds, the API returns a response that includes the new CA. For example:

     HTTP/1.1 200 OK
     {
           "guid": "f7bc18f34f2a7a9403c3",
           "issuer": "VMware",
           "created_on": "2017-01-19",
           "expires_on": "2021-01-19",
           "active": false,
           "cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
           MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
           EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
           EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
           EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
           EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
           EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
           EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
           EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
           EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
           EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
           EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
           EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
           EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
           EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
           EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
           EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
           -----END EXAMPLE CERTIFICATE-----"
     }
     

3. Confirm that the new CA is added by listing all the CAs for Tanzu Operations Manager. Run:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
         -X GET \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • UAA-ACCESS-TOKEN is your UAA access token.

   The API call returns the GUID of the new CA. For example:

   HTTP/1.1 200 OK
   {
         "certificate_authorities": [
           {
             "guid": "f7bc18f34f2a7a9403c3",
             "issuer": "VMware",
             "created_on": "2017-01-09",
             "expires_on": "2021-01-09",
             "active": true,
             "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
           }
           {
             "guid": "a8ee01e33e3e3e3303e3",
             "issuer": "VMware",
             "created_on": "2017-04-09",
             "expires_on": "2021-04-09",
             "active": false,
             "cert_pem": "-----BEGIN CERTIFICATE-----\nzBBBC+eAAAe1gAwAAAeZ....etc"
           }
         ]
   }
   

4. Identify the new CA, which has active set to false. Record its GUID.

5. Return to Tanzu Operations Manager Installation Dashboard.

6. If you have service tiles installed, for each service tile, do the following:

   The name of the Upgrade all service instances errand might be slightly different between services. In the TKGi tile, the errand is called Upgrade all clusters.

   1. Click the tile.
   2. Click the Errands tab.
   3. Enable the Upgrade all service instances errand. Running this errand is necessary to push certificate updates to each service instance.

7. Click Review Pending Changes.

8. Click Apply Changes.

   If you are running a deployment with old Xenial or Windows stemcells, you must recreate all VMs. See Running with old versions of Xenial or Windows stemcells.

When the deployment finishes and you have completed all necessary tasks to update service instances, continue to the next section, Activate the CAs.

Step 2: Activate the CAs

To activate the new CAs created in Regenerate CAs:

1. Call the Tanzu Operations Manager API activate endpoint by running:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/CERTIFICATE-GUID/activate" \
     -X POST \
     -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
     -H "Content-Type: application/json" \
     -d '{}' \
     -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • CERTIFICATE-GUID is the GUID of your CA that you retrieved in the previous section.
   • UAA-ACCESS-TOKEN is your UAA access token.

   The API returns a successful response:

   HTTP/1.1 200 OK

2. List your Tanzu Operations Manager root CAs to confirm that the new root CA is active. Run:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
         -X GET \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • UAA-ACCESS-TOKEN is your UAA access token.

3. Examine the response to ensure that your new root CA has the active property set to true.

Step 3: Regenerate configurable leaf certificates

Any configurable leaf certificates generated from the Tanzu Operations Manager root CA must also be regenerated. These are the leaf certificates that you generated using the Generate RSA Certificate button in the Tanzu Operations Manager UI. For example, see Providing a Certificate for Your TLS Termination Point in the TAS for VMs documentation.

To regenerate each configurable leaf certificate that expires soon:

1. Find the text field where the leaf certificate is configured in the Tanzu Operations Manager UI. You might need to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:

   • The product_guid field in the Tanzu Operations Manager API output can help identify the tile in which the certificate is configured. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the TAS for VMs tile.
   • The property_reference field in the Tanzu Operations Manager API output can help identify the configuration pane in which the certificate is configured. For example, the uaa.service_provider_key_credentials property is configured in the UAA pane of the TAS for VMs tile.

2. Copy a new value for the leaf certificate into the text field or generate a new leaf certificate using the Generate RSA Certificate button.

3. Click Save on the pane in which you added new leaf certificates.

Step 4: Regenerate non-configurable leaf certificates

After you activate the new CAs, you must regenerate the non-configurable leaf certificates to ensure they are signed by the new CAs.

To regenerate the non-configurable leaf certificates:

1. Call the Tanzu Operations Manager API regenerate endpoint by running:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/active/regenerate" \
         -X POST \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -H "Content-Type: application/json" \
         -d '{}' \
         -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • UAA-ACCESS-TOKEN is your UAA access token.

   The API returns a successful response:

   HTTP/1.1 200 OK

2. Return to Tanzu Operations Manager Installation Dashboard.

3. If you have service tiles installed, for each service tile do the following:

   The name of the Upgrade all service instances errand might be slightly different between services. In the TKGi tile, the errand is called Upgrade all clusters.

   1. Click the tile.
   2. Click the Errands tab.
   3. Enable the Upgrade all service instances errand. Running this errand is necessary to push certificate updates to each service instance.

4. Click Review Pending Changes.

5. Click Apply Changes.

   If you are running a deployment with old Xenial or Windows stemcells, you must recreate all VMs. See Running with old versions of Xenial or Windows stemcells.

6. When the deployment finishes and you have completed all necessary tasks to update service instances, continue to the next section, Delete the old CAs.

Step 5: Delete the old CAs

After activating new CAs and regenerating leaf certificates from the new CAs, you must delete the old CAs. Deleting the old CAs ensures that each leaf certificate is associated only with the new CA that signs it, and that the old CAs are no longer trusted.

To delete the old CAs:

1. List your root CAs and retrieve the GUID of the old, inactive Tanzu Operations Manager root CA. Run:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
         -X GET \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • UAA-ACCESS-TOKEN is your UAA access token.

2. Delete the old Tanzu Operations Manager root CA by running:

   curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/OLD-CERTIFICATE-GUID" \
         -X DELETE \
         -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
         -i
   

   Where:

   • OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
   • OLD-CERTIFICATE-GUID is the GUID of the old Tanzu Operations Manager root CA.
   • UAA-ACCESS-TOKEN is your UAA access token.

   The API returns a successful response:

   HTTP/1.1 200 OK

3. If you have service tiles installed, for each service tile, do the following:

   The name of the Upgrade all service instances errand might be slightly different between services. In the TKGi tile, the errand is called Upgrade all clusters.

   1. Click the tile.
   2. Click the Errands tab.
   3. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

4. Click Review Pending Changes.

5. Click Apply Changes.

   If you are running a deployment with old Xenial or Windows stemcells, you must recreate all VMs. See Running with old versions of Xenial or Windows stemcells.

When the deployment finishes and you have completed all necessary tasks to update service instances, the rotation procedure is complete.

Certificates with specialized procedures

Some certificates are not rotated when using the standard rotation procedure. The following table lists certificates that use a separate, specialized rotation procedure and provides links to documentation and help for rotating these certificates.

Certificate	Tile	Procedure
/services/tls_ca	n/a	Rotate the Services TLS CA and its Leaf Certificates
/telemetry-ca-cert	Telemetry v1.2.1 and earlier	Telemetry v1.2.1 and earlier are out of support. If you need assistance, contact Broadcom Support.
/services/tls_leaf	TAS for VMs v2.10.0 – v2.10.13	These TAS for VMs versions are out of support. If you need assistance, contact Broadcom Support.

Troubleshooting

Safety Violation Errors

The Tanzu Operations Manager API runs CredHub Maestro when rotating certificates. If a certificate rotation API command is unsafe, CredHub Maestro stops the command and returns one or more safety violations.

For example, CredHub Maestro stops a certificate rotation API command if you try to follow certificate rotation steps in the wrong order. Because performing these steps in the wrong order can make your deployment unstable or cause downtime, CredHub Maestro stops the command and returns an error message.

For information about how to troubleshoot safety violation errors that are returned when rotating certificates, see Troubleshooting CredHub Maestro safety violations during certificate rotation.

CredHub Garbage Collection

This procedure does not delete leaf certificates stored in CredHub. It marks CredHub leaf certificates as transitional, which indicates that the leaf certificate is inactive. To delete CredHub leaf certificates, use the CredHub Maestro garbage collection feature. For information about CredHub Maestro certificate garbage collection, see Delete Inactive Certificate Versions in Advanced Certificate Rotation with CredHub Maestro.

Running with old versions of Xenial or Windows stemcells

If you are running old versions of the Xenial and/or Jammy stemcells, you should upgrade the stemcells. If there is some reason you cannot upgrade the stemcells, then you must recreate all VMs by enabling the service tile Recreate all service instances errand. If you are running with Jammy stemcells, you don’t need to recreate the VMs.

If the tile does not include this errand, use the procedure here. This procedure propagates the new CA to the BOSH Agent on each VM deployed by the BOSH Director, which enables the new leaf certificates to trust the new CA.

• Go to https://OPS-MANAGER-FQDN in a browser, where OPS-MANAGER-FQDN is the FQDN of your Tanzu Operations Manager deployment.
• Log in to the Tanzu Operations Manager Installation Dashboard.
• Click the BOSH Director tile, then click Director Config.
• Select the Recreate VMs deployed by the BOSH Director check box.

Or you can run:

bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate

Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.

For details, see the Prerequisites section earlier in this topic.