In Tanzu Operations Manager 3.0.17+LTS-T, the Director Config pane includes the “Enable automatic rotation of the BOSH DNS CA certificate (experimental)” check box. When you activate this check box, it causes Tanzu Operations Manager to automatically rotate the BOSH DNS CA certificate and associated leaf certificates during the Apply Changes action. The rotated BOSH DNS certificates can be deployed to VMs during the Apply Changes action when new stemcells are also being deployed to those VMs. This allows the BOSH DNS certificates to be rotated gradually over time without user intervention, and without encountering unexpected long Apply Changes action times.

Certificates covered by automatic rotation

Currently, only the BOSH DNS CA certificate and its associated leaf certificates are rotated by the automatic rotation feature. These certificates are stored in CredHub:

  • /opsmgr/bosh_dns/tls_ca
  • /bosh_dns_health_client_tls
  • /bosh_dns_health_server_tls
  • /dns_api_client_tls
  • /dns_api_server_tls
  • /opsmgr/bosh_dns/san_migrated

Enable automatic rotation of the BOSH DNS certificates

By default, the automatic rotation feature is deactivated. To enable the automatic rotation feature feature:

  1. Click the BOSH Director tile.
  2. Click Director Config.
  3. Check the “Enable automatic rotation of the BOSH DNS CA certificate (experimental)” check box
  4. Click Save.

Advancing the certificate rotation

After you enable the automatic rotation feature, a new step at the start of each Apply Changes tries to advance the rotation of the BOSH DNS CA certificate and its’ corresponding leaf certificates. If the certificate is ready to move to the next step of the rotation, then the Apply Changes step will automatically run the next step of the rotation. If the new versions of the certificates have not been deployed, the Apply Changes step will wait until the next Apply Changes before checking again.

While the commands to generate new BOSH DNS certificates are automated, the feature still requires you to redeploy each VM to distribute the new versions of the certificates.

New versions of the certificates are deployed when also you deploy a new stemcell to a VM. If a new stemcell is not deployed to a VM during Apply Changes, then that VM continues to use the old version of the certificate. This is to ensure that Apply Changes do not take an unexpected long time due to certificate updates.

Forcing new certificates to be deployed

There might be circumstances where you need to deploy new versions of the certificates sooner than your next stemcell upgrade. For example, the BOSH DNS CA certificate might expire in one month, but there isn’t a newer available stemcell. In this case, you can force the Apply Changes to deploy the new versions without needing a new stemcell.

To force an Apply Changes to deploy the new versions of the certificates:

  1. Click Review Pending Changes.
  2. Check the “Force latest BOSH DNS certificates to be used without requiring a stemcell update” check box.
  3. Check the tiles you want to deploy and errands you want to run as normal.
  4. Click Apply Changes.

If there are new versions of certificates that need to be deployed, the Apply Changes action takes as long as if other configuration changes had been made.

When using bosh deploy to manually re-deploy a deployment outside of Tanzu Operations Manager, use the --force-latest-variables flag to force the latest certificate versions to be used.

Monitoring progress of automatic rotation

It is important to continue monitoring the expiration date of the BOSH DNS certificates even when the automatic rotation feature is enabled. You still need to click Apply Changes on each tile and run their corresponding upgrade errands to re-deploy each VM with the new certificate versions. You can check certificate expiration on the Certificates page, or using the API.

In addition to the certificate expiration, a new step appears in the Apply Changes log showing the status of the rotation of the BOSH DNS certificates. This step includes information about which tiles and service instances still need to be deployed.

To determine what still needs to be redeployed in order to distribute the new versions of the certificates:

  1. Click Change Log.
  2. Click Logs for the most recent deployment.
  3. Click the step labelled “Checking BOSH DNS CA certificate status” to expand it.
  4. You will see a text description of what still needs to be deployed.

Once new versions of the certificates have been deployed to all VMs, the next Apply Changes action triggers the next step of the rotation.

Troubleshooting

This section describes scenarios that can occur when using the automatic rotation feature and how to troubleshoot them.

The rotation has not progressed to the next step after an Apply Changes action

The most common cause for the rotation not progressing is you still have VMs that need to be redeployed.

  • Check the “Checking BOSH DNS CA certificate status” step in the Apply Changes logs to see what deployments still need to be deployed.
  • Deployments created outside of Tanzu Operations Manager using manual bosh CLI command must be manually redeployed. Typical examples include if you’ve deployed Concourse using the bosh CLI, or have the Healthwatch tile installed and it has created a bosh-health-check deployment.
  • Ensure that when you deploy the remaining deployments that you either use a newer stemcell or use the force option to ensure that certificates are updated.

Apply Changes action fails with “Failed to run the <STEP NAME> step on ‘/opsmgr/bosh_dns/tls_ca’” or “Cannot detect next rotation step for certificate ‘/opsmgr/bosh_dns/tls_ca’”

Apply Changes fails if an error occurs when you run the next step of the certificate rotation, or it cannot determine what the next step of the rotation can be. In both cases, the BOSH DNS certificates are likely in an state that is unsafe to proceed with the rotation. If this occurs, deactivate the automatic rotation feature and contact Support for assistance.

Running maestro CLI commands fail with safety violations about the BOSH DNS certificates

When the automatic rotation feature is active, the BOSH DNS certificates might no longer be in the same rotation step as other certificates in the platform. This can cause safety violations when you manually run maestro CLI commands, because maestro expects all certificates that it rotates to be in the same rotation step.

If you need to run maestro CLI commands manually and you run into safety violations about BOSH DNS certificates, use --exclude /opsmgr/bosh_dns/tls_ca to exclude BOSH DNS.

check-circle-line exclamation-circle-line close-line
Scroll to top icon