In Tanzu Operations Manager 3.0.17+LTS-T, the Director Config pane includes the “Enable automatic rotation of the BOSH DNS CA certificate (experimental)” check box. When you activate this check box, it causes Tanzu Operations Manager to automatically rotate the BOSH DNS CA certificate and associated leaf certificates during the Apply Changes action. The rotated BOSH DNS certificates can be deployed to VMs during the Apply Changes action when new stemcells are also being deployed to those VMs. This allows the BOSH DNS certificates to be rotated gradually over time without user intervention, and without encountering unexpected long Apply Changes action times.
Currently, only the BOSH DNS CA certificate and its associated leaf certificates are rotated by the automatic rotation feature. These certificates are stored in CredHub:
By default, the automatic rotation feature is deactivated. To enable the automatic rotation feature feature:
After you enable the automatic rotation feature, a new step at the start of each Apply Changes tries to advance the rotation of the BOSH DNS CA certificate and its’ corresponding leaf certificates. If the certificate is ready to move to the next step of the rotation, then the Apply Changes step will automatically run the next step of the rotation. If the new versions of the certificates have not been deployed, the Apply Changes step will wait until the next Apply Changes before checking again.
While the commands to generate new BOSH DNS certificates are automated, the feature still requires you to redeploy each VM to distribute the new versions of the certificates.
New versions of the certificates are deployed when also you deploy a new stemcell to a VM. If a new stemcell is not deployed to a VM during Apply Changes, then that VM continues to use the old version of the certificate. This is to ensure that Apply Changes do not take an unexpected long time due to certificate updates.
There might be circumstances where you need to deploy new versions of the certificates sooner than your next stemcell upgrade. For example, the BOSH DNS CA certificate might expire in one month, but there isn’t a newer available stemcell. In this case, you can force the Apply Changes to deploy the new versions without needing a new stemcell.
To force an Apply Changes to deploy the new versions of the certificates:
If there are new versions of certificates that need to be deployed, the Apply Changes action takes as long as if other configuration changes had been made.
bosh deploy to manually re-deploy a deployment outside of Tanzu Operations Manager, use the
--force-latest-variables flag to force the latest certificate versions to be used.
It is important to continue monitoring the expiration date of the BOSH DNS certificates even when the automatic rotation feature is enabled. You still need to click Apply Changes on each tile and run their corresponding upgrade errands to re-deploy each VM with the new certificate versions. You can check certificate expiration on the Certificates page, or using the API.
In addition to the certificate expiration, a new step appears in the Apply Changes log showing the status of the rotation of the BOSH DNS certificates. This step includes information about which tiles and service instances still need to be deployed.
To determine what still needs to be redeployed in order to distribute the new versions of the certificates:
Once new versions of the certificates have been deployed to all VMs, the next Apply Changes action triggers the next step of the rotation.
This section describes scenarios that can occur when using the automatic rotation feature and how to troubleshoot them.
The most common cause for the rotation not progressing is you still have VMs that need to be redeployed.
boshCLI command must be manually redeployed. Typical examples include if you’ve deployed Concourse using the
boshCLI, or have the Healthwatch tile installed and it has created a
Apply Changes fails if an error occurs when you run the next step of the certificate rotation, or it cannot determine what the next step of the rotation can be. In both cases, the BOSH DNS certificates are likely in an state that is unsafe to proceed with the rotation. If this occurs, deactivate the automatic rotation feature and contact Support for assistance.
When the automatic rotation feature is active, the BOSH DNS certificates might no longer be in the same rotation step as other certificates in the platform. This can cause safety violations when you manually run
maestro CLI commands, because
maestro expects all certificates that it rotates to be in the same rotation step.
If you need to run
maestro CLI commands manually and you run into safety violations about BOSH DNS certificates, use
--exclude /opsmgr/bosh_dns/tls_ca to exclude BOSH DNS.