Troubleshooting CredHub Maestro safety violations

You can troubleshoot safety violation errors that you encounter when rotating certificates with either the VMware Tanzu Operations Manager API or the CredHub Maestro CLI.

About CredHub Maestro safety checks

CredHub Maestro completes basic safety checks when rotating certificates to prevent unsafe operations.

The Tanzu Operations Manager API runs CredHub Maestro when rotating certificates. If a certificate rotation API command is unsafe, CredHub Maestro stops the command and returns one or more safety violations.

CredHub Maestro returns safety violations if a rotation step is run out of order. For example, if you generate a new root certificate authority (CA), and attempt to activate it before clicking Apply Changes. Performing steps out of the correct order during certificate rotations can make your deployment unstable or risk downtime.

For information about certificate rotation using the Tanzu Operations Manager API, see Overview of Certificate Rotation.

For information about using the CredHub Maestro CLI, see Getting started with CredHub Maestro.

Skip safety checks

To skip CredHub Maestro safety checks in Tanzu Operations Manager:

In API calls - Pass skip_safety_check=true as a parameter to certificate rotation API endpoints.
In the CredHub Maestro CLI - Specify the --skip-safety-check flag when running certificate rotation commands.

Skipping safety checks is strongly discouraged. Use the option only if absolutely necessary. For example, you might want to skip safety checks if your CA has already expired and your apps are experiencing downtime. Skipping safety checks can help bring your apps back online faster. If certificates have expired, contact Broadcom Support for guidance.

Troubleshoot certificate rotation safety violations

This section lists known safety violations that might occur when using the Tanzu Operations Manager API or the CredHub Maestro CLI to rotate certificates. Each table includes the safety violation message, describes the safety violation, and presents possible solutions to resolve the underlying cause.

Multiple signing versions of a CA

Safety Violation there is more than one signing version of a certificate authority

Related Commands

Safety Violation	there is more than one signing version of a certificate authority
Related Commands	This safety violation might appear after calling the following Tanzu Operations Manager API endpoints: `POST /api/v0/certificate_authorities/generate` `POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate` `POST /api/v0/certificate_authorities/active/regenerate` `DELETE /api/v0/certificate_authorities/{certificate_authority_guid}` Or after running one of the following commands: `maestro regenerate ca` `maestro update-transitional signing` `maestro regenerate leaf` `maestro update-transitional remove`
Cause	CredHub Maestro expects that only one version of a CA is used to sign active leaf certificates. During the rotation process, Maestro moves forward from one CA version to another in bulk, but is not equipped to handle mixed-mode operations where multiple CA versions are actively in use.
Solution	There are two possible problems, with different solutions. To identify and resolve the root issue: Run `maestro topology --name CERTIFICATE-NAME`, where `CERTIFICATE-NAME` is the name and full path of the certificate returned in the error message. Check the command line output for the following: If the `signs` field contains a child certificate that has two versions set to `active: true` with different product tile deployment names: Go to the Tanzu Operations Manager Installation Dashboard. Click Review Pending Changes. Select the product tile that is associated with the older certificate version. Click Apply Changes. If there are multiple child certificates and their active versions have different `signed-by` versions: Run `maestro regenerate leaf --signed-by PARENT-CA-NAME`, where `PARENT-CA-NAME` is the name of the signing CA. Go to the Tanzu Operations Manager Installation Dashboard. Click Review Pending Changes. Select all products. Click Apply Changes.

This safety violation might appear after calling the following Tanzu Operations Manager API endpoints:

POST /api/v0/certificate_authorities/generate
POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate
POST /api/v0/certificate_authorities/active/regenerate
DELETE /api/v0/certificate_authorities/{certificate_authority_guid}

Or after running one of the following commands:

maestro regenerate ca
maestro update-transitional signing
maestro regenerate leaf
maestro update-transitional remove

Cause CredHub Maestro expects that only one version of a CA is used to sign active leaf certificates. During the rotation process, Maestro moves forward from one CA version to another in bulk, but is not equipped to handle mixed-mode operations where multiple CA versions are actively in use.

Solution

There are two possible problems, with different solutions. To identify and resolve the root issue:

Run maestro topology --name CERTIFICATE-NAME, where CERTIFICATE-NAME is the name and full path of the certificate returned in the error message.
Check the command line output for the following:
- If the signs field contains a child certificate that has two versions set to active: true with different product tile deployment names:
  1. Go to the Tanzu Operations Manager Installation Dashboard.
  2. Click Review Pending Changes.
  3. Select the product tile that is associated with the older certificate version.
  4. Click Apply Changes.
- If there are multiple child certificates and their active versions have different signed-by versions:
  1. Run maestro regenerate leaf --signed-by PARENT-CA-NAME, where PARENT-CA-NAME is the name of the signing CA.
  2. Go to the Tanzu Operations Manager Installation Dashboard.
  3. Click Review Pending Changes.
  4. Select all products.
  5. Click Apply Changes.

Transitional version of CA detected

Safety Violation	certificate authorities can not have a transitional version
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/generate` Tanzu Operations Manager API endpoint or after running the `maestro regenerate ca` command.
Cause	CredHub Maestro has detected an existing transitional CA version. This means that the certificate rotation process has already started.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Multiple active child certificates

Safety Violation active child certificate version is not the latest non-transitional version or more than one active version exists

Related Commands

Safety Violation	active child certificate version is not the latest non-transitional version or more than one active version exists
Related Commands	This safety violation might appear after calling the following Tanzu Operations Manager API endpoints: `POST /api/v0/certificate_authorities/generate` `POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate` Or after running the following commands: `maestro regenerate ca` `maestro update-transitional signing`
Cause	This error usually occurs if you have not applied your changes to all the deployments that are required to propagate the latest version of the child certificate. A child certificate can refer to a leaf certificate or an intermediate certificate.
Solution	To resolve this error: Go to the Tanzu Operations Manager Installation Dashboard. Click Review Pending Changes. Select all products. Click Apply Changes. After the deployment finishes, confirm that the latest version is propagated. Run `maestro topology --name CERTIFICATE-NAME`, where `CERTIFICATE-NAME` is the name of the certificate returned in the error message.

This safety violation might appear after calling the following Tanzu Operations Manager API endpoints:

POST /api/v0/certificate_authorities/generate
POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate

Or after running the following commands:

maestro regenerate ca
maestro update-transitional signing

Cause This error usually occurs if you have not applied your changes to all the deployments that are required to propagate the latest version of the child certificate. A child certificate can refer to a leaf certificate or an intermediate certificate.

Solution

To resolve this error:

Go to the Tanzu Operations Manager Installation Dashboard.
Click Review Pending Changes.
Select all products.
Click Apply Changes.
After the deployment finishes, confirm that the latest version is propagated. Run maestro topology --name CERTIFICATE-NAME, where CERTIFICATE-NAME is the name of the certificate returned in the error message.

Latest CA is not transitional

Safety Violation	certificate authorities latest version is not transitional
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate` Tanzu Operations Manager API endpoint or after running the `maestro update-transitional signing` command.
Cause	The latest version of each CA is expected to be transitional before the latest CA version becomes the future signing version. This allows trust to be incrementally added to the latest version before it ever gets used.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. If you believe you are running the steps in the correct order, it is possible that a new CA has been created in CredHub after the rotation process has begun. To work around this situation, take one of the following actions: If the CA is new, exclude this CA by adding the `--exclude=CA-NAME` option for future commands during this rotation. For example, `maestro update-transitional signing --exclude=CA-NAME`. If you need to rotate the CA now, re-run the earlier CredHub Maestro rotation steps `maestro regenerate ca CA-NAME`, where `CA-NAME` is the name of a CA that was not originally present. Repeat this step for all CAs not originally present. This gets all targeted CAs into the same state. Then, run the `maestro update-transitional signing` step again.

Signing version must be transitional

Safety Violation	signing version has to be transitional if there is a transitional certificate authority
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/active/regenerate` Tanzu Operations Manager API endpoint or after running the `maestro regenerate leaf` command.
Cause	Before regenerating new leaf certificates, all the existing leaf certificates must only be signed by older versions of their appropriate CAs. These older CA versions are marked as transitional, which indicates that they are inactive but still need to be trusted by clients. This means the new leaf certificates are signed by the newer CA version and clients trust both the old and new CA versions.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. If you believe you are running the steps in the correct order, it is possible that a new leaf was issued while the rotation process was in progress. You might also see the `there is more than one signing version` error described in Multiple Signing Versions of a CA. If this safety violation error appears, follow the mitigation steps for that error.

Active child must trust signing CA

Safety Violation	active child certificate does not trust the certificate authority that will regenerate it
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/active/regenerate` Ops Manager API endpoint or after running the `maestro regenerate leaf` command.
Cause	This safety violation occurs when the active leaf certificate is not signed by the CA version that would sign new leaf certificates, and also the leaf does not include that CA in the list of trusted CAs. This situation can occur after running `maestro regenerate ca` or `maestro update-transitional` while skipping safety checks, or using the `credhub` CLI or API to manually generate a new version of the CA. During a CA rotation, if the active version of the leaf certificate does not trust the signing version of the CA, then trust will be broken when deploying a regenerated leaf certificate.
Solution	The repair procedure for this safety violation can vary depending on the exact state of the CA and leaf certificates. Contact Support for guidance.

CA must have a non-transitional version

Safety Violation	certificate authorities must have a non-transitional version
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/active/regenerate` Ops Manager API endpoint or after running the `maestro regenerate leaf` command.
Cause	This safety violation occurs when the CA that would sign the leaf certificate has a single version, and that version is marked as transitional. This situation can occur after running `maestro garbage-collect ca --force`, or using the `credhub` CLI or API to delete CA versions. When regenerating leaf certificates, if there is only a transitional version of the CA, then regenerating leaf certificates will fail.
Solution	Remove the transitional flag from the CA certificate using the Update Transitional Version CredHub API endpoint and setting the version to `null`. `credhub curl -X PUT -p /api/v1/certificates/CA_CERTIFICATE_ID/update_transitional_version -d '{ "version": null }'`

Signed by non-latest CA version

Safety Violation	active child certificate is signed by a certificate authority that is not the latest version
Related Commands	This safety violation might appear after calling the `POST /api/v0/certificate_authorities/active/regenerate` Ops Manager API endpoint or after running the `maestro regenerate leaf` command.
Cause	This safety violation occurs when the existing leaf certificates are not signed by the CA version that will sign the new leaf certificate versions, and they are also not signed by the current CA version that is marked as transitional (if one exists). This situation can occur after running `maestro regenerate ca` or `maestro update-transitional` while skipping safety checks, or using the CredHub CLI or API to generate new CA versions or change the transitional flag of existing CAs. During a CA rotation, if the leaf certificate does not trust both the old and new versions of the CA, then trust will be broken when deploying a regenerated leaf certificate.
Solution	If the active leaf certificate is signed by one of the two most recent versions of the CA, the leaf certificate trusts both CA versions, and neither CA version is marked transitional, then use the Update Transitional Version CredHub API endpoint to set the older CA version as transitional. `credhub curl -X PUT -p /api/v1/certificates/CA_CERTIFICATE_ID/update_transitional_version -d '{ "version": "OLDER_CA_VERSION_ID" }'` Then continue the CA rotation starting on the `maestro regenerate leaf` step of the rotation procedure. For other scenarios, please reach out to Support for guidance.

Active certificates not signed by latest CA detected

Safety Violation	active child certificate version is signed by a certificate authority that is not the latest version
Related Commands	This safety violation might appear after calling the `DELETE /api/v0/certificate_authorities/{certificate_authority_guid}` Tanzu Operations Manager API endpoint or after running the `maestro update-transitional remove` command.
Cause	There are still active intermediate CAs or leaf certificates that are not signed by the latest CA version.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Stale leaf certificates detected

Safety Violation	active certificate version is not the latest non transitional version
Related Commands	This safety violation might appear after calling the `DELETE /api/v0/certificate_authorities/{certificate_authority_guid}` Tanzu Operations Manager API endpoint or after running the `maestro update-transitional remove` command.
Cause	There are still stale leaf certificates in use with new versions ready to be deployed.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Transitional CA detected as latest CA

Safety Violation	transitional certificate authority is the latest version
Related Commands	This safety violation might appear after calling the `DELETE /api/v0/certificate_authorities/{certificate_authority_guid}` Tanzu Operations Manager API endpoint or after running the `maestro update-transitional remove` command.
Cause	The latest version of one or more CAs is transitional, when CredHub Maestro is expecting the transitional version to be an older version.
Solution	Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Tanzu Operations Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. It is possible that you are attempting to halt a rotation in the early stages, likely after new transitional CAs have been issued but not yet used. Continue with the rotation at this point. If continuing with the rotation is impossible due to other deployment needs, reach out to Support for guidance.