This topic describes how to rotate certificate authorities (CAs) and leaf certificates using the Ops Manager API. To rotate only leaf certificates, see one of the leaf certificate rotation topics listed in Certificate Rotation Procedures in Overview of Certificate Rotation.
This procedure uses the Ops Manager API to rotate the VMware Tanzu Operations Manager (Ops Manager) root CA and leaf certificates. When you rotate the Ops Manager root CA, the Ops Manager API automatically rotates the BOSH NATS CA as well as CAs stored in CredHub, such as the BOSH DNS CA and the Diego root CA.
This procedure includes three redeploys. The first deploy adds the new CA certificates to Ops Manager components. The second deploy replaces the old leaf certificates with the new leaf certificates. The third deploy removes the old CA certificates, leaving only the new CA certificates.
Before you rotate certificates:
You must have Ops Manager v2.10.
You must reset any certificates that were manually set in CredHub on Ops Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, see Reviewing Manually Set Certificates in CredHub.
You cannot have any version of VMware Tanzu Kubernetes Grid Integrated Edition installed and rotate CredHub-managed certificates with the Ops Manager API. Instead, you can follow this procedure to rotate certificates managed by Ops Manager. Then, to rotate certificates managed in CredHub, see Advanced Certificate Rotation with CredHub Maestro.
Your product tile must have one of the following versions installed to rotate CredHub-managed certificates with the Ops Manager API:
If you have an earlier version of a product tile listed above, use this procedure to rotate certificates managed only by Ops Manager. Then, to rotate certificates managed in CredHub, see Advanced Certificate Rotation with CredHub Maestro.
Note: You can set a value to override the duration for certificates within the BOSH Director tile. VMware recommends that you set the value before starting a certificate rotation. For more information, see Overriding Duration for Certificates.
To rotate CAs, configurable leaf certificates, and non-configurable leaf certificates:
Caution: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment.
Add a new Ops Manager root CA. This step also automatically adds new CAs for all CAs visible to the Ops Manager API. For more information, see Add New CAs below.
Activate the Ops Manager root CA. This step also automatically activates any other new CAs added in the previous step. For more information, see Activate the CAs below.
Rotate configurable leaf certificates. For more information, see Rotate Configurable Leaf Certificates below.
Rotate non-configurable leaf certificates. For more information, see Rotate Non-Configurable Leaf Certificates from the New CAs below.
Delete the old root CA and mark the old CAs in CredHub as inactive. For more information, see Delete the Old Root CA and Mark Old CAs in CredHub as Inactive below.
Caution: The Services TLS CA, /services/tls_ca, cannot be rotated using this procedure. For information about how to rotate the Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates in Advanced Certificate Rotation with CredHub Maestro.
This section describes how to add a new root CA for Ops Manager. The new root CA can be an Ops Manager-generated CA or your own custom CA.
This procedure also automatically regenerates the BOSH NATS CA as well as CAs stored in CredHub, such as the BOSH DNS CA and the Diego root CA.
Caution: The Ops Manager API also excludes from rotation the CAs for certain versions of the VMware Tanzu SQL [MySQL for VMs], Redis for VMware Tanzu, VMware Tanzu RabbitMQ, and VMware Tanzu GemFire tiles and IPsec for VMware Tanzu. For information about rotating these CAs, see the documentation for those products.
To add new CAs:
If you have not already done so, follow the procedure in Using Ops Manager API to target and authenticate with the Ops Manager UAA server. Record your Ops Manager access token, and use it for UAA-ACCESS-TOKEN in the following steps.
Note: When you record your Ops Manager access token, remove any newline characters such as \n.
Use Ops Manager to generate a new CA, or add your own custom CA:
Note: Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in Ops Manager.
To use your own custom CA: Call the Ops Manager API certificate_authorities endpoint by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
-X POST \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-H "Content-Type: application/json" \
-d '{"cert_pem": "-----BEGIN CERTIFICATE-----CUSTOM-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----RSA-KEY"}' \
-i
Where:
OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Ops Manager deployment.CUSTOM-CERTIFICATE is your custom CA.RSA-KEY is your RSA key.UAA-ACCESS-TOKEN is your UAA access token.If the command succeeds, the API returns a response that includes the new CA. For example:
HTTP/1.1 200 OK
{
"certificate_authorities": [
{
"guid": "f7bc18f34f2a7a9403c3",
"issuer": "CUSTOM-CERTIFICATE",
"created_on": "2017-01-09",
"expires_on": "2021-01-09",
"active": true,
"cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
}
]
}
To use an Ops Manager-generated CA: Call the Ops Manager API generate endpoint by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/generate" \
-X POST \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-H "Content-Type: application/json" \
-d '{}' \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN is your UAA access token.If the command succeeds, the API returns a response that includes the new CA. For example:
HTTP/1.1 200 OK
{
"guid": "f7bc18f34f2a7a9403c3",
"issuer": "VMware",
"created_on": "2017-01-19",
"expires_on": "2021-01-19",
"active": false,
"cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
-----END EXAMPLE CERTIFICATE-----"
}
Confirm that the new CA is added by listing all the CAs for Ops Manager. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
-X GET \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN is your UAA access token.The API call returns the GUID of the newly added CA. For example:
HTTP/1.1 200 OK
{
"certificate_authorities": [
{
"guid": "f7bc18f34f2a7a9403c3",
"issuer": "VMware",
"created_on": "2017-01-09",
"expires_on": "2021-01-09",
"active": true,
"cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
}
{
"guid": "a8ee01e33e3e3e3303e3",
"issuer": "VMware",
"created_on": "2017-04-09",
"expires_on": "2021-04-09",
"active": false,
"cert_pem": "-----BEGIN CERTIFICATE-----\nzBBBC+eAAAe1gAwAAAeZ....etc"
}
]
}
Identify the newly added CA, which has active set to false. Record its GUID.
Configure the BOSH Director to re-create VMs:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Navigate to https://OPS-MANAGER-FQDN in a browser, where OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
Log in to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate VMs deployed by the BOSH Director checkbox. This propagates the new CA to the BOSH Agent on each VM deployed by the BOSH Director, which enables the new leaf certificates to trust the new CA.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade all service instances and Recreate all service instances errands may be slightly different between services.
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Click Review Pending Changes.
If you have service tiles installed and the service tile does not have the Recreate all service instances errand, manually push BOSH Agent certificate updates to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Activate the CAs.
To activate the CAs that you added in Add New CAs above:
Call the Ops Manager API activate endpoint by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/CERTIFICATE-GUID/activate" \
-X POST \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-H "Content-Type: application/json" \
-d '{}' \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.CERTIFICATE-GUID is the GUID of your CA that you retrieved in the previous section.UAA-ACCESS-TOKEN is your UAA access token.The API returns a successful response:
HTTP/1.1 200 OK
List your CAs to confirm that the new Ops Manager root CA is active. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
-X GET \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN is your UAA access token.Examine the response to ensure that your new root CA has the active property set to true.
Any configurable leaf certificates generated from the Ops Manager root CA must be rotated. These are the leaf certificates that you generated using the Generate RSA Certificate button in the Ops Manager UI. For example, see Providing a Certificate for Your TLS Termination Point in the TAS for VMs documentation.
To regenerate and rotate each configurable leaf certificate that expires soon:
Find the text field where the leaf certificate is configured within the Ops Manager UI. You might need to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:
product_guid field in the Ops Manager API output can help identify the tile in which the certificate is configured. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the TAS for VMs tile.property_reference field in the Ops Manager API output can help identify the configuration pane in which the certificate is configured. For example, the uaa.service_provider_key_credentials property is configured in the UAA pane of the TAS for VMs tile.Copy a new value for the leaf certificate into the text field or generate a new leaf certificate using the Generate RSA Certificate button.
Click Save at the bottom of each pane in which you added new leaf certificates.
After activating the new Ops Manager root CA and other CAs, you must rotate non-configurable leaf certificates from the new CAs.
Caution: The Ops Manager API also excludes from rotation the CAs for certain versions of the VMware Tanzu SQL [MySQL for VMs], Redis for VMware Tanzu, VMware Tanzu RabbitMQ, and VMware Tanzu GemFire tiles and IPsec for VMware Tanzu. For information about rotating these CAs, see the documentation for those products.
To rotate non-configurable leaf certificates:
Call the Ops Manager API regenerate endpoint by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/active/regenerate" \
-X POST \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-H "Content-Type: application/json" \
-d '{}' \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN is your UAA access token.The API returns a successful response:
HTTP/1.1 200 OK
Configure the BOSH Director to re-create VMs:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Navigate to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate VMs deployed by the BOSH Director checkbox. This propagates the new CAs to all VMs deployed by the BOSH Director to prevent downtime.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade all service instances and Recreate all service instances errands may be slightly different between services.
If the service tile has the Recreate all service instances errand:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Click Review Pending Changes.
If you have service tiles installed and the service tile does not have the Recreate all service instances errand:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Manually push BOSH Agent certificate updates to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.
When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Delete the Old Root CA and Mark Old CAs in CredHub as Inactive.
After activating new CAs and rotating leaf certificates from the new CAs, you must:
Within CredHub, the leaf certificates that are generated when you rotate your deployment’s root CA are associated with both the new and old versions of the root CA. Deleting the old root CA ensures that these leaf certificates are only associated with the new root CA.
The Ops Manager API call you use to delete your deployment’s old root CA also marks the old versions of all CAs stored in CredHub as inactive, including those not generated by the old root CA.
Note: This procedure does not delete leaf certificates stored in CredHub. It marks CredHub leaf certificates as transitional, which indicates that the leaf certificate is inactive. To delete CredHub leaf certificates, use the CredHub Maestro garbage collection feature. For information about CredHub Maestro certificate garbage collection, see Delete Inactive Certificate Versions in Advanced Certificate Rotation with CredHub Maestro.
Caution: Before you delete the old CA, ensure that you reviewed the Review Pending Changes page and clicked Apply Changes as part of Rotate Non-Configurable Leaf Certificates from the New CAs above.
To delete the old Ops Manager root CA and mark the old CredHub CAs as inactive:
List your root CAs to retrieve the GUID of the old, inactive Ops Manager root CA. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
-X GET \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN is your UAA access token.Delete the old Ops Manager root CA by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/OLD-CERTIFICATE-GUID" \
-X DELETE \
-H "Authorization: Bearer UAA-ACCESS-TOKEN" \
-i
Where:
OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.OLD-CERTIFICATE-GUID is the GUID of the old Ops Manager root CA.UAA-ACCESS-TOKEN is your UAA access token.The API returns a successful response:
HTTP/1.1 200 OK
Configure the BOSH Director to re-create VMs:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Navigate to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate VMs deployed by the BOSH Director checkbox.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade all service instances and Recreate all service instances errands may be slightly different between services.
If the service tile has the Recreate all service instances errand:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Click Review Pending Changes.
If you have service tiles installed and the service tile does not have the Recreate all service instances errand:
Note: If you are using Ops Manager v2.10.20 or later and all VMs are deployed with stemcell Xenial 621.171 or later and Windows 2019.41 or later, you do not need to re-create VMs.
Manually push BOSH Agent certificate updates to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT re-create
Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.
Some certificates are not rotated when using the standard rotation procedure. The following table lists certificates that use a separate, specialized rotation procedure and provides links to documentation and Knowledge Base articles to help you rotate these certificates.
| Certificate | Tile | Procedure |
|---|---|---|
/services/tls_ca |
n/a | Rotating the Services TLS CA and Its Leaf Certificates |
/telemetry-ca-cert |
Telemetry v1.2.1 and earlier | How to rotate the telemetry-ca-cert in Operations Manager |
/services/tls_leaf |
TAS for VMs v2.10.0 – v2.10.13 | Rotate /services/tls_leaf manually |
The Ops Manager API invokes CredHub Maestro when rotating certificates. If a certificate rotation API command is unsafe, CredHub Maestro stops the command and returns one or more safety violations.
For example, CredHub Maestro stops a certificate rotation API command if you try to perform certificate rotation steps in the wrong order. Because performing these steps in the wrong order can make your deployment unstable or cause downtime, CredHub Maestro stops the command and returns an error message.
For information about how to troubleshoot safety violation errors that are returned when rotating certificates, see Troubleshooting CredHub Maestro Safety Violations During Certificate Rotation.
