Rotating Certificates (Gemfire for TAS WAN Setups only)

This procedure is used for Gemfire for TAS for VMs, WAN setups only. For the procedure you need to use for all other apps and configurations, return to To rotate the Services TLS CA and its leaf certificates.

This procedure assumes two foundations with service instances: Foundation A and Foundation B. However, it can be applied to installations with more than two foundations. To expand the procedure for additional foundations, wherever Foundation A and Foundation B are mentioned, also apply the instructions to a third foundation (Foundation C), a fourth foundation (Foundation D), and so on.

To rotate the Services TLS CA and its leaf certificates (Gemfire for TAS, WAN setup only)

Check the CredHub Maestro tile compatibility document to ensure that all service tile versions currently deployed are compatible with Maestro.

If you have any service tiles that are not compatible with CredHub Maestro, follow their rotation procedure. Do not follow this procedure. If you do, you might experience extended downtime or data loss.

Collect Existing TLS CA certificates

To collect the existing TLS CA certificates, follow these steps for both Foundations A and B:
1. Log in to CredHub.
2. For Foundation A, Get the current Services TLS CA certificate by capturing the certificate in the output of:
```
credhub get --name=/services/tls_ca -k ca > /tmp/old_ca_foundation_A.txt
```
3. For Foundation B, Get the current Services TLS CA certificate by capturing the certificate in the output of:
```
credhub get --name=/services/tls_ca -k ca > /tmp/old_ca_foundation_B.txt
```
Regenerate the Services TLS CA and mark the latest version of the CA as transitional. This creates a new version of the Services TLS CA and indicates that the latest version is inactive, so that all leaf certificates trust the new CA.
1. On the Services TLS CA, see if your Services TLS CA was CredHub-set. Run:
```
maestro list
```
  After you run this command, you see one Services TLS CA listed per deployment, but it is the same certificate.
2. If the Services TLS CA was CredHub-set, run:
```
maestro regenerate ca --name "/services/tls_ca" --force
```

Collect Newly-Generated TLS CA certificates

To collect the newly-generated TLS CA certificates, follow these steps for both Foundations (A & B):

Before following these steps, ensure that certificates had been regenerated and the previous section Collect Existing TLS CA certificate had been completed.

For Foundation A, Get the new Services TLS CA certificate by capturing the certificate in the output of:

NEW_CA_ID="$(credhub curl -p api/v1/certificates | jq -r '.certificates[] | select(.name == "/services/tls_ca") | .versions[] | select(.transitional == true) | .id')"

credhub get --id "$NEW_CA_ID" -k ca > /tmp/new_ca_foundation_A.txt

For Foundation B, Get the new Services TLS CA certificate by capturing the certificate in the output of:

NEW_CA_ID="$(credhub curl -p api/v1/certificates | jq -r '.certificates[] | select(.name == "/services/tls_ca") | .versions[] | select(.transitional == true) | .id')"

credhub get --id "$NEW_CA_ID" -k ca > /tmp/new_ca_foundation_B.txt

Add All the Certificates to the Tiles

To add all of the certificates (current and new) to the tiles:

For a distributed system with two foundations, the four certificates are:
- The current TLS CA certificate for Foundation A (under /tmp/old_ca_foundation_A.txt)
- The current TLS CA certificate for Foundation B (under /tmp/new_ca_foundation_A.txt)
- The new TLS CA certificate for Foundation A (under /tmp/old_ca_foundation_B.txt)
- The new TLS CA certificate for Foundation B (under /tmp/new_ca_foundation_B.txt)
On Foundation A, paste all certificates (ordering of the certificates does not matter) into BOSH Director, then Security, and Trusted Certificates. Click Save.
On Foundation A, paste all certificates (ordering of the certificates does not matter) into two fields of the VMware Tanzu Application Service tile: Networking, then Certificate Authorities trusted by the Gorouter and Networking, then Certificate Authorities trusted by the HAProxy. Click Save.
On Foundation A, paste all certificates (ordering of the certificates does not matter) into two fields of the Isolation Segment tile, if the environment has this optional tile: Networking, then Certificate Authorities trusted by the Gorouter and Networking, then Certificate Authorities trusted by the HAProxy. Click Save.
On Foundation B, paste all certificates (ordering of the certificates does not matter) into BOSH Director, then Security, and Trusted Certificates. Click Save.
On Foundation B, paste all certificates (ordering of the certificates does not matter) into two fields of the VMware Tanzu Application Service tile: Networking, then Certificate Authorities trusted by the Gorouter and Networking, then Certificate Authorities trusted by the HAProxy. Click Save.
On Foundation B, paste all certificates (ordering of the certificates does not matter) into two fields of the Isolation Segment tile, if the environment has this optional tile: Networking, then Certificate Authorities trusted by the Gorouter and Networking, then Certificate Authorities trusted by the HAProxy. Click Save.

Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the Tanzu GemFire for VMs apps.
Mark the signing version of the Services TLS CA as transitional. Run:
```
maestro update-transitional signing --name "/services/tls_ca"
```
Regenerate all service instance certificates signed by the Services TLS CA. Run:
```
maestro regenerate leaf --signed-by "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the Tanzu GemFire for VMs apps.
Remove the transitional flag from the Services TLS CA. This removes the old, inactive version of the Services TLS CA on the next deployment. Run:
```
maestro update-transitional remove --name "/services/tls_ca"
```
Redeploy:
1. Return to the Tanzu Operations Manager Installation Dashboard.
2. On the Review Pending Changes page, select the Select All Products check box.
3. For each service tile, enable the Upgrade all service instances errand.
  The name of the Upgrade all service instances errand might be slightly different between services.
4. Click Apply Changes.
Rebind and restage the Tanzu GemFire for VMs apps.

Remove the Old Services TLS CA Certificates from Ops Manager

After your apps have reconnected to service instances with the certificates generated by the new CA, both of the old CA certificates must be removed. There will be one old CA certificate for Foundation A and one old certificate for Foundation B.

As a security best practice, you should remove outdated certificates as soon as possible from your deployment. You can schedule this step to a convenient time, because for most deployments you will not lose any deployment functionality if you do not perform this step immediately. For some deployments, you may encounter an error if you create a service instance in a deployment that contains an expired Services TLS CA certificate. For more information, see Cloud Controller fails to create service instance when there is a expired cert installed on the system in the Tanzu Support Knowledge Base.

This procedure involves redeploying all of the VMs in your Tanzu Operations Manager deployment to removed old certificates. The operation can take a long time to complete.

You may apply the changes to both Foundation A and Foundation B at the same time.

Follow these steps twice, once for Foundation A, and once for Foundation B.

Delete the two old CA certificates from the BOSH Director tile: Security, then Trusted Certificates. Remove one old CA certificate for Foundation A and one old certificate for Foundation B. Click Save.
Delete the two old CA certificates from the two fields of the VMware Tanzu Application Service tile:
1. Networking, then Certificate Authorities trusted by the Gorouter and
2. Networking, then Certificate Authorities trusted by the HAProxy. Remove one old CA certificate for Foundation A and one old certificate for Foundation B.
3. Click Save.
Delete the two old CA certificates from the two fields of the Isolation Segment tile, (if the environment has this optional tile):
1. Networking, then Certificate Authorities trusted by the Gorouter and Networking, then Certificate Authorities trusted by the HAProxy. Remove one old CA certificate for Foundation A and one old certificate for Foundation B.
2. Click Save.

Complete the certificate rotation

Navigate back to the Installation Dashboard.
Click Review Pending Changes.
Ensure that all product tiles, including , TAS for VMs [Windows], Isolation Segment, and partner tiles, are selected.
Select only the on-demand services tiles that use TLS, such as MySQL for VMware Tanzu, , VMware Tanzu for Valkey on Cloud Foundry (formerly Redis for VMware Tanzu), and RabbitMQ for VMware Tanzu [VMs].
For each on-demand service tile that uses TLS:
1. Expand the errands.
2. Enable the errand to upgrade all service instances.
  The name of the upgrade all service instances errand may differ slightly between services.
Click Apply Changes.
Continue following the instructions in Rotate CredHub-set CAs.