Rotating the Cloud Controller database encryption key

Here is how to rotate the Cloud Controller database (CCDB) encryption key.

The Cloud Controller is the primary API of VMware Tanzu Application Service for VMs (TAS for VMs). It has its own database, the CCDB, in which it stores information about objects in TAS for VMs such as apps. The CCDB encryption key is used to encrypt sensitive data at rest in the CCDB such as app environment variables.

The CCDB encryption key is automatically generated when you first deploy TAS for VMs. You can rotate the CCDB encryption key as described in this topic. For example, you might want to perform this procedure if your existing key is leaked.

Rotate the CCDB encryption key

Caution: Do not modify the Cloud Controller database encryption key field in the TAS for VMs tile. This field is only for use when restoring TAS for VMs from a backup. If you modify this value at a time other than restore, commands such as cf push might fail. Instead, change keys using the Encryption key ledger as described below.

To rotate the CCDB encryption key:

Go to the TAS for VMs tile in Ops Manager and select the Cloud Controller pane.
Review the Encryption key ledger field. If you previously added a key and enabled the Primary check box, deselect the check box. The TAS for VMs deploy fails if there is more than one key with the Primary check box enabled.
In the Encryption key ledger field:

Do not remove any existing keys until after you complete the last step of this procedure. Removing keys now causes the TAS for VMs deploy to fail.
1. Click Add.
2. Enter values for Encryption key label and Encryption key.
3. Select the check box to mark this as your Primary key.
Click Save.
Select the Errands pane.
Set the Rotate CC Database Key errand to On.
Click Save.
Redeploy TAS for VMs. After TAS for VMs deploys and the errand runs successfully, you can remove any previously specified CCDB encryption keys in the ledger.

Troubleshoot failed rotation

This section describes how to troubleshoot errors that you may encounter while rotating the CCDB encryption key.

Error: Encryption key(s) have had their values changed

Error	The TAS for VMs deploy fails with a Cloud Controller error. The Cloud Controller logs include the following message: `Encryption key(s) mykey have had their values changed. Label and value pairs should not change, rather a new label and value pair should be added.`
Cause	TAS for VMs does not support editing key names or values in the ledger. A previously existing key was edited in the TAS for VMs UI. For example, a key with the name `key-name` previously had the value `key-value-1` and now it is `key-value-2`.
Solution	Replace the value for the key with its original value and follow the rotation procedure to add a new key. You may need to find the original value of the key using the steps in View Encryption Keys.

Error: Encryption key(s) are still in use but not present in ‘cc.database_encryption.keys’

Error	The TAS for VMs deploy fails with a Cloud Controller error. The Cloud Controller logs include the following message: `Encryption key(s) mykey are still in use but not present in 'cc.database_encryption.keys'`
Cause	A previous key was removed from the ledger in the TAS for VMs UI before the key rotation errand ran.
Solution	Add the previously used key back to the ledger and redeploy TAS for VMs. You may need to find the value of the previously used key using the steps in View Encryption Keys.

View encryption keys

The section describes how to retrieve the value of an existing or previous encryption key. You may need to do this when troubleshooting a failed key rotation.

Ops Manager actual-installation.yml

The Ops Manager actual-installation.yml file shows properties from the most recent successful deploy. To view the encryption keys listed in this file:

Follow the steps in Modify the Installation Files in Modifying Your Ops Manager Installation and Product Template Files to decrypt the actual-installation.yml file.
Open the actual-installation.yml file and view the cloud_controller_encryption_keys property.

CredHub

To view current and previous encryption keys using CredHub:

Follow the steps in How to access CredHub with the CredHub CLI to log in to the CredHub CLI.
Run the following command to view the five most recent encryption keys:
```
credhub get -n /opsmgr/TAS-GUID/cloud_controller/INDEX/encryption_keys --versions=5
```
Where:
- TAS-GUID is the GUID of your TAS for VMs BOSH deployment.
- INDEX is the zero-based index of the key. This corresponds to the order of the Encryption key ledger in the Ops Manager UI.