Encrypt Secrets in an etcd Database

This topic describes how to create and use a Kubernetes profile to encrypt a cluster’s etcd database with the Tanzu Kubernetes Grid Integrated Edition Command Line Interface (TKGI CLI).

For more information and other uses of Kubernetes profiles, see Using Kubernetes Profiles.

Overview

To encrypt a cluster’s etcd database:

1. Create Kubernetes Profile
2. Create Kubernetes Cluster
3. Verify Your Data is Encrypted
4. Ensure Existing Data is Encrypted

To rotate a etcd database encryption key:
1. Rotate Encryption Key for Secrets in etcd Database

Create Kubernetes Profile

To create a new create Kubernetes profile:

1. Create a new a base64-encoded, 32-byte random key string to use as a secret.

   • On Linux or MacOS, you can generate the secret by running:

     head -c 32 /dev/urandom | base64
     

     This command returns a base64-encoded, 32-byte key string, for example: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=

2. To create an encryption provider configuration file, create a file named encryption-provider-config.yml containing the following content:

   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
       - secrets
       providers:
       - aescbc:
           keys:
           - name: key1
             secret: BASE-64-ENCODED-SECRET
       - identity: {}
   

   Where BASE-64-ENCODED-SECRET is the secret key string created in the last step.

   Note: To read unencrypted secrets, include the identity provider as the last provider as shown above.

3. To create a Kubernetes profile configuration file that customizes the kube-apiserver using your encryption provider configuration file, create a JSON file containing the following content:

   {
     "name": "PROFILE-TITLE",
     "description": "PROFILE-DESC",
     "customizations": [
       {
         "component": "kube-apiserver",
         "arguments": {
         },
         "file-arguments": {
           "encryption-provider-config": "/LOCAL-DIR/encryption-provider-config.yml"
         }
       }
     ]
   }
   

   Where:

   • PROFILE-TITLE is the name of your profile, for example profile1.
   • PROFILE-DESC is the description you want to use for the profile.
   • LOCAL-DIR is the directory containing your encryption provider configuration file.

4. To create a Kubernetes profile based on your profile configuration file, use the TKGI CLI:

   tkgi create-k8s-profile PROFILE-PATH
   

   Where PROFILE-PATH is the path and filename of the JSON profile file you created in the step above.
   
   For example:

   $ tkgi create-k8s-profile /tmp/profile1.json
   Kubernetes profile profile1 successfully created
   

Create Kubernetes Cluster

To create a Kubernetes cluster based on a Kubernetes profile:

1. To create a cluster based on a Kubernetes profile, use the TKGI CLI:

   tkgi create-cluster CLUSTER-NAME -e EXTERNAL-HOSTNAME -p small -n 1 --kubernetes-profile K8S-PROFILE
   

   Where:

   • CLUSTER-NAME is the name to apply to the new cluster.
     

     Note: Use only lowercase characters when naming your cluster if you manage your clusters with Tanzu Mission Control (TMC). Clusters with names that include an uppercase character cannot be attached to TMC.

   • EXTERNAL-HOSTNAME is the address to use to access Kubernetes API.
   • K8S-PROFILE is the Kubernetes profile name. For more information, see tkgi create-cluster in TKGI CLI.

   Running this command restarts your kube-apiserver with your encryption provider configuration file set as the --encryption-provider-config parameter.

   For example:

   $ tkgi create-cluster cluster1 -e cluster1-internal.com -p small -n 1 --kubernetes-profile profile1  
   TKGI Version:             1.18.5-build.10  
   Name:                     cluster1  
   K8s Version:              1.27.15  
   Plan Name:                small  
   UUID:                     22f78823-0b70-4684-be2f-8457d6f3b1f1  
   Last Action:              CREATE  
   Last Action State:        in progress  
   Last Action Description:  Creating cluster  
   Kubernetes Master Host:   cluster1-internal.com  
   Kubernetes Master Port:   8443  
   Worker Nodes:             1  
   Kubernetes Master IP(s):  In Progress  
   Network Profile Name:  
   Kubernetes Profile Name:  profile1  
   Tags:  
   

Ensure Existing Data is Encrypted

If you have existing data, the data has been stored without being encrypted.

To encrypt your existing data:

1. Run the following:

   kubectl get secrets --all-namespaces -o json | kubectl replace -f -
   

   This command reads all secrets, applies encryption, and saves the data encrypted. For more information, see Ensure all secrets are encrypted in the Kubernetes documentation.

2. Be sure to complete the steps in Verify Your Data is Encrypted below.

If you use a Key Management Service (KMS) provider, see Using a KMS provider for data encryption, and Encrypting Secret Data at Rest in the Kubernetes documentation.

Verify Your Data is Encrypted

After restarting the kube-apiserver, all existing and later-added data stored in the etcd database is encrypted.

To ensure the etcd data has been encrypted, create and store a test secret and then retrieve it:

1. To create and store a test secret:

   • To use a basic create secret command:

     kubectl create secret generic SECRET-NAME -n NAMESPACE --from-literal=KEY-NAME=KEY-VALUE
     

     Where:

     • SECRET-NAME is the name to apply to the secret.
     • NAMESPACE is the namespace.
     • KEY-NAME is the name of the key.
     • KEY-VALUE is the value to encrypt.

     For example:

     kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
     

   • To create a secret for vSphere, use your csi-vsphere.conf configuration file that contains details for connecting to vSphere:

     kubectl create secret generic SECRET-NAME --from-file=csi-vsphere.conf  --namespace=SECRET-NAMESPACE
     

     Where:

     • SECRET-NAME is the name to apply to the secret.
     • SECRET-NAMESPACE is the secret namespace, for example velero.

     For example:

     kubectl create secret generic secret1 --from-file=csi-vsphere.conf --namespace=velero
     

   For more information, see create secret generic in the Kubernetes documentation.

2. To read the test secret out of etcd, use the etcdctl command line:

   ETCDCTL_API=3 etcdctl get /registry/secrets/default/SECRET-NAME [ETCD-ARGS] | hexdump -C
   

   Where:

   • SECRET-NAME is the name to apply to the secret.
   • ETCD-ARGS are the additional arguments for connecting to the etcd server.

   For example:
   To get the secret and view it after you SSH into your control plane node with admin privileges:

   ETCDCTL_API=3 /var/vcap/jobs/etcd/bin/etcdctl get /registry/secrets/default/secret1
   

3. Review the output and verify the stored test secret is prefixed with k8s:enc:aescbc:v1:.

   For example:

   master/3f9c5ca9-a2b1-469c-9007-6fdd0844a5ec:/var/vcap/bosh_ssh/bosh_2f4aaa514474422# ETCDCTL_API=3 /var/vcap/jobs/etcd/bin/etcdctl get /registry/secrets/default/secret1
   
   /registry/secrets/default/secret1
   k8s:enc:aescbc:v1:key1:����@�8�����A������2cM7������uL�/
   

   The k8s:enc:aescbc:v1: prefix indicates the aescbc provider has encrypted the resulting data.

4. To retrieve the stored test secret:

   kubectl describe secret secret1 -n default
   

   Where: SECRET-NAME is the name of the secret.
   
   For example:

   kubectl describe secret secret1 -n default
   

5. Review the output and verify the secret is correctly decrypted. The mydata content is returned encoded. For information about decoding mydata, see decoding a secret.

Rotate Encryption Key for Secrets in etcd Database

This section describes how to use encryption provider config to rotate a key. For highly-available deployments running multiple kube-apiserver processes, changing a secret without incurring downtime requires a multi-step process.

WARNING: Update the Kubernetes profile only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.

To rotate an encryption key for a secret in an etcd database:

1. Generate a new key. For more information, see Create Kubernetes Profile above.

2. Add the new key as the second key entry for the current provider in your encryption-provider-config.yml keys: list.
   
   For example:

   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
       - secrets
       providers:
       - aescbc:
           keys:
           - name: key1
             secret: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=
           - name: key2
             secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
       - identity: {}
   

   Note: To read unencrypted secrets, include the identity provider as the last provider, as shown above.

3. Create a Kubernetes profile configuration file that references the modified encryption-provider-config.yml.
   
   For example:

   {
     "name": "profile2",
     "description": "Testing profile two",
     "customizations": [
       {
         "component": "kube-apiserver",
         "arguments": { },
         "file-arguments": {
           "encryption-provider-config": "/tmp/encryption-provider-config.yml"
         }
       }
     ]
   }
   

   For information about creating a Kubernetes profile configuration file, see Create Kubernetes Profile above.

4. Use the TKGI CLI to create a profile based on the configuration file.

   For example:

   $ tkgi create-k8s-profile /tmp/profile2.json  
   Kubernetes profile profile2 successfully created  
   

   For information about using the TKGI CLI to create a Kubernetes profile, see Create Kubernetes Profile above.

5. If you are updating a cluster that uses a public cloud CSI driver, see Limitations on Using a Public Cloud CSI Driver in Release Notes for additional requirements.

6. Update the cluster with this new profile.

   tkgi update-cluster CLUSTER-NAME --kubernetes-profile PROFILE-TITLE
   

   Where:

   • CLUSTER-NAME is the name to apply to the new cluster.
   • PROFILE-TITLE is the name of your profile, for example profile2.

   For example:

   $ tkgi update-cluster cluster1 --kubernetes-profile profile2  
   
   Update summary for cluster cluster1:  
   Kubernetes Profile Name: profile2  
   Are you sure you want to continue? (y/n): y  
   Use 'tkgi cluster cluster1' to monitor the state of your cluster  
   

   Running this command restarts your kube-apiserver with your encryption provider configuration file set as the --encryption-provider-config parameter. This ensures that each server can decrypt using the new key.

7. Edit the encryption provider configuration file so that it lists the new key as the first entry in the keys: property, swapping its position with the old key.

   For example:

   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
       - secrets
       providers:
       - aescbc:
           keys:
           - name: key2
             secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
           - name: key1
             secret: jHc3NMp7s7T7JoJuZF7NUSkHVYCSikJCNJ+LrltbkJk=
       - identity: {}
   

8. Create a Kubernetes profile configuration file that references the modified encryption-provider-config.yml.

   For example:

   {
     "name": "profile3",
     "description": "Testing profile three",
     "customizations": [
       {
         "component": "kube-apiserver",
         "arguments": { },
         "file-arguments": {
           "encryption-provider-config": "/tmp/encryption-provider-config.yml"
         }
       }
     ]
   }
   

9. Use the TKGI CLI to create a profile based on the new Kubernetes profile configuration file.

   For example:

   $ tkgi create-k8s-profile /tmp/profile3.json  
   Kubernetes profile profile2 successfully created  
   

10. Update the cluster with the new profile.

    For example:

    $ tkgi update-cluster cluster1 --kubernetes-profile profile3  
    
    Update summary for cluster cluster1:  
    Kubernetes Profile Name: profile3  
    Are you sure you want to continue? (y/n): y  
    Use 'tkgi cluster cluster1' to monitor the state of your cluster  
    

    This restarts the kube-apiserver processes to ensure that each server now encrypts using the new key.

11. To encrypt all existing secrets with the new key:

    kubectl get secrets --all-namespaces -o json | kubectl replace -f - 
    

12. Back up etcd to snapshot etcd with the new key.

13. Edit the encryption-provider-config.yml again to remove the old decryption key from the config keys: list.

    For example:

    apiVersion: apiserver.config.k8s.io/v1
    kind: EncryptionConfiguration
    resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            Keys:
            - name: key2
              secret: MNI7xeE48/1dH+pE5LLHblhId6AjbzdN2I6rubh8AfE=
        - identity: {}
    

14. Create a new Kubernetes profile configuration file that references the modified encryption-provider-config.yml.

    For example:

    {
      "name": "profile4",
      "description": "Testing profile four",
      "customizations": [
        {
          "component": "kube-apiserver",
          "arguments": { },
          "file-arguments": {
            "encryption-provider-config": "/tmp/encryption-provider-config.yml"
          }
        }
      ]
    }
    

    For information about creating a Kubernetes profile configuration file, see Create Kubernetes Profile above.

15. Use the TKGI CLI to create a profile based on the new Kubernetes profile configuration file.

    For example:

    $ tkgi create-k8s-profile /tmp/profile4.json  
    Kubernetes profile profile4 successfully created  
    

16. Update the cluster with the new profile.

    For example:

    $ tkgi update-cluster cluster1 --kubernetes-profile profile4  
    
    Update summary for cluster cluster1:  
    Kubernetes Profile Name: profile4  
    Are you sure you want to continue? (y/n): y  
    Use 'tkgi cluster cluster1' to monitor the state of your cluster  
    

    This restarts the kube-apiserver processes to ensure that each server now encrypts using the new key.

17. Verify the stored secret is prefixed with k8s:enc:aescbc:v1:key2 which indicates the aescbc provider has encrypted the resulting data.

    For example:

    master/3f9c5ca9-a2b1-469c-9007-6fdd0844a5ec:/var/vcap/bosh_ssh/bosh_2f4aaa514474422\#  
    ETCDCTL_API=3 /var/vcap/packages/etcdctl/etcdctl --endpoints  
    https://master-0.etcd.cfcr.internal:2379 --cert  
    /var/vcap/jobs/etcd/config/etcdctl.crt --key  
    /var/vcap/jobs/etcd/config/etcdctl.key --cacert  
    /var/vcap/jobs/etcd/config/etcdctl-ca.crt get  
    /registry/secrets/default/secret1  
    
    /registry/secrets/default/secret1  
    
    k8s:enc:aescbc:v1:key2:����@�8�����A������2cM7������uL�/  
    

Decrypt Secrets in the etcd Database

1. To deactivate encryption of data at rest, place the identity provider as the first entry in the encryption provider configuration file.

   For example:

   apiVersion: apiserver.config.k8s.io/v1
   kind: EncryptionConfiguration
   resources:
     - resources:
       - secrets
       providers:
       - identity: {}
       - aescbc:
           keys:
           - name: key1
             secret: <BASE 64 ENCODED SECRET>
   
   

2. Create a new Kubernetes profile configuration file that references the modified encryption-provider-config.yml.

   For example:

   {
     "name": "profile5",
     "description": "Testing profile five",
     "customizations": [
       {
         "component": "kube-apiserver",
         "arguments": { },
         "file-arguments": {
           "encryption-provider-config": "/tmp/encryption-provider-config.yml"
         }
       }
     ]
   }
   

   For information about creating a Kubernetes profile configuration file, see Create Kubernetes Profile above.

3. Use the TKGI CLI to create a profile based on the modified Kubernetes profile configuration file.

   For example:

   $ tkgi create-k8s-profile /tmp/profile5.json  
   Kubernetes profile profile5 successfully created  
   

4. Update the cluster with the new profile.

   For example:

   $ tkgi update-cluster cluster1 --kubernetes-profile profile5  
   
   Update summary for cluster cluster1:  
   Kubernetes Profile Name: profile5  
   Are you sure you want to continue? (y/n): y  
   Use 'tkgi cluster cluster1' to monitor the state of your cluster  
   

   For information about using update-cluster, see Rotate Encryption Key for Secrets in etcd Database above.

   WARNING: Update the Kubernetes profile only on a TKGI cluster that has been upgraded to the current TKGI version. For more information, see Tasks Supported Following a TKGI Control Plane Upgrade in About Tanzu Kubernetes Grid Integrated Edition Upgrades.

5. To force all secrets to be decrypted:

   kubectl get secrets --all-namespaces -o json | kubectl replace -f -