Manage Cluster Secrets

This topic explains how to configure and manage secrets used by management and workload clusters in Tanzu Kubernetes Grid, including:

  • Credentials that clusters use to access cloud infrastructure APIs and resources.
  • Certificate Authority (CA) certificates that clusters use to access private container registries.
  • Credentials that clusters use to access authenticated private container registries.

Update Management and Workload Cluster Credentials (vSphere)

To update the vSphere credentials used by the current management cluster and by all of its workload clusters, use the tanzu mc credentials update --cascading command:

  1. Run tanzu login to log in to the management cluster that you are updating.

  2. Run tanzu mc credentials update

    You can pass values to the following command options, or let the CLI prompt you for them:

    • --vsphere-user: Name for the vSphere account.
    • --vsphere-password: Password the vSphere account.

Update Management Cluster Credentials Only

To update a management cluster’s vSphere credentials without also updating them for its workload clusters, use the tanzu mc credentials update command as above, but without the --cascading option.

Update Workload Cluster Credentials (vSphere)

To update the credentials that a single workload cluster uses to access vSphere, use the tanzu cluster credentials update command:

  1. Run tanzu login to log in to the management cluster that created the workload cluster that you are updating.

  2. Run tanzu cluster credentials update CLUSTER_NAME

    You can pass values to the following command options, or let the CLI prompt you for them:

    • --namespace: The namespace of the cluster you are updating credentials for, such as default.
    • --vsphere-user: Name for the vSphere account.
    • --vsphere-password: Password the vSphere account.

You can also use tanzu mc credentials update --cascading to update vSphere credentials for a management cluster and all of the workload clusters it manages.

Update Management Cluster Credentials (Microsoft Azure)

The tanzu management-cluster credentials update command, which updates the management cluster credentials, can only be used on a cluster that is deployed on vSphere. Use this procedure to update the credentials (client secret) of a cluster that is deployed on Microsoft Azure.

Prerequisites

Before you begin, obtain the new client secret from the Azure portal, or from the Azure administrator. For more information on how to obtain the client secret from the Azure portal, see Azure documentation.

Procedure

  1. Obtain the name of the pod that hosts the Cluster API provider for Azure (CAPZ) controller:

    kubectl get pods -n capz-system
    

    In the following example, the pod name is capz-controller-manager-7f94686c74-mkj9n.

    capi@bootstrap-vm:~$ kubectl config use-context mgmt-admin@mgmt
    Switched to context "mgmt-admin@mgmt".
    capi@bootstrap-vm:~$ kubectl get pods -n capz-system
    NAME                                       READ
    capz-controller-manager-7f94686c74-mkj9n   2/2     Running   0          28h
    capz-nmi-v4gl9                             1/1     Running   0          5d23h
    
  2. Base64 encode the Azure client secret:

    echo 'CLIENT-SECRET' | base64 -w 0
    

    Where CLIENT-SECRET is the new client secret that you obtained from Azure. Ensure that there are no additional line breaks when you run this command.

  3. In the Tanzu CLI, set the context of kubectl to your management cluster:

    kubectl config use-context MY-MGMT-CLUSTER-admin@MY-MGMT-CLUSTER
    

    Where MY-MGMT-CLUSTER is the name of your management cluster.

  4. Update the capz-manager-bootstrap-credentials secret:

    kubectl patch secret capz-manager-bootstrap-credentials -n capz-system --type merge -p '{"data":{"client-secret": ENCODED-CREDENTIAL}}'
    

    Where ENCODED-CREDENTIAL is the encoded client secret that you created in Step 1.

  5. Restart the capz-controller-manager pod:

    kubectl delete pod CONTROLLER-MANAGER-POD -n capz-system
    
    kubectl get pods -n capz-system
    

    Where CONTROLLER-MANAGER-POD is the name of the capz-controller-manager pod.

  6. Verify that the Azure json secrets have changed:

    a. Obtain the list of Azure json secrets from the cluster:

    kubectl get secrets -A | grep azure-json
    

    b. Verify that the Azure secret is updated in the aadClientSecret field in each secret:

    kubectl get secret AZURE-JSON-SECRET -o jsonpath='{.data.azure\.json}' | base64 -d
    

    Where AZURE-JSON-SECRET is the name of the Azure json secret, for example, dl-md-0-azure-json.

  7. Recycle all the worker nodes in the cluster:

    a. Go to the kubectl context of the management cluster:

    kubectl config use-context MC-CONTEXT
    

    b. Run the following command on the management cluster and each workload cluster:

    kubectl patch kcp CLUSTER-NAME -n tkg-system --type merge -p "{\"spec\":{\"rolloutAfter\":\"`date +'%Y-%m-%dT%TZ'`\"}}"
    

    Where CLUSTER-NAME is the name of the Azure cluster.

    Note: Ensure that your bootstrap machine and the cluster are using the same time zone.

Trust Custom CA Certificates in New Clusters

You can add custom CA certificates in workload cluster nodes by using a ytt overlay file to enable the cluster nodes to pull images from a container registry that uses self signed certificates.

The overlay code below adds custom CA certificates to all nodes in a new cluster, so that containerd and other tools trust the certificate. The code works on all cloud infrastructure providers, for clusters based on Photon or Ubuntu VM image templates.

For overlays that customize clusters and create a new cluster plan, see ytt Overlays in the Configure Tanzu Kubernetes Plans and Clusters topic.

  1. Choose whether to apply the custom CA to all new clusters, just the ones created on one cloud infrastructure, or ones created with a specific Cluster API provider version, such as Cluster API Provider vSphere v0.7.4.

  2. In your local ~/.config/tanzu/tkg/providers/ directory, find the ytt directory that covers your chosen scope. For example, /ytt/03_customizations/ applies to all clusters, and /infrastructure-vsphere/ytt/ applies to all vSphere clusters.

  3. In your chosen ytt directory, create a new .yaml file or augment an existing overlay file with the following code:

    #@ load("@ytt:overlay", "overlay")
    #@ load("@ytt:data", "data")
    
    #! This ytt overlay adds additional custom CA certificates on TKG cluster nodes, so containerd and other tools trust these CA certificates.
    #! It works when using Photon or Ubuntu as the TKG node template on all TKG infrastructure providers.
    
    #! Trust your custom CA certificates on all Control Plane nodes.
    #@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
    ---
    spec:
      kubeadmConfigSpec:
        #@overlay/match missing_ok=True
        files:
          #@overlay/append
          - content: #@ data.read("tkg-custom-ca.pem")
            owner: root:root
            permissions: "0644"
            path: /etc/ssl/certs/tkg-custom-ca.pem
        #@overlay/match missing_ok=True
        preKubeadmCommands:
          #! For Photon OS
          #@overlay/append
          - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
          #! For Ubuntu
          #@overlay/append
          - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
    
    #! Trust your custom CA certificates on all worker nodes.
    #@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"}), expects="1+"
    ---
    spec:
      template:
        spec:
          #@overlay/match missing_ok=True
          files:
            #@overlay/append
            - content: #@ data.read("tkg-custom-ca.pem")
              owner: root:root
              permissions: "0644"
              path: /etc/ssl/certs/tkg-custom-ca.pem
          #@overlay/match missing_ok=True
          preKubeadmCommands:
            #! For Photon OS
            #@overlay/append
            - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
            #! For Ubuntu
            #@overlay/append
            - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
    
  4. In the same ytt directory, add the Certificate Authority to a new or existing tkg-custom-ca.pem file.

Add Custom CA Certificate Trust to Existing Clusters

To enable trusted communication between existing clusters and a Harbor registry with a custom, self-signed CA:

  1. Retrieve the Harbor CA certificate:

    1. Switch context to your shared services cluster:

      kubectl config use-context SERVICES-CLUSTER-CONTEXT
      

      Where SERVICES-CLUSTER-CONTEXT is the context of your shared services cluster. For example, tkg-wld-admin@tkg-wld.

    2. Retrieve the certificate:

      kubectl -n tanzu-system-registry get secret harbor-tls -o=jsonpath="{.data.ca\.crt}" | base64 -d
      -----BEGIN CERTIFICATE-----
      MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
      [...]
      yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
      -----END CERTIFICATE-----
      
  2. Add the custom CA to the management cluster’s kubeadmconfigtemplate:

    1. Switch context to management cluster.

      kubectl config use-context MANAGEMENT-CLUSTER-CONTEXT
      

      Where MANAGEMENT-CLUSTER-CONTEXT is the context of your management cluster. For example, tkg-mgmt-admin@tkg-mgmt.

    2. In an editor, open the cluster’s kubeadmconfigtemplate template file:

      kubectl edit kubeadmconfigtemplate CLUSTER-NAME-md-0
      

      Where CLUSTER-NAME is the name of the cluster that has to be modified.

    3. Change the spec.template.spec.files section of the file to include the certificate, as shown here:

      spec:
      template:
        spec:
          files:
          - content: |
                -----BEGIN CERTIFICATE-----
               MIIFazCCA1OgAwIBAgIQMfZy08muvIVKdZVDz7/rYzANBgkqhkiG9w0BAQsFADBI
               [...]
               yiDghW7antzYL9S1CC8sVgVOwFJwfFXpdiir35mQlySG301V4FsRV+Z0cFp4Ni0=
               -----END CERTIFICATE-----
             owner: root:root
             path: /etc/ssl/certs/tkg-custom-ca.pem
             permissions: "0644"
      
    4. At the bottom of the file, add a preKubeadmCommands block as shown here:

      preKubeadmCommands:
      - '! which rehash_ca_certificates.sh 2>/dev/null || rehash_ca_certificates.sh'
      - '! which update-ca-certificates 2>/dev/null || (mv /etc/ssl/certs/tkg-custom-ca.pem
         /usr/local/share/ca-certificates/tkg-custom-ca.crt && update-ca-certificates)'
      
  3. Save the kubeadmconfigtemplate template file with your changes.

  4. Patch the management cluster with the changes:

    kubectl patch machinedeployments.cluster.x-k8s.io tkg-test-md-0 --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"
    

    Running this command triggers a rolling update of the cluster nodes and updates their timestamp.

Configure Authentication to a Private Container Registry

You can add authentication secrets to enable a cluster to access a private container registry, which requires user authentication to pull images. You can also view, update, or delete the authentication secrets that you have configured for the private registries that are accessed by a cluster.

Add an Authentication Secret for a Private Container Registry

Using the Tanzu CLI, you can add authentication secrets to access a private container registry from a cluster. After the registry secret is added to the namespaces in your cluster, you can pull all the package repositories, packages, and the container images that are hosted in the private registry. Subsequently, you can add the package repository and package resources to your cluster.

Before performing this procedure, obtain the username and the password for the private container registry.

To add an authentication secret to a private registry, run the following command:

tanzu secret registry add SECRET-NAME -n NAMESPACE --server REGISTRY-URL --username USERNAME --password PASSWORD

Where:

  • SECRET-NAME is the name of the registry authentication secret that you want to add.
  • NAMESPACE is the Tanzu Kubernetes Grid namespace to which the registry belongs.
  • USERNAME is the username to access the registry. Wrap the username in single quotes if it contains special characters.
  • PASSWORD the password to access the registry. Wrap the password in single quotes if it contains special characters. You can also specify the password in the following formats:

    • Replace the --password PASSWORD string in the command with --password-env-var ENV-VAR to specify password through the environment variable, which you have configured already. The format of the command is as follows:

      tanzu secret registry add SECRET-NAME -n NAMESPACE --server REGISTRY-URL --username USERNAME --password-env-var ENV-VAR

    • Replace the --password PASSWORD string in the command with --password-stdin string to specify password through standard input, and enter the password when prompted. The format of the command is as follows:

      tanzu secret registry add SECRET-NAME -n NAMESPACE --server REGISTRY-URL --username USERNAME --password-stdin

    • Replace the --password PASSWORD string in the command with --password-file PASSWORD-FILE string to specify password through a password file. The format of the command is as follows:

      tanzu secret registry add SECRET-NAME -n NAMESPACE --server REGISTRY-URL --username USERNAME --password-file PASSWORD-FILE

Optionally, to make the registry secret available across all namespaces in a cluster, use the --export-to-all-namespaces option as shown in the following format:

tanzu secret registry add SECRET-NAME -n NAMESPACE --server REGISTRY-URL --username USERNAME --password PASSWORD --export-to-all-namespaces

The following is a sample output of this command:

tanzu secret registry add tanzu-net -n test-ns --server registry.pivotal.io --username test-user --password-file pass-file --export-to-all-namespaces
Warning: By choosing --export-to-all-namespaces, given secret contents will be available to ALL users in ALL namespaces. Please ensure that included registry credentials allow only read-only access to the registry with minimal necessary scope.
/ Adding registry secret 'test-secret'...
  Added registry secret 'test-secret' into namespace 'test-ns'

View the Registry Authentication Secrets in a Cluster

You can view the registry authentication secrets in the default namespace, or all the namespaces in a cluster. You can view the secrets in the form of a table or a JSON or a YAML file.

To view the registry authentication secrets in a specific namespace in a cluster, run the following:

tanzu secret registry list -n NAMESPACE

Where NAMESPACE is the Tanzu Kubernetes Grid namespace to which the registry belongs.

The following is an example of this command:

tanzu secret registry list -n test-ns
/ Retrieving registry secrets...
  NAME         REGISTRY                 EXPORTED           AGE
  pkg-dev-reg  registry.pivotal.io      to all namespaces  15d

To view the list of registry authentication secrets in all the namespaces in a cluster, run the following:

tanzu secret registry list -A

The following is an example of this command:

tanzu secret registry list -A
\ Retrieving registry secrets...
 NAME                          REGISTRY             EXPORTED           AGE  NAMESPACE
 pkg-dev-reg                   registry.pivotal.io  to all namespaces  15d  test-ns
 tanzu-standard-fetch-0        registry.pivotal.io  not exported       15d  tanzu-package-repo-global
 private-repo-fetch-0          registry.pivotal.io  not exported       15d  test-ns
 antrea-fetch-0                registry.pivotal.io  not exported       15d  tkg-system
 metrics-server-fetch-0        registry.pivotal.io  not exported       15d  tkg-system
 tanzu-addons-manager-fetch-0  registry.pivotal.io  not exported       15d  tkg-system
 tanzu-core-fetch-0            registry.pivotal.io  not exported       15d  tkg-system

To view the list of registry authentication secrets in the JSON file format, run the following command:

tanzu secret registry list -n kapp-controller-packaging-global -o json

The following is an example of this command:

tanzu secret registry list -n kapp-controller-packaging-global -o json
[
 {
   "age": "15d",
   "exported": "to all namespaces",
   "name": "pkg-dev-reg",
   "registry": "us-east4-docker.pkg.dev"
 }
]

To view the list of registry authentication secrets in the YAML file format, run the following:

tanzu secret registry list -n kapp-controller-packaging-global -o yaml

The following is an example of this command:

 tanzu secret registry list -n kapp-controller-packaging-global -o yaml
 - age: 15d
   exported: to all namespaces
   name: pkg-dev-reg
   registry: us-east4-docker.pkg.dev

To view the list of registry authentication secrets in a table format, run the following:

tanzu secret registry list -n kapp-controller-packaging-global -o table

The following is an example of this command:

tanzu secret registry list -n kapp-controller-packaging-global -o table
/ Retrieving registry secrets...
  NAME         REGISTRY                 EXPORTED           AGE
  pkg-dev-reg  us-east4-docker.pkg.dev  to all namespaces  15d

Update a Registry Authentication Secret

You can update the credentials in a secret, make a secret available across all the namespaces, or make it available in only one namespace in the cluster.

To update the secret in the namespace where it was created, run the following command:

tanzu secret registry update SECRET-NAME --username USERNAME -n NAMESPACE --password PASSWORD

Where:

  • SECRET-NAME is the name of the registry secret that you want to update.
  • NAMESPACE is the Tanzu Kubernetes Grid namespace where you are updating the registry authentication secret.
  • USERNAME is the new username to access the registry (if you want to update the username).
  • PASSWORD is the new password for the registry (if you want to update the password).

Note: You can update either the username or the password, or both of them.

The following is an example of this command:

tanzu secret registry update test-secret --username test-user -n test-ns --password-env-var PASSENV
\ Updating registry secret 'test-secret'...
 Updated registry secret 'test-secret' in namespace 'test-ns'  

To update the registry authentication secret and make it available in other namespaces in the cluster, run the following command:

tanzu secret registry update SECRET-NAME --username USERNAME -n NAMESPACE --password PASSWORD --export-to-all-namespaces=true

Where:

  • SECRET-NAME is the name of the registry secret that you want to update.
  • NAMESPACE is the Tanzu Kubernetes Grid namespace where you are updating the registry authentication secret.
  • USERNAME is the username to access the registry. Enter a new username if you want to update the username.
  • PASSWORD is the password for the registry. Enter a new password if you want to update the password.

The following is an example of this command:

tanzu secret registry update test-secret--username test-user -n test-ns --password-env-var PASSENV --export-to-all-namespaces=true
Warning: By specifying --export-to-all-namespaces as true, given secret contents will be available to ALL users in ALL namespaces. Please ensure that included registry credentials allow only read-only access to the registry with minimal necessary scope.
Are you sure you want to proceed? [y/N]: y

\ Updating registry secret 'test-secret'...
  Updated registry secret 'test-secret' in namespace 'test-ns'
  Exported registry secret 'test-secret' to all namespaces

To make a registry authentication secret unavailable in other namespaces in the cluster, run the following command:

tanzu secret registry update SECRET-NAME --username USERNAME -n NAMESPACE --password PASSWORD --export-to-all-namespaces=false

Where:

  • SECRET-NAME is the name of the registry secret that you want to update.
  • NAMESPACE is the Tanzu Kubernetes Grid namespace where you are updating the registry authentication secret.
  • USERNAME is the username to access the registry.
  • PASSWORD is password for the registry.

The following is an example of this command:

tanzu secret registry update test-secret --username test-user -n test-ns --password-env-var PASSENV --export-to-all-namespaces=false
Warning: By specifying --export-to-all-namespaces as false, the secret contents will get unexported from ALL namespaces in which it was previously available to.
Are you sure you want to proceed? [y/N]: y

\ Updating registry secret 'test-secret'...
  Updated registry secret 'test-secret' in namespace 'test-ns'
  Unexported registry secret 'test-secret' from all namespaces

Delete a Registry Authentication Secret

Using the Tanzu CLI, you can delete a registry authentication secret in a cluster. To delete a registry authentication secret in a specific namespace, run the following command:

tanzu secret registry delete SECRET-NAME -n NAMESPACE

Where:

  • SECRET-NAME is the name of the registry secret that you want to delete.
  • (Optional) NAMESPACE is the Tanzu Kubernetes Grid namespace from where you want to delete the registry authentication secret. If you do not specify a namespace, the authentication secret is deleted from the default namespace. If the secret had been exported to other namespaces in the cluster, it is also deleted.

The following is an example of this command:

tanzu secret registry delete test-secret -n test-ns
Deleting registry secret 'test-secret' from namespace 'test-ns'. Are you sure? [y/N]: y
\ Deleting registry secret 'test-secret'...
 Deleted registry secret 'test-secret' from namespace 'test-ns'
check-circle-line exclamation-circle-line close-line
Scroll to top icon