vSphere IaaS control plane uses Transport Layer Security (TLS) encryption to secure communications among components. TKG on Supervisor includes several TLS certificates supporting this cryptographic infrastructure. Supervisor certificate rotatation is manual. TKG certificate rotation is automated, but can be done manually if necessary.

About TLS Certificates for TKG Service Clusters

vSphere IaaS control plane uses TLS certificates for securing communication among the following components:
  • vCenter Server
  • Supervisor control plane nodes
  • ESXi hosts functioning as worker nodes for vSphere Pods
  • TKG cluster nodes, both control plane and worker
vSphere IaaS control plane operates in the following trust domains to authenticate system users.
Trust Domain Description
vCenter trust domain The default signer for TLS certificates in this trust domain is the VMware Certificate Authority (VMCA) built into vCenter Server.
Kubernetes trust domain The default signer for TLS certificates in this trust domain is the Kubernetes Certificate Authority (CA)
Refer to the KB article vSphere with Tanzu Certificate Guide for additional details and complete lists of each TLS certificate used in the vSphere IaaS control plane environment.

TLS Certificate Rotation

The procedure for rotating TLS certificates differs depending on if the certificates are used for Supervisor or for TKG Service clusters.
Supervisor Cert Rotation

TLS certificates for Supervisor are derived from the VMCA certificate. Refer to the KB article vSphere with Tanzu Certificate Guide for details on Supervisor certificates.

Certificate rotation for Supervisor is automated beginning with vSphere 8 U3. Refer to the platform release notes for more information.

For releases prior to vSphere 8 U3, certificate rotation for Supervisor is manual. Refer to the KB article Replace vSphere with Tanzu Supervisor Certificates for instructions on replacing Supervisor certificates using the WCP Cert Manager tool.

TKG 2.0 Cluster Cert Rotation

Typically you do not need to manually rotate the TLS certificates for a TKG cluster because when you update a TKG cluster, the rolling update process automatically rotates the TLS certificates for you.

If the TLS certificates for a TKG cluster have not expired, and you need to manually rotate these certificates, you can do so by completing the steps in the following section.

Manually Rotate the TLS Certificates for a TKG Service Cluster

These instructions assume advanced knowledge of and experience with TKG cluster administration. In addition, these instructions assume the TLS certificates are not expired. If the certificates are expired, do not complete these steps.
  1. To run these steps, SSH to one of the Supervisor nodes. See Connecting to TKG Service Clusters as a Kubernetes Administrator and System User.
  2. Get the TKG cluster name.
    export CLUSTER_NAMESPACE="tkg-cluster-ns"

kubectl get clusters -n $CLUSTER_NAMESPACE
NAME                    PHASE         AGE   VERSION
tkg-cluster             Provisioned   43h
  3. Get the TKG cluster kubeconfig.
    export CLUSTER_NAME="tkg-cluster"

kubectl get secrets -n $CLUSTER_NAMESPACE $CLUSTER_NAME-kubeconfig -o jsonpath='{.data.value}' | base64 -d > $CLUSTER_NAME-kubeconfig
  4. Get the TKG cluster SSH key.
    kubectl get secrets -n $CLUSTER_NAMESPACE $CLUSTER_NAME-ssh -o jsonpath='{.data.ssh-privatekey}' | base64 -d > $CLUSTER_NAME-ssh-privatekey
chmod 600 $CLUSTER_NAME-ssh-privatekey
  5. Check the environment before certificate rotation.
    export KUBECONFIG=$CLUSTER_NAME-kubeconfig
    kubectl get nodes -o wide
    kubectl get nodes \
-o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \
-l node-role.kubernetes.io/master= > nodes
    for i in `cat nodes`; do
    printf "\n######\n"
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo kubeadm certs check-expiration
done;

    Example result from the previous commands:

    tkg-cluster-control-plane-k8bqh
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 04, 2023 23:00 UTC   363d                                    no
apiserver                  Oct 04, 2023 23:00 UTC   363d            ca                      no
apiserver-etcd-client      Oct 04, 2023 23:00 UTC   363d            etcd-ca                 no
apiserver-kubelet-client   Oct 04, 2023 23:00 UTC   363d            ca                      no
controller-manager.conf    Oct 04, 2023 23:00 UTC   363d                                    no
etcd-healthcheck-client    Oct 04, 2023 23:00 UTC   363d            etcd-ca                 no
etcd-peer                  Oct 04, 2023 23:00 UTC   363d            etcd-ca                 no
etcd-server                Oct 04, 2023 23:00 UTC   363d            etcd-ca                 no
front-proxy-client         Oct 04, 2023 23:00 UTC   363d            front-proxy-ca          no
scheduler.conf             Oct 04, 2023 23:00 UTC   363d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 01, 2032 22:56 UTC   9y              no
etcd-ca                 Oct 01, 2032 22:56 UTC   9y              no
front-proxy-ca          Oct 01, 2032 22:56 UTC   9y              no
  6. Rotate the TLS certificates for the TKG 2.0 cluster.
    Switch context back to the Supervisor before proceeding with the following steps. 
    unset KUBECONFIG
kubectl config current-context
kubernetes-admin@kubernetes
    kubectl get kcp  -n $CLUSTER_NAMESPACE $CLUSTER_NAME-control-plane -o jsonpath='{.apiVersion}{"\n"}'
controlplane.cluster.x-k8s.io/v1beta1
    kubectl get kcp -n $CLUSTER_NAMESPACE $CLUSTER_NAME-control-plane
NAME                        CLUSTER       INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
tkg-cluster-control-plane   tkg-cluster   true          true                   3          3       3         0             43h   v1.21.6+vmware.1

    kubectl patch kcp $CLUSTER_NAME-control-plane -n $CLUSTER_NAMESPACE --type merge -p "{\"spec\":{\"rolloutAfter\":\"`date +'%Y-%m-%dT%TZ'`\"}}"
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/tkg-cluster-control-plane patched
    Machine rollout started: 
    kubectl get machines -n $CLUSTER_NAMESPACE
NAME                                              CLUSTER       NODENAME                                          PROVIDERID                                       PHASE          AGE   VERSION
tkg-cluster-control-plane-k8bqh                   tkg-cluster   tkg-cluster-control-plane-k8bqh                   vsphere://420a2e04-cf75-9b43-f5b6-23ec4df612eb   Running        43h   v1.21.6+vmware.1
tkg-cluster-control-plane-l7hwd                   tkg-cluster   tkg-cluster-control-plane-l7hwd                   vsphere://420a57cd-a1a0-fec6-a741-19909854feb6   Running        43h   v1.21.6+vmware.1
tkg-cluster-control-plane-mm6xj                   tkg-cluster   tkg-cluster-control-plane-mm6xj                   vsphere://420a67c2-ce1c-aacc-4f4c-0564daad4efa   Running        43h   v1.21.6+vmware.1
tkg-cluster-control-plane-nqdv6                   tkg-cluster                                                                                                      Provisioning   25s   v1.21.6+vmware.1
tkg-cluster-workers-v8575-59c6645b4-wvnlz         tkg-cluster   tkg-cluster-workers-v8575-59c6645b4-wvnlz         vsphere://420aa071-9ac2-02ea-6530-eb59ceabf87b   Running        43h   v1.21.6+vmware.1
    Machine rollout completed: 
    kubectl get machines -n $CLUSTER_NAMESPACE
NAME                                              CLUSTER       NODENAME                                          PROVIDERID                                       PHASE     AGE   VERSION
tkg-cluster-control-plane-m9745                   tkg-cluster   tkg-cluster-control-plane-m9745                   vsphere://420a5758-50c4-3172-7caf-0bbacaf882d3   Running   17m   v1.21.6+vmware.1
tkg-cluster-control-plane-nqdv6                   tkg-cluster   tkg-cluster-control-plane-nqdv6                   vsphere://420ad908-00c2-4b9b-74d8-8d197442e767   Running   22m   v1.21.6+vmware.1
tkg-cluster-control-plane-wdmph                   tkg-cluster   tkg-cluster-control-plane-wdmph                   vsphere://420af38a-f9f8-cb21-e05d-c1bcb6840a93   Running   10m   v1.21.6+vmware.1
tkg-cluster-workers-v8575-59c6645b4-wvnlz         tkg-cluster   tkg-cluster-workers-v8575-59c6645b4-wvnlz         vsphere://420aa071-9ac2-02ea-6530-eb59ceabf87b   Running   43h   v1.21.6+vmware.1
  7. Verify manual certificate rotation for the TKG 2.0 cluster.
    Run the following commands to verify certificate rotation: 
    export KUBECONFIG=$CLUSTER_NAME-kubeconfig

kubectl get nodes -o wide
NAME                                        STATUS   ROLES                  AGE     VERSION            INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                 KERNEL-VERSION       CONTAINER-RUNTIME
tkg-cluster-control-plane-m9745             Ready    control-plane,master   15m     v1.21.6+vmware.1   10.244.0.55   <none>        VMware Photon OS/Linux   4.19.198-1.ph3-esx   containerd://1.4.11
tkg-cluster-control-plane-nqdv6             Ready    control-plane,master   21m     v1.21.6+vmware.1   10.244.0.54   <none>        VMware Photon OS/Linux   4.19.198-1.ph3-esx   containerd://1.4.11
tkg-cluster-control-plane-wdmph             Ready    control-plane,master   9m22s   v1.21.6+vmware.1   10.244.0.56   <none>        VMware Photon OS/Linux   4.19.198-1.ph3-esx   containerd://1.4.11
tkg-cluster-workers-v8575-59c6645b4-wvnlz   Ready    <none>                 43h     v1.21.6+vmware.1   10.244.0.51   <none>        VMware Photon OS/Linux   4.19.198-1.ph3-esx   containerd://1.4.11

kubectl get nodes \
-o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \
-l node-role.kubernetes.io/master= > nodes

for i in `cat nodes`; do
    printf "\n######\n"
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo kubeadm certs check-expiration
done;
    Example result showing the updated expiration dates. 
    ######
tkg-cluster-control-plane-m9745
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 06, 2023 18:18 UTC   364d                                    no
apiserver                  Oct 06, 2023 18:18 UTC   364d            ca                      no
apiserver-etcd-client      Oct 06, 2023 18:18 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Oct 06, 2023 18:18 UTC   364d            ca                      no
controller-manager.conf    Oct 06, 2023 18:18 UTC   364d                                    no
etcd-healthcheck-client    Oct 06, 2023 18:18 UTC   364d            etcd-ca                 no
etcd-peer                  Oct 06, 2023 18:18 UTC   364d            etcd-ca                 no
etcd-server                Oct 06, 2023 18:18 UTC   364d            etcd-ca                 no
front-proxy-client         Oct 06, 2023 18:18 UTC   364d            front-proxy-ca          no
scheduler.conf             Oct 06, 2023 18:18 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 01, 2032 22:56 UTC   9y              no
etcd-ca                 Oct 01, 2032 22:56 UTC   9y              no
front-proxy-ca          Oct 01, 2032 22:56 UTC   9y              no
  8. Verify the Kubelet certificate.

    You do not need to rotate the Kubelet certificate assuming the parameter rotateCertificates in the kubelet config is set to true, which is the default configuration.

    This configuration can be verified using the following commands: 
    kubectl get nodes \
-o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \
-l node-role.kubernetes.io/master!= > workernodes

for i in `cat workernodes`; do
    printf "\n######\n"
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname
    ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo grep rotate /var/lib/kubelet/config.yaml
done;
    Example result: 
    ######
tkg-cluster-workers-v8575-59c6645b4-wvnlz
rotateCertificates: true