vSphere IaaS control plane uses Transport Layer Security (TLS) encryption to secure communications among components. TKG on Supervisor includes several TLS certificates supporting this cryptographic infrastructure. Supervisor certificate rotatation is manual. TKG certificate rotation is automated, but can be done manually if necessary.
About TLS Certificates for TKG Service Clusters
vSphere IaaS control plane uses TLS certificates for securing communication among the following components:
Refer to the KB article
vSphere with Tanzu Certificate Guide for additional details and complete lists of each TLS certificate used in the
vSphere IaaS control plane environment.
- vCenter Server
- Supervisor control plane nodes
- ESXi hosts functioning as worker nodes for vSphere Pods
- TKG cluster nodes, both control plane and worker
| Trust Domain | Description |
|---|---|
| vCenter trust domain | The default signer for TLS certificates in this trust domain is the VMware Certificate Authority (VMCA) built into vCenter Server. |
| Kubernetes trust domain | The default signer for TLS certificates in this trust domain is the Kubernetes Certificate Authority (CA) |
TLS Certificate Rotation
The procedure for rotating TLS certificates differs depending on if the certificates are used for
Supervisor or for TKG Service clusters.
- Supervisor Cert Rotation
-
TLS certificates for Supervisor are derived from the VMCA certificate. Refer to the KB article vSphere with Tanzu Certificate Guide for details on Supervisor certificates.
Certificate rotation for Supervisor is manual. Refer to the KB article Replace vSphere with Tanzu Supervisor Certificates for instructions on replacing Supervisor certificates using the WCP Cert Manager tool.
- TKG 2.0 Cluster Cert Rotation
-
Typically you do not need to manually rotate the TLS certificates for a TKG cluster because when you update a TKG cluster, the rolling update process automatically rotates the TLS certificates for you.
If the TLS certificates for a TKG cluster have not expired, and you need to manually rotate these certificates, you can do so by completing the steps in the following section.
Manually Rotate the TLS Certificates for a TKG Service Cluster
These instructions assume advanced knowledge of and experience with TKG cluster administration. In addition, these instructions assume the TLS certificates are not expired. If the certificates are expired, do not complete these steps.
- To run these steps, SSH to one of the Supervisor nodes. See Connecting to TKG Service Clusters as a Kubernetes Administrator and System User.
- Get the TKG cluster name.
export CLUSTER_NAMESPACE="tkg-cluster-ns" kubectl get clusters -n $CLUSTER_NAMESPACE NAME PHASE AGE VERSION tkg-cluster Provisioned 43h
- Get the TKG cluster kubeconfig.
export CLUSTER_NAME="tkg-cluster" kubectl get secrets -n $CLUSTER_NAMESPACE $CLUSTER_NAME-kubeconfig -o jsonpath='{.data.value}' | base64 -d > $CLUSTER_NAME-kubeconfig - Get the TKG cluster SSH key.
kubectl get secrets -n $CLUSTER_NAMESPACE $CLUSTER_NAME-ssh -o jsonpath='{.data.ssh-privatekey}' | base64 -d > $CLUSTER_NAME-ssh-privatekey chmod 600 $CLUSTER_NAME-ssh-privatekey - Check the environment before certificate rotation.
export KUBECONFIG=$CLUSTER_NAME-kubeconfig
kubectl get nodes -o wide
kubectl get nodes \ -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \ -l node-role.kubernetes.io/master= > nodesfor i in `cat nodes`; do printf "\n######\n" ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo kubeadm certs check-expiration done;Example result from the previous commands:
tkg-cluster-control-plane-k8bqh [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Oct 04, 2023 23:00 UTC 363d no apiserver Oct 04, 2023 23:00 UTC 363d ca no apiserver-etcd-client Oct 04, 2023 23:00 UTC 363d etcd-ca no apiserver-kubelet-client Oct 04, 2023 23:00 UTC 363d ca no controller-manager.conf Oct 04, 2023 23:00 UTC 363d no etcd-healthcheck-client Oct 04, 2023 23:00 UTC 363d etcd-ca no etcd-peer Oct 04, 2023 23:00 UTC 363d etcd-ca no etcd-server Oct 04, 2023 23:00 UTC 363d etcd-ca no front-proxy-client Oct 04, 2023 23:00 UTC 363d front-proxy-ca no scheduler.conf Oct 04, 2023 23:00 UTC 363d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Oct 01, 2032 22:56 UTC 9y no etcd-ca Oct 01, 2032 22:56 UTC 9y no front-proxy-ca Oct 01, 2032 22:56 UTC 9y no
- Rotate the TLS certificates for the TKG 2.0 cluster.
Switch context back to the Supervisor before proceeding with the following steps.
unset KUBECONFIG kubectl config current-context kubernetes-admin@kubernetes
kubectl get kcp -n $CLUSTER_NAMESPACE $CLUSTER_NAME-control-plane -o jsonpath='{.apiVersion}{"\n"}' controlplane.cluster.x-k8s.io/v1beta1kubectl get kcp -n $CLUSTER_NAMESPACE $CLUSTER_NAME-control-plane NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION tkg-cluster-control-plane tkg-cluster true true 3 3 3 0 43h v1.21.6+vmware.1
kubectl patch kcp $CLUSTER_NAME-control-plane -n $CLUSTER_NAMESPACE --type merge -p "{\"spec\":{\"rolloutAfter\":\"`date +'%Y-%m-%dT%TZ'`\"}}" kubeadmcontrolplane.controlplane.cluster.x-k8s.io/tkg-cluster-control-plane patchedMachine rollout started:kubectl get machines -n $CLUSTER_NAMESPACE NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION tkg-cluster-control-plane-k8bqh tkg-cluster tkg-cluster-control-plane-k8bqh vsphere://420a2e04-cf75-9b43-f5b6-23ec4df612eb Running 43h v1.21.6+vmware.1 tkg-cluster-control-plane-l7hwd tkg-cluster tkg-cluster-control-plane-l7hwd vsphere://420a57cd-a1a0-fec6-a741-19909854feb6 Running 43h v1.21.6+vmware.1 tkg-cluster-control-plane-mm6xj tkg-cluster tkg-cluster-control-plane-mm6xj vsphere://420a67c2-ce1c-aacc-4f4c-0564daad4efa Running 43h v1.21.6+vmware.1 tkg-cluster-control-plane-nqdv6 tkg-cluster Provisioning 25s v1.21.6+vmware.1 tkg-cluster-workers-v8575-59c6645b4-wvnlz tkg-cluster tkg-cluster-workers-v8575-59c6645b4-wvnlz vsphere://420aa071-9ac2-02ea-6530-eb59ceabf87b Running 43h v1.21.6+vmware.1
Machine rollout completed:kubectl get machines -n $CLUSTER_NAMESPACE NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION tkg-cluster-control-plane-m9745 tkg-cluster tkg-cluster-control-plane-m9745 vsphere://420a5758-50c4-3172-7caf-0bbacaf882d3 Running 17m v1.21.6+vmware.1 tkg-cluster-control-plane-nqdv6 tkg-cluster tkg-cluster-control-plane-nqdv6 vsphere://420ad908-00c2-4b9b-74d8-8d197442e767 Running 22m v1.21.6+vmware.1 tkg-cluster-control-plane-wdmph tkg-cluster tkg-cluster-control-plane-wdmph vsphere://420af38a-f9f8-cb21-e05d-c1bcb6840a93 Running 10m v1.21.6+vmware.1 tkg-cluster-workers-v8575-59c6645b4-wvnlz tkg-cluster tkg-cluster-workers-v8575-59c6645b4-wvnlz vsphere://420aa071-9ac2-02ea-6530-eb59ceabf87b Running 43h v1.21.6+vmware.1
- Verify manual certificate rotation for the TKG 2.0 cluster.
Run the following commands to verify certificate rotation:
export KUBECONFIG=$CLUSTER_NAME-kubeconfig kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME tkg-cluster-control-plane-m9745 Ready control-plane,master 15m v1.21.6+vmware.1 10.244.0.55 <none> VMware Photon OS/Linux 4.19.198-1.ph3-esx containerd://1.4.11 tkg-cluster-control-plane-nqdv6 Ready control-plane,master 21m v1.21.6+vmware.1 10.244.0.54 <none> VMware Photon OS/Linux 4.19.198-1.ph3-esx containerd://1.4.11 tkg-cluster-control-plane-wdmph Ready control-plane,master 9m22s v1.21.6+vmware.1 10.244.0.56 <none> VMware Photon OS/Linux 4.19.198-1.ph3-esx containerd://1.4.11 tkg-cluster-workers-v8575-59c6645b4-wvnlz Ready <none> 43h v1.21.6+vmware.1 10.244.0.51 <none> VMware Photon OS/Linux 4.19.198-1.ph3-esx containerd://1.4.11 kubectl get nodes \ -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \ -l node-role.kubernetes.io/master= > nodes for i in `cat nodes`; do printf "\n######\n" ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo kubeadm certs check-expiration done;Example result showing the updated expiration dates.###### tkg-cluster-control-plane-m9745 [check-expiration] Reading configuration from the cluster... [check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Oct 06, 2023 18:18 UTC 364d no apiserver Oct 06, 2023 18:18 UTC 364d ca no apiserver-etcd-client Oct 06, 2023 18:18 UTC 364d etcd-ca no apiserver-kubelet-client Oct 06, 2023 18:18 UTC 364d ca no controller-manager.conf Oct 06, 2023 18:18 UTC 364d no etcd-healthcheck-client Oct 06, 2023 18:18 UTC 364d etcd-ca no etcd-peer Oct 06, 2023 18:18 UTC 364d etcd-ca no etcd-server Oct 06, 2023 18:18 UTC 364d etcd-ca no front-proxy-client Oct 06, 2023 18:18 UTC 364d front-proxy-ca no scheduler.conf Oct 06, 2023 18:18 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Oct 01, 2032 22:56 UTC 9y no etcd-ca Oct 01, 2032 22:56 UTC 9y no front-proxy-ca Oct 01, 2032 22:56 UTC 9y no
- Verify the Kubelet certificate.
You do not need to rotate the Kubelet certificate assuming the parameter
rotateCertificatesin the kubelet config is set totrue, which is the default configuration.This configuration can be verified using the following commands:kubectl get nodes \ -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' \ -l node-role.kubernetes.io/master!= > workernodes for i in `cat workernodes`; do printf "\n######\n" ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i hostname ssh -o "StrictHostKeyChecking=no" -i $CLUSTER_NAME-ssh-privatekey -q vmware-system-user@$i sudo grep rotate /var/lib/kubelet/config.yaml done;Example result:###### tkg-cluster-workers-v8575-59c6645b4-wvnlz rotateCertificates: true
