Update a v1 Workload Cluster CA Certificate Manually

This document guides user to update airgap server CA certificate of v1 workload cluster manually. Creating node pools, scaling in/out control plane, and nodepool cluster operations can recover after updating the cluster with new airgap server CA certificate.

Prerequisites

Airgap server is updated with new self-signed certificate or private root CA signed certificate.

Note:

If airgap server is updated with public signed certificate, updating existing cluster or following the steps in this guide is not required.

Workload cluster's associated management cluster is updated.

Procedure

Get Resources.

Find the workload cluster id via CCLI.

# ccli list wc
+-------+------------------+------------------+------------------------------------------+----------------------------+--------------+----------+---------------------+
| INDEX | NAME             | CONTEXT          | ID                                       | VCENTER ADDRESSES          | CONNECTIVITY | SELECTED | TYPE                |
+-------+------------------+------------------+------------------------------------------+----------------------------+--------------+----------+---------------------+
| 0     | ipv4-airgap-v1wc | ipv4-airgap-v1wc | 1fabca0a-ac69-406f-950a-92e7ddf7c4ae     | vc70u3.ipv6.eng.vmware.com | true         |          | KBS                 |
| 1     | ipv4-airgap-v2wc | ipv4-airgap-v2wc | aa9fbf86d970421cea233a00b3ea62bf64354f5c | vc70u3.ipv6.eng.vmware.com | true         |          | KubeClusterOperator |
+-------+------------------+------------------+------------------------------------------+----------------------------+--------------+----------+---------------------+
# ccli go 0

In the following example, we will update cluster "ipv4-airgap-v1wc" with id "1fabca0a-ac69-406f-950a-92e7ddf7c4ae".

Locate workload cluster kubeconfig file and create alias of the cluster kubectl command.

The v1 workload cluster kubeconfig file is located at /opt/vmware/k8s-bootstrapper/<cluster id>/kubeconfig
```
# alias wk="kubectl --kubeconfig /opt/vmware/k8s-bootstrapper/1fabca0a-ac69-406f-950a-92e7ddf7c4ae/kubeconfig"
```

Locate workload cluster's management cluster kubeconfig file and create alias of the management cluster.

# ccli show spec
tkgID: 189118fc-232c-4119-9ad6-5669b0a63654
id: 1fabca0a-ac69-406f-950a-92e7ddf7c4ae
tkgMgmtClusterId: 97bad1f2-bbed-4019-b226-b211f50bb3f4
...
# alias mk="kubectl --kubeconfig /opt/vmware/k8s-bootstrapper/97bad1f2-bbed-4019-b226-b211f50bb3f4/kubeconfig"

Update tkgcontext of v1 workload cluster.

Get cluster tkgcontext id.

In the output of ccli show spec command, tkgID is the tkgcontext id.

# ccli show spec
tkgID: 189118fc-232c-4119-9ad6-5669b0a63654
id: 1fabca0a-ac69-406f-950a-92e7ddf7c4ae
tkgMgmtClusterId: 97bad1f2-bbed-4019-b226-b211f50bb3f4
tkgManagementNamespace: ipv4-airgap-v1wc
clusterName: ipv4-airgap-v1wc
clusterPassword: '***'
kubernetesVersion: v1.22.9+vmware.1
serviceCIDR: 100.64.0.0/13
clusterCIDR: 100.96.0.0/11

tkgID: 189118fc-232c-4119-9ad6-5669b0a63654 is the tkgcontext id to update.

Save, edit, and update tkgcontext.

This step is the same as Save, edit and update tkgcontext.

Update v1 workload cluster node.

This step is the same as Update management cluster nodes, but with current context to show the node IP of v1 workload cluster by ccli show status.
Update v1 workload cluster kapp controller.
1. Update kapp controller addon secret on management cluster.
  1.Save cluster kapp controller addon secret to a file.
  
  # mk get secret -n ipv4-airgap-v1wc ipv4-airgap-v1wc-kapp-controller-addon -o jsonpath={.data."values\.yaml"} | base64 -d > kapp.yaml
  
  2.Edit kapp.yaml with new CA certificate and save it with ":wq".
```
# vi kapp.yaml
#@data/values
#@overlay/match-child-defaults missing_ok=True
---
kappController:
  ...
  config:
    caCerts: |-
      -----BEGIN CERTIFICATE-----
      <update this content with new CA certificate content>
      -----END CERTIFICATE-----
```
  3.Patch secret.
  
  # mk patch secret -n ipv4-airgap-v1wc ipv4-airgap-v1wc-kapp-controller-addon -p "{\"data\":{\"values.yaml\": \"`base64 kapp.yaml -w 0`\"}}"
2. Update kapp controller configmap.
  
  This step is the same as Update kapp-controller-config with new CA certificate content. The only different is replacing "mk" with "wk" so that the change will be applied to workload cluster.

Update v1 workload cluster kubecontrolplane and kubeadminconfigtemplate CRs on the management cluster.

Update v1 workload cluster kubecontrolplane CR.

# mk get kcp -n ipv4-airgap-v1wc
NAME                                    CLUSTER            INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
ipv4-airgap-v1wc-master-control-plane   ipv4-airgap-v1wc   true          true                   1          1       1         0             11d   v1.22.9+vmware.1
# mk edit kcp -n ipv4-airgap-v1wc ipv4-airgap-v1wc-master-control-plane
...
    - content: <cert-base64-content>
      encoding: base64
      path: /etc/containerd/airgap-repo-server.example.crt
      permissions: "0444"
...

Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

Save it with ":wq".

Note:

This operation will result in control plane nodes redeployment.

Update v1 workload cluster kubeadmconfigtemplate CRs.

# mk get kubeadmconfigtemplate -n ipv4-airgap-v1wc
NAME                   AGE
ipv4-airgap-v1wc-np1   11d
# mk edit kubeadmconfigtemplate -n ipv4-airgap-v1wc ipv4-airgap-v1wc-np1
...
spec:
  template:
    spec:
      files:
      - content: <cert-base64-content>
        encoding: base64
        path: /etc/containerd/airgap-repo-server.example.com.crt
        permissions: "0444"
...

Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

Save it with ":wq".

Remember:

Note that in this example there is only one node pool in the cluster. For multiple node pools, kubeadmconfigtemplate of every node pool must be updated one by one.