This document guides user to update airgap server CA certificate of a management cluster manually. Cluster operations, such as creating workload clusters, scaling in/out control plane or node pools, can recover after updating the cluster with new airgap server CA certificate.

Prerequisites

Airgap server is updated with new self-signed certificate or private root CA signed certificate.

Note:

If airgap server is updated with public signed certificate, updating existing cluster or following the steps in this guide is not required.

Procedure

  1. Login to TCA CP appliance via SSH with admin account and enter root mode.
  2. Get Resources.
    1. Find the management cluster id via CCLI. 
      # ccli list mc
+-------+------------------+------------------+--------------------------------------+--------------+----------+
| INDEX | NAME             | CONTEXT          | ID                                   | CONNECTIVITY | SELECTED |
+-------+------------------+------------------+--------------------------------------+--------------+----------+
| 0     | ipv4-airgap-mgmt | ipv4-airgap-mgmt | 97bad1f2-bbed-4019-b226-b211f50bb3f4 | true         |          |
| 1     | ipv4-mgmt        | ipv4-mgmt        | b8e55039-63f8-447c-93c3-b86bf26c8412 | true         |          |
+-------+------------------+------------------+--------------------------------------+--------------+----------+
# ccli go 0

      In the following examples, "ipv4-airgap-mgmt" cluster is used, with id "97bad1f2-bbed-4019-b226-b211f50bb3f4".

    2. Locate management cluster kubeconfig file and create alias of management cluster kubectl command.

      The management cluster kubeconfig file is located at /opt/vmware/k8s-bootstrapper/<cluster id>/kubeconfig.

      # alias mk="kubectl --kubeconfig /opt/vmware/k8s-bootstrapper/97bad1f2-bbed-4019-b226-b211f50bb3f4/kubeconfig"
  3. Update tkgcontext of management cluster.
    1. Get cluster tkgcontext id via CCLI.

      In the output of ccli show spec command, tkgID is the tkgcontext id.

      # ccli show spec
tkgID: 38561a18-3ea9-402f-9568-dca4889a6b11
id: 97bad1f2-bbed-4019-b226-b211f50bb3f4
clusterName: ipv4-airgap-mgmt
kubernetesVersion: v1.22.9+vmware.1
serviceCIDR: 100.64.0.0/13
clusterCIDR: 100.96.0.0/11
....
    2. Save, edit and update tkgcontext.

      1. Get tkgcontext json configuration file via curl command and save it to a file.

      2. Modify the file and update its airgap/caCert field with ca file base64 encoding string. If no caCert field, create it in json format. Add vsphere/password field with vc password as value.

      3. Modify tkgcontext via curl command and saved json file.

      4. Delete the json file for security consideration.

      # curl -X GET http://localhost:8888/api/v1/tkgcontext/38561a18-3ea9-402f-9568-dca4889a6b11 | python -m json.tool > tkgcontext.json
# vi tkgcontext.json # update vc password and caCert
{
    "id": "38561a18-3ea9-402f-9568-dca4889a6b11",
    "ipFamilies": "IPv4",
    "infrastructure": {
        "type": "vsphere",
        "version": "7.0.3"
    },
    "vsphere": {
        "password": "<my vc password>"
        ...
    },
    "airgap": {
        "fqdn": "airgap-repo-serve.example.com",
        "caCert": "<update it with new ca.crt base64 string"
        }
    }
}
# curl -X PUT http://localhost:8888/api/v1/tkgcontext/38561a18-3ea9-402f-9568-dca4889a6b11 -d "`cat tkgcontext.json`"
  4. Update management cluster nodes.

    This step will update all the management cluster nodes. User needs to login to each node with capv account for updating. The user needs to get cluster node IP addresses by ccli show status command. For each output IP address, user need to follow the sub-steps.

    # ccli show status
status: Running
phase: PostConfig
nodes:
- ip: 172.16.69.112
  vmName: ipv4-airgap-mgmt-master-control-plane-qc6nx
  ...
- ip: 172.16.69.213
  vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-79xvg
  ...
- ip: 172.16.70.143
  vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-svmz9
  ...
    1. Login to node from TCA CP appliance via ssh capv@<node ip> and enter root mode. 
      # ssh [email protected]
capv@ipv4-airgap-mgmt-master-control-plane-qc6nx [ ~ ]$ sudo su
    2. Copy the CA file with .pem suffix to /etc/ssl/certs folder and run rehash_ca_certificate.sh. Test the connectivity with curl command. 
      root [ /home/capv ]# cp ca.crt /etc/ssl/certs/airgap-repo-server.example.com.pem # end with .pem is a must!
root [ /home/capv ]# rehash_ca_certificates.sh
root [ /home/capv ]# curl https://airgap-repo-server.example.com -v --head # test it works
    3. Copy CA file to/etc/containerd/<airgap-server-fqdn>.crt and restart containerd with systemctl restart containerd, then verify it can pull image from airgap server with command crictl pull <airgap-server-fqdn>/registry/tkr-compatibility:v1. 
      root [ /home/capv ]# cp ca.crt /etc/containerd/airgap-repo-server.example.com.crt
root [ /home/capv ]# systemctl restart containerd
root [ /home/capv ]# systemctl status containerd
root [ /home/capv ]# crictl pull airgap-repo-server.example.com/registry/tkr-compatibility:v1 # test pulling from airgap repo
  5. Update management cluster configmaps.
    1. Update tkr-controller-config with new CA certificate content. 
      # mk edit cm tkr-controller-config -n tkr-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  caCerts: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  imageRepository: airgap-repo-server.example.com/registry
...

      Copy the new CA certificate content and overwrite the existing caCerts field.

      Save it with ":wq".

      Then restart tkr-controller-manager deployment.

      # mk rollout restart deployment -n tkr-system tkr-controller-manager

    2. Update kapp-controller-config with new CA certificate content. 
      # mk edit cm kapp-controller-config -n tkg-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  caCerts: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  imageRepository: airgap-repo-server.example.com/registry
...

      Copy the new CA certificate content and overwrite existing caCerts field.

      Save it with ":wq".

      Tfhen restart kapp-controller-manager deployment.

      # mk rollout restart deployment -n tkg-system kapp-controller

  6. Update manangement cluster kubecontrolplane and kubeadmconfigtemplate CRs.
    1. Update kubecontrolplane CR. 
      # mk get kcp -n tkg-system
NAME                                    CLUSTER            INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
ipv4-airgap-mgmt-master-control-plane   ipv4-airgap-mgmt   true          true                   1          1       1         0             11d   v1.22.9+vmware.1
# mk edit kcp -n tkg-system ipv4-airgap-mgmt-master-control-plane
...
    - content: <cert-base64-content>
      encoding: base64
      path: /etc/containerd/airgap-repo-server.example.crt
      permissions: "0444"
...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

      Note:

      This operation will result in control plane nodes redeployment.

    2. Update kubeadmconfigtemplate CR. 
      # mk get kubeadmconfigtemplate -n tkg-system
NAME                   AGE
ipv4-airgap-mgmt-np1   11d
# mk edit kubeadmconfigtemplate -n tkg-system ipv4-airgap-mgmt-np1
...
spec:
  template:
    spec:
      files:
      - content: <cert-base64-content>
        encoding: base64
        path: /etc/containerd/airgap-repo-server.example.com.crt
        permissions: "0444"
...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

  7. Update tcakubenetescluster CR in TCA CP minikube. 
    # kubectl config get-contexts
CURRENT   NAME                                      CLUSTER            AUTHINFO                 NAMESPACE
          ipv4-airgap-mgmt-admin@ipv4-airgap-mgmt   ipv4-airgap-mgmt   ipv4-airgap-mgmt-admin
          ipv4-mgmt-admin@ipv4-mgmt                 ipv4-mgmt          ipv4-mgmt-admin
*         minikube                                  minikube           minikube                 default
# kubectl config use-context minikube # if star is not on minikube in the last command
# kubectl get tkc -A
NAMESPACE          NAME               AGE
ipv4-airgap-mgmt   ipv4-airgap-mgmt   11d
ipv4-mgmt          ipv4-mgmt          13d
# kubectl edit tkc -n ipv4-airgap-mgmt ipv4-airgap-mgmt
...
    telco.vmware.com/airgap-ca-cert: <cert-base64-content>
    telco.vmware.com/airgap-fqdn: airgap-repo-server.example.com
...

    Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

    Save it with ":wq".