This document guides user to update airgap server CA certificate of a management cluster manually. Cluster operations, such as creating workload clusters, scaling in/out control plane or node pools, can recover after updating the cluster with new airgap server CA certificate.

Prerequisites

Airgap server is updated with new self-signed certificate or private root CA signed certificate.

Note:

If airgap server is updated with public signed certificate, updating existing cluster or following the steps in this guide is not required.

Procedure

  1. Login to TCA CP appliance via SSH with admin account and enter root mode.
  2. Get Resources.
    1. Find the management cluster id via CCLI.
      # ccli list mc
      +-------+------------------+------------------+--------------------------------------+--------------+----------+
      | INDEX | NAME             | CONTEXT          | ID                                   | CONNECTIVITY | SELECTED |
      +-------+------------------+------------------+--------------------------------------+--------------+----------+
      | 0     | ipv4-airgap-mgmt | ipv4-airgap-mgmt | 97bad1f2-bbed-4019-b226-b211f50bb3f4 | true         |          |
      | 1     | ipv4-mgmt        | ipv4-mgmt        | b8e55039-63f8-447c-93c3-b86bf26c8412 | true         |          |
      +-------+------------------+------------------+--------------------------------------+--------------+----------+
      # ccli go 0

      In the following examples, "ipv4-airgap-mgmt" cluster is used, with id "97bad1f2-bbed-4019-b226-b211f50bb3f4".

    2. Locate management cluster kubeconfig file and create alias of management cluster kubectl command.

      The management cluster kubeconfig file is located at /opt/vmware/k8s-bootstrapper/<cluster id>/kubeconfig.

      # alias mk="kubectl --kubeconfig /opt/vmware/k8s-bootstrapper/97bad1f2-bbed-4019-b226-b211f50bb3f4/kubeconfig"
  3. Update tkgcontext of management cluster.
    1. Get cluster tkgcontext id via CCLI.

      In the output of ccli show spec command, tkgID is the tkgcontext id.

      # ccli show spec
      tkgID: 38561a18-3ea9-402f-9568-dca4889a6b11
      id: 97bad1f2-bbed-4019-b226-b211f50bb3f4
      clusterName: ipv4-airgap-mgmt
      kubernetesVersion: v1.22.9+vmware.1
      serviceCIDR: 100.64.0.0/13
      clusterCIDR: 100.96.0.0/11
      ....
    2. Save, edit and update tkgcontext.
      1. Get tkgcontext json configuration file via curl command and save it to a file.

      2. Modify the file and update its airgap/caCert field with ca file base64 encoding string. If no caCert field, create it in json format. Add vsphere/password field with vc password as value.

      3. Modify tkgcontext via curl command and saved json file.

      4. Delete the json file for security consideration.

      # curl -X GET http://localhost:8888/api/v1/tkgcontext/38561a18-3ea9-402f-9568-dca4889a6b11 | python -m json.tool > tkgcontext.json
      # vi tkgcontext.json # update vc password and caCert
      {
          "id": "38561a18-3ea9-402f-9568-dca4889a6b11",
          "ipFamilies": "IPv4",
          "infrastructure": {
              "type": "vsphere",
              "version": "7.0.3"
          },
          "vsphere": {
              "password": "<my vc password>"
              ...
          },
          "airgap": {
              "fqdn": "airgap-repo-serve.example.com",
              "caCert": "<update it with new ca.crt base64 string"
              }
          }
      }
      # curl -X PUT http://localhost:8888/api/v1/tkgcontext/38561a18-3ea9-402f-9568-dca4889a6b11 -d "`cat tkgcontext.json`"
  4. Update management cluster nodes.

    This step will update all the management cluster nodes. User needs to login to each node with capv account for updating. The user needs to get cluster node IP addresses by ccli show status command. For each output IP address, user need to follow the sub-steps.

    # ccli show status
    status: Running
    phase: PostConfig
    nodes:
    - ip: 172.16.69.112
      vmName: ipv4-airgap-mgmt-master-control-plane-qc6nx
      ...
    - ip: 172.16.69.213
      vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-79xvg
      ...
    - ip: 172.16.70.143
      vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-svmz9
      ...
    1. Login to node from TCA CP appliance via ssh capv@<node ip> and enter root mode.
      # ssh capv@172.16.69.112
      capv@ipv4-airgap-mgmt-master-control-plane-qc6nx [ ~ ]$ sudo su
    2. Copy the CA file with .pem suffix to /etc/ssl/certs folder and run rehash_ca_certificate.sh. Test the connectivity with curl command.
      root [ /home/capv ]# cp ca.crt /etc/ssl/certs/airgap-repo-server.example.com.pem # end with .pem is a must!
      root [ /home/capv ]# rehash_ca_certificates.sh
      root [ /home/capv ]# curl https://airgap-repo-server.example.com -v --head # test it works
    3. Copy CA file to/etc/containerd/<airgap-server-fqdn>.crt and restart containerd with systemctl restart containerd, then verify it can pull image from airgap server with command crictl pull <airgap-server-fqdn>/registry/tkr-compatibility:v1.
      root [ /home/capv ]# cp ca.crt /etc/containerd/airgap-repo-server.example.com.crt
      root [ /home/capv ]# systemctl restart containerd
      root [ /home/capv ]# systemctl status containerd
      root [ /home/capv ]# crictl pull airgap-repo-server.example.com/registry/tkr-compatibility:v1 # test pulling from airgap repo
  5. Update management cluster configmaps.
    1. Update tkr-controller-config with new CA certificate content.
      # mk edit cm tkr-controller-config -n tkr-system
      # Please edit the object below. Lines beginning with a '#' will be ignored,
      # and an empty file will abort the edit. If an error occurs while saving this file will be
      # reopened with the relevant failures.
      #
      apiVersion: v1
      data:
        caCerts: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
        imageRepository: airgap-repo-server.example.com/registry
      ...

      Copy the new CA certificate content and overwrite the existing caCerts field.

      Save it with ":wq".

      Then restart tkr-controller-manager deployment.

      # mk rollout restart deployment -n tkr-system tkr-controller-manager

    2. Update kapp-controller-config with new CA certificate content.
      # mk edit cm kapp-controller-config -n tkg-system
      # Please edit the object below. Lines beginning with a '#' will be ignored,
      # and an empty file will abort the edit. If an error occurs while saving this file will be
      # reopened with the relevant failures.
      #
      apiVersion: v1
      data:
        caCerts: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
        imageRepository: airgap-repo-server.example.com/registry
      ...

      Copy the new CA certificate content and overwrite existing caCerts field.

      Save it with ":wq".

      Tfhen restart kapp-controller-manager deployment.

      # mk rollout restart deployment -n tkg-system kapp-controller

  6. Update manangement cluster kubecontrolplane and kubeadmconfigtemplate CRs.
    1. Update kubecontrolplane CR.
      # mk get kcp -n tkg-system
      NAME                                    CLUSTER            INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
      ipv4-airgap-mgmt-master-control-plane   ipv4-airgap-mgmt   true          true                   1          1       1         0             11d   v1.22.9+vmware.1
      # mk edit kcp -n tkg-system ipv4-airgap-mgmt-master-control-plane
      ...
          - content: <cert-base64-content>
            encoding: base64
            path: /etc/containerd/airgap-repo-server.example.crt
            permissions: "0444"
      ...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

      Note:

      This operation will result in control plane nodes redeployment.

    2. Update kubeadmconfigtemplate CR.
      # mk get kubeadmconfigtemplate -n tkg-system
      NAME                   AGE
      ipv4-airgap-mgmt-np1   11d
      # mk edit kubeadmconfigtemplate -n tkg-system ipv4-airgap-mgmt-np1
      ...
      spec:
        template:
          spec:
            files:
            - content: <cert-base64-content>
              encoding: base64
              path: /etc/containerd/airgap-repo-server.example.com.crt
              permissions: "0444"
      ...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

  7. Update tcakubenetescluster CR in TCA CP minikube.
    # kubectl config get-contexts
    CURRENT   NAME                                      CLUSTER            AUTHINFO                 NAMESPACE
              ipv4-airgap-mgmt-admin@ipv4-airgap-mgmt   ipv4-airgap-mgmt   ipv4-airgap-mgmt-admin
              ipv4-mgmt-admin@ipv4-mgmt                 ipv4-mgmt          ipv4-mgmt-admin
    *         minikube                                  minikube           minikube                 default
    # kubectl config use-context minikube # if star is not on minikube in the last command
    # kubectl get tkc -A
    NAMESPACE          NAME               AGE
    ipv4-airgap-mgmt   ipv4-airgap-mgmt   11d
    ipv4-mgmt          ipv4-mgmt          13d
    # kubectl edit tkc -n ipv4-airgap-mgmt ipv4-airgap-mgmt
    ...
        telco.vmware.com/airgap-ca-cert: <cert-base64-content>
        telco.vmware.com/airgap-fqdn: airgap-repo-server.example.com
    ...

    Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

    Save it with ":wq".