This document guides user to update airgap server CA certificate of an old version management cluster which is not upgraded to TCA 2.3.

Prerequisites

The tkgcontext of management cluster has been updated.

Procedure

  1. Update management cluster nodes.

    This step will update all the management cluster nodes. User needs to login to each node with capv account for updating. The user needs to get cluster node IP addresses by ccli show status command. For each output IP address, user need to follow the sub-steps.

    # ccli show status
status: Running
phase: PostConfig
nodes:
- ip: 172.16.69.112
  vmName: ipv4-airgap-mgmt-master-control-plane-qc6nx
  ...
- ip: 172.16.69.213
  vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-79xvg
  ...
- ip: 172.16.70.143
  vmName: ipv4-airgap-mgmt-np1-7648d7cd8f-svmz9
  ...
    1. Login to node from TCA CP appliance via ssh capv@<node ip> and enter root mode. 
      # ssh [email protected]
capv@ipv4-airgap-mgmt-master-control-plane-qc6nx [ ~ ]$ sudo su
    2. Copy the CA file with .pem suffix to /etc/ssl/certs folder and run rehash_ca_certificate.sh. Test the connectivity with curl command. 
      root [ /home/capv ]# cp ca.crt /etc/ssl/certs/airgap-repo-server.example.com.pem # end with .pem is a must!
root [ /home/capv ]# rehash_ca_certificates.sh
root [ /home/capv ]# curl https://airgap-repo-server.example.com -v --head # test it works
    3. Copy CA file to/etc/containerd/<airgap-server-fqdn>.crt and restart containerd with systemctl restart containerd, then verify it can pull image from airgap server with command crictl pull <airgap-server-fqdn>/registry/tkr-compatibility:v1. 
      root [ /home/capv ]# cp ca.crt /etc/containerd/airgap-repo-server.example.com.crt
root [ /home/capv ]# systemctl restart containerd
root [ /home/capv ]# systemctl status containerd
root [ /home/capv ]# crictl pull airgap-repo-server.example.com/registry/tkr-compatibility:v1 # test pulling from airgap repo
  2. Update management cluster configmaps.
    1. Update tkr-controller-config with new CA certificate content. 
      # mk edit cm tkr-controller-config -n tkr-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  caCerts: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  imageRepository: airgap-repo-server.example.com/registry
...

      Copy the new CA certificate content and overwrite the existing caCerts field.

      Save it with ":wq".

      Then restart tkr-controller-manager deployment.

      # mk rollout restart deployment -n tkr-system tkr-controller-manager

    2. Update kapp-controller-config with new CA certificate content. 
      # mk edit cm kapp-controller-config -n tkg-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  caCerts: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  imageRepository: airgap-repo-server.example.com/registry
...

      Copy the new CA certificate content and overwrite existing caCerts field.

      Save it with ":wq".

      Tfhen restart kapp-controller-manager deployment.

      # mk rollout restart deployment -n tkg-system kapp-controller

  3. Update manangement cluster kubecontrolplane and kubeadmconfigtemplate CRs.
    1. Update kubecontrolplane CR. 
      # mk get kcp -n tkg-system
NAME                                    CLUSTER            INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
ipv4-airgap-mgmt-master-control-plane   ipv4-airgap-mgmt   true          true                   1          1       1         0             11d   v1.22.9+vmware.1
# mk edit kcp -n tkg-system ipv4-airgap-mgmt-master-control-plane
...
    - content: <cert-base64-content>
      encoding: base64
      path: /etc/containerd/airgap-repo-server.example.crt
      permissions: "0444"
...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

      Note:

      This operation will result in control plane nodes redeployment.

    2. Update kubeadmconfigtemplate CR. 
      # mk get kubeadmconfigtemplate -n tkg-system
NAME                   AGE
ipv4-airgap-mgmt-np1   11d
# mk edit kubeadmconfigtemplate -n tkg-system ipv4-airgap-mgmt-np1
...
spec:
  template:
    spec:
      files:
      - content: <cert-base64-content>
        encoding: base64
        path: /etc/containerd/airgap-repo-server.example.com.crt
        permissions: "0444"
...

      Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

      Save it with ":wq".

  4. Update tcakubenetescluster CR in TCA CP minikube. 
    # kubectl config get-contexts
CURRENT   NAME                                      CLUSTER            AUTHINFO                 NAMESPACE
          ipv4-airgap-mgmt-admin@ipv4-airgap-mgmt   ipv4-airgap-mgmt   ipv4-airgap-mgmt-admin
          ipv4-mgmt-admin@ipv4-mgmt                 ipv4-mgmt          ipv4-mgmt-admin
*         minikube                                  minikube           minikube                 default
# kubectl config use-context minikube # if star is not on minikube in the last command
# kubectl get tkc -A
NAMESPACE          NAME               AGE
ipv4-airgap-mgmt   ipv4-airgap-mgmt   11d
ipv4-mgmt          ipv4-mgmt          13d
# kubectl edit tkc -n ipv4-airgap-mgmt ipv4-airgap-mgmt
...
    telco.vmware.com/airgap-ca-cert: <cert-base64-content>
    telco.vmware.com/airgap-fqdn: airgap-repo-server.example.com
...

    Locate the airgap server CA certificate file content and update it with new CA certificate base64 encoding string.

    Save it with ":wq".