This section describes procedure to configure VMware Tanzu Kubernetes Grid cluster using secure Harbor registry.

Procedure

  1. From VMware Telco Cloud Automation, add the secure Harbor registry in Partner Systems page by registering the available Harbor instance. Use fqdn with https in URL field. Click the Trust Certificate checkbox. In the VIM Associations tab, select the workload cluster that you use for VMware Telco Cloud Service Assurance deployment. To finish the registration, click the Finish button. For more information on how to add a Harbor registry in Partner Systems, see Add a Harbor Repository in VMware Telco Cloud Automation documentation.
  2. ssh to one of the CPN nodes of the management cluster. You can find the IPs under the Caas Infrastructure page on the left navigation link, select the management cluster from the list, then select the Control Plane Nodes tab, and the Nodes table must list the available CPN nodes.
  3. After you have logged in, use the following kubectl command to find the kapp-controller instance for the workload cluster that is used for deployment.
    capv@small-mgmt-cluster-master-control-plane-nsvtp [ ~ ]$ kubectl get apps -A
    NAMESPACE                   NAME                                        DESCRIPTION                                                                       SINCE-DEPLOY   AGE
    tcsa-test                   tcsa-test-kapp-controller                   Reconcile succeeded                                                               27s            2d6h
    tcsa-xlarge-cluster         tcsa-xlarge-cluster-kapp-controller         Canceled/paused                                                                   23h            26h
    tkg-system                  antrea                                      Reconcile succeeded                                                               4m43s          34d
    tkg-system                  metrics-server                              Reconcile succeeded                                                               22s            34d
    tkg-system                  tanzu-addons-manager                        Reconcile succeeded                                                               5m24s          34d
    tkg-system                  vsphere-cpi                                 Reconcile succeeded                                                               77s            34d
    tkg-system                  vsphere-csi                                 Reconcile succeeded                                                               2m21s          34d
    

    The kapp-controller instance to be updated is listed under the namespace of the same name of the workload cluster and the name of the app instance can be <workload_cluster_name>-kapp-controller.

    After you identify the kapp-controller instance for the workload cluster, edit the configuration by using the following command.
    kubectl edit app -n <workload_cluster_name> <workload_cluster_name>-kapp-controller
    For example:
    kubectl edit app -n tcsa-xlarge-cluster tcops-xlarge-cluster-kapp-controller
    You can edit the following two properties for the values.
    paused: true
    syncPeriod: 100000h0s
    You can find the properties defined in the following section of the application definition.
    spec:
      cluster:
        kubeconfigSecretRef:
          key: value
          name: tcops-xlarge-cluster-kubeconfig
      deploy:
      - kapp:
          rawOptions:
          - --wait-timeout=30s
      fetch:
      - imgpkgBundle:
          image: projects.registry.vmware.com/tkg/packages/core/kapp-controller:v0.23.0_vmware.1-tkg.1
      noopDelete: true
      paused: true
      syncPeriod: 5m0s
      template:

    If the paused property is not already defined, then add it to the spec as shown. Save the changes and exit.

  4. ssh to one of the CPN nodes of the workload cluster. You can find the IPs under the Caas Infrastructure page on the left navigation link, select the workload cluster from the list, then select the Control Plane Nodes tab, and the Nodes table must list the available CPN nodes. Alternatively, you can use the KUBECONFIG file for the workload cluster to execute kubectl commands against the cluster.
  5. After you have logged in, use the following kubectl command to find the kapp-controller configuration map instance used by kapp-controller running on the workload cluster.
    [root@tcsa ~]$ kubectl get cm -n tkg-system kapp-controller-config -o yaml | head -n 8
    apiVersion: v1
    data:
      caCerts: ""
      dangerousSkipTLSVerify: ""
      httpProxy: ""
      httpsProxy: ""
      noProxy: ""
    kind: ConfigMap
  6. Update the configuration map definition to add the certificate information to the {{caCerts }} using the following command:
    kubectl edit cm -n tkg-system kapp-controller-config
    After the update, caCerts property looks like the following:
    [root@tcsa ~]$ kubectl get cm -n tkg-system kapp-controller-config -o yaml | head -n 8
    apiVersion: v1
    data:
      caCerts: |
        -----BEGIN CERTIFICATE-----
        MIIGNDCCBBygAwIBAgIUeB0MR1bIB3wUlnTGoAs3JYUGcXMwDQYJKoZIhvcNAQEN
        BQAwgYcxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTESMBAGA1UEBwwJUGFsbyBB
        bHRvMQ4wDAYDVQQKDAVUZWxjbzEdMBsGA1UECwwUU29sdXRpb24gRW5naW5lZXJp
        bmcxKDAmBgNVBAMMH2hhYXMtd2d0MS05Ny0xMjAuZW5nLnZtd2FyZS5jb20wHhcN
    <removed some entries>
        ZQK7iLY80tbbSLuxnyrX1Oaq5U9pYsxjiCEt2XVzgOgfaZKUL6kD9U5LhI8Zj1qY
        nE3TsevcNE4LH3OXZqjUvpNhfBbMh2u+Ui3wFiwV0prjBQKeg8MCxBQJCVSmb/en
        q+UD0IwbIlg=
        -----END CERTIFICATE-----
    Note: The caCerts is a yaml file and proper indentation and spacing is required to keep the format of the file valid. The "|" character is the first character after the caCerts property name, which denotes a multi-line string. Lastly, there are four spaces of indentation for every line of the certificate string.
    If you want to add multiple CA certificates to the kapp-controller configuration, you must use the following format:
    # A cert chain of trusted ca certs. These will be added to the system-wide
      # cert pool of trusted ca's (optional)
      caCerts: |
        -----BEGIN CERTIFICATE-----
        Certificate 1
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        Certificate 2
        -----END CERTIFICATE-----
  7. After the configuration map is updated, a restart of the kapp-controller pod is required. Use the following command to restart the pod:
    kubectl rollout restart deployment -n tkg-system kapp-controller
    After the restart is complete, you can proceed with the VMware Telco Cloud Service Assurance deployment using the registry information including the CA cert property.
    For example:
    REGISTRY_URL=10.198.97.120/tcxdeployer/tcx
    REGISTRY_USERNAME=<username>
    REGISTRY_PASSWORD=<password>
    REGISTRY_CERTS_PATH=/home/root/harbor.crt
    Note: Customization to the deployment of the VMware Tanzu Kubernetes Grid management cluster has to be reverted in the event that the workload cluster must be updated or to perform any type of maintenance from the VMware Telco Cloud Automation Manager UI, including upgrades and cluster lifecycle management.