Integrate TKG Service Clusters with a Private Container Registry

Refer to this topic to integrate TKG Service clusters with a private container registry.

Private Container Registry Use Case

Container registries provide a critical function for Kubernetes deployments, serving as a centralized repository for storing and sharing container images. The most commonly used public container registry is Docker Hub. There are many private container registry offerings. VMware Harbor is an open-source, cloud native, private container registry that ships with Supervisor.

Private Container Registry Integration

To integrate a private registry with a TKG Service cluster, you configure the cluster with one or more self-signed CA certificates to serve private registry content over HTTPS. To do this, in the cluster spec you include a trust field with the additionalTrustedCAs value. You can define any number of self-signed CA certificates that the TKGS cluster should trust. This functionality lets you easily define a list of certificates, and update those that need rotation.

You can configure the private registry certificate when you initially create the cluster, or you can update an existing cluster and provide the private registry certificate. To edit an existing cluster and add the private registry certificate, use the kubectl edit method. See Configure a Text Editor for Kubectl.

Be aware that the implementation of the trust.additionalTrustedCAs field differs slightly between the supported APIs for provisioning TKGS clusters. Refer to the tables v1alpha3 API Trust Fields and v1beta1 API Trust Variable for details.

v1alpha3 API Example

The following example demonstrates how to integrate a TKG Service cluster provisioned using the v1alpha3 API with a private container registry using its CA certificate.

With the TanzuKubernetesCluster v1alpha3 API, the trust.additionalTrustedCAs field includes one or more name-data pairs, each of which can include a TLS certificate for a private registry.

Table 1. v1alpha3 API Trust Fields
Field	Description
`trust`	Section marker. Accepts no data.
`additionalTrustedCAs`	Section marker. Includes a array of certificates with `name` and `data` fields for each.
`name`	User defined name of the CA certificate.
`data`	Contents of the CA certificate (`ca.crt`) in PEM format that is double base64-encoded. Note: The v1alpha3 API requires the certificate contents to be single base64-encoded. If the contents are not single base6-encoded, the resulting PEM file cannot be processed.

Use the following procedure to integrate Harbor with a v1alpha3 API cluster using the Harbor Registry Certificate.

Download the Harbor Registry Certificate from the Harbor web interface at the Projects > Repositories screen.
The CA certificate file downloads as ca.crt.
Single base64-encode the contents of the CA certificate.
- Linux: base64 -w 0 ca.crt
- Windows: https://www.base64encode.org/.

Include the trust.additionalTrustedCAs fields in the cluster specification and populate with the name and data values.

#tkc-with-trusted-private-reg-cert.yaml
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: tkc01
  namespace: tkgs-cluster-ns
spec:
   topology:
     controlPlane:
       replicas: 3
       storageClass: tkgs-storage-policy
       vmClass: guaranteed-medium
       tkr:
         reference:
           name: v1.25.7---vmware.3-fips.1-tkg.1
     nodePools:
     - name: nodepool-01
       replicas: 3
       storageClass: tkgs-storage-policy
       vmClass: guaranteed-medium
       tkr:
         reference:
           name: v1.25.7---vmware.3-fips.1-tkg.1
   settings:
     storage:
       defaultClass: tkgs-storage-policy
     network:
       cni:
         name: antrea
       services:
         cidrBlocks: ["198.51.100.0/12"]
       pods:
         cidrBlocks: ["192.0.2.0/16"]
       serviceDomain: cluster.local
       trust:
         additionalTrustedCAs:
         - name: CompanyInternalCA-1
           data: LS0tLS1C...LS0tCg==
         - name: CompanyInternalCA-2
           data: MTLtMT1C...MT0tPg==

To rotate a certificate, kubectl edit the cluster specification and update the trust.additionalTrustedCAs.data value, then initiate a rolling update.

v1beta1 API Example

The following example describes how to integrate a TKG Service cluster provisioned using the v1beta1 API with a private container registry using its CA certificate.

To integrate a private container registry with a TKGS cluster provisioned with the Cluster v1beta1 API, you use the trust variable and populate it with a Kubernetes secret containing the private registry certificate.

Table 2. v1beta1 API Trust Variable
Field	Description
`trust`	Section marker. Accepts no data.
`additionalTrustedCAs`	Section marker. Includes an array of certificates with the name for each.
`name`	The user-defined name for the `data` map field in the Kubernetes secret that contains the CA certificate in PEM format that is double base64-encoded. Note: The v1beta1 API requires the certificate contents to be double base64-encoded. If the contents are not double base6-encoded, the resulting PEM file cannot be processed.

Use the following procedure to integrate Harbor with a v1beta1 API cluster using the Harbor Registry Certificate.

Download the Harbor Registry Certificate from the Harbor web interface at the Projects > Repositories screen.
The CA certificate file downloads as ca.crt.
Double base64-encode the contents of the CA certificate.
- Linux: base64 -w 0 ca.crt | base64 -w 0
- Windows: https://www.base64encode.org/.
Create a Kubernetes secret definition YAML file with the following contents.
```
#additional-ca-1.yaml
apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09
kind: Secret
metadata:
  name: cluster01-user-trusted-ca-secret
  namespace: tkgs-cluster-ns
type: Opaque
```
Where:
- The value of the secret's data map is a user-defined string which is the name for the CA certificate (additional-ca-1 in this example) whose value is a double base64-encoded certificate.
- In the metadata section, name the secret CLUSTER-NAME-user-trusted-ca-secret, where CLUSTER-NAME is the name of the Cluster. This secret must be created in the same vSphere Namespace as the Cluster.
Create the Kubernetes secret declaratively.
```
kubectl apply -f additional-ca-1.yaml
```

Verify secret creation.

kubeclt get secret -n tkgs-cluster-ns cluster01-user-trusted-ca-secret
NAME                                             TYPE     DATA   AGE
cluster01-user-trusted-ca-secret                 Opaque   12     2d22h

Include the trust variable in the Cluster spec that references the name of the data map for the secret, which in this example is additional-ca-1.

#cluster-with-trusted-private-reg-cert.yaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: cluster01
  namespace: tkgs-cluster-ns
spec:
  clusterNetwork:
    services:
      cidrBlocks: ["198.52.100.0/12"]
    pods:
      cidrBlocks: ["192.101.2.0/16"]
    serviceDomain: "cluster.local"
  topology:
    class: tanzukubernetescluster
    version: v1.26.5+vmware.2-fips.1-tkg.1
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
        - class: node-pool
          name: node-pool-01
          replicas: 3
    variables:
      - name: vmClass
        value: guaranteed-medium
      - name: storageClass
        value: tkgs-storage-profile
      - name: defaultStorageClass
        value: tkgs-storage-profile
      - name: trust
        value:
          additionalTrustedCAs:
          - name: additional-ca-1

To rotate the certificate, create a new secret and edit the cluster specification with the appropriate value. This will trigger a rolling update of the cluster.
Note: The system does not monitor changes to the CLUSTER-NAME-user-trusted-ca-secret. If its data map value changes, these changes will not be reflected in the cluster. You will need to create a new secret and its data map name to trust.additionalTrustCAs.