You can use an external container registry with Tanzu Kubernetes cluster pods. This is an alternative to using the embedded Harbor Registry.
External Private Registry Use Case
Container registries provide a critical function for Kubernetes deployments, serving as a centralized repository for storing and sharing container images. The most commonly used public container registry is DockerHub. There are many private container registry offerings. VMware Harbor is an open-source, cloud native, private container registry. vSphere with Tanzu embeds an instance of Harbor that you can use as the private container registry for vSphere Pods and for pods running on Tanzu Kubernetes clusters. For more information, see Enable the Embedded Harbor Registry on the Supervisor Cluster.
The embedded Harbor Registry that ships with vSphere with Tanzu requires NSX-T networking. If you are using vSphere networking, you cannot use it. In addition, you may already be running your own private container registry that you want to integrate with your Tanzu Kubernetes clusters. In this case you can configure the Tanzu Kubernetes Grid Service to trust private registries with self-signed certificates, thereby allowing Kubernetes pods running on Tanzu Kubernetes clusters to use the external registry.
External Private Registry Requirements
To use an external private registry with Tanzu Kubernetes clusters, you must use vSphere with Tanzu version 7 U2 or later.
You can only use your own private registry with Kubernetes pods that run on Tanzu Kubernetes clusters and Tanzu Kubernetes release node VMs. You cannot use your own private registry with vSphere Pods that run natively on ESXi hosts. The supported registry for vSphere Pods is the Harbor Registry that is embedded in the vSphere with Tanzu platform.
Once you configure the Tanzu Kubernetes Grid Service for a private registry, any new cluster that is provisioned will support the private registry. For existing clusters to support the private registry, a rolling update is required to apply the
TkgServiceConfiguration. See Update Tanzu Kubernetes Clusters. In addition, the first time you create a custom
TkgServiceConfiguration, the system will initiate a rolling update.
External Private Registry Configuration
To use your own private registry with Tanzu Kubernetes clusters, you configure the Tanzu Kubernetes Grid Service with one or more self-signed certificates to serve private registry content over HTTPS.
TkgServiceConfiguration is updated to support self-signed certificates for the private registry. Specifically, a new
trust section with the
additionalTrustedCAs field is added, allowing you to define any number of self-signed certificates that Tanzu Kubernetes clusters should trust. This functionality lets you easily define a list of certificates, and update those certificates should they need rotation.
TkgServiceConfiguration is updated and applied, the TLS certificates will be applied to new clusters the next time a cluster is created. In other words, applying an update to
TkgServiceConfiguration.trust.additionalTrustedCAs does not trigger an automatic rolling update of Tanzu Kubernetes clusters.
apiVersion: run.tanzu.vmware.com/v1alpha1 kind: TkgServiceConfiguration metadata: name: tkg-service-configuration spec: defaultCNI: antrea trust: additionalTrustedCAs: - name: first-cert-name data: base64-encoded string of a PEM encoded public cert 1 - name: second-cert-name data: base64-encoded string of a PEM encoded public cert 2
kubectl apply -f tkgserviceconfiguration.yaml
Because the Tanzu Kubernetes Grid Service specification is being updated with the private registry certificates, you do not need to add the public key to the Tanzu Kubernetes cluster kubeconfig as you do when using the embedded Harbor Registry with Tanzu Kubernetes clusters.
Configure a Tanzu Kubernetes Workload to Pull Images from a Private Container Registry
To pull images from a private container registry for a Tanzu Kubernetes cluster workload, configure the workload YAML with the private registry details.
- Create an example pod spec with the details about the private registry.
apiVersion: v1 kind: Pod metadata: name: <workload-name> namespace: <kubernetes-namespace> spec: containers: - name: private-reg-container image: <Registry-IP-Address>/<vsphere-namespace>/<image-name>:<version> imagePullSecrets: - name: <registry-secret-name>
<workload-name>with the name of the pod workload.
<kubernetes-namespace>with the Kubernetes namespace in the cluster where the pod will be created. This must be the same Kubernetes namespace where the Registry Service image pull secret is stored in the Tanzu Kubernetes cluster (such as the default namespace).
<Registry-IP-Address>with the IP address for the embedded Harbor Registry instance running on the Supervisor Cluster.
- Replace <vsphere-namespace> with the vSphere Namespace where the target Tanzu Kubernetes is provisioned.
<image-name>with an image name of your choice.
<version>with an appropriate version of the image, such as "latest".
<registry-secret-name>with the name of the Registry Service image pull secret that you created previously.
- Create a workload in the Tanzu Kubernetes cluster based on the pod specification you defined.
kubectl --kubeconfig=<path>/cluster-kubeconfig apply -f <pod.yaml>
The pod should be created from the image pulled from the registry.
Trust Fields for External Private Registries
Add a certificate entry (base64-encoded string of a PEM-encoded public certificate) to the
additionalTrustedCAs section in the
TkgServiceConfiguration. The data is public certificates stored in plain text in the TkgServiceConfiguration.
||Section marker. Accepts no data.|
||Section marker. Includes a array of certificates with name and data for each.|
||The name of the TLS certificate.|
||The base64-encoded string of a PEM encoded public certificate.|
Removing External Private Registry Certificates
Remove a certificate from the list of certificates in the
additionalTrustedCAs section in the
TkgServiceConfiguration and apply the
TkgServiceConfiguration to the Tanzu Kubernetes Grid Service.
Rotating External Private Registry Certificates
To rotate a certificate, the VI Admin or DevOps Engineer would change the contents of the certificate in the
TkgServiceConfiguration or the Tanzu Kubernetes cluster specification and apply that configuration to trigger a rolling update of that TKC.
Troubleshooting External Private Registry Certificates
If you configure the Tanzu Kubernetes Grid Service with the certificates to trust, and you add the self-signed certificate to the cluster kubeconfig, you should be able to successfully pull a container image from a private registry that uses that self-signed certificate.
The following command can help you determine if the container image has been successfully pulled for a Pod workload:
kubectl describe pod PODNAME
This commands shows detailed status and error messages for a given pod. An example of attempting to pull an image before adding custom certificates to the cluster:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 33s default-scheduler ... Normal Image 32s image-controller ... Normal Image 15s image-controller ... Normal SuccessfulRealizeNSXResource 7s (x4 over 31s) nsx-container-ncp ... Normal Pulling 7s kubelet Waiting test-gc-e2e-demo-ns/testimage-8862e32f68d66f727d1baf13f7eddef5a5e64bbd-v10612 Warning Failed 4s kubelet failed to get images: ... Error: ... x509: certificate signed by unknown authority
kubectl get pods
ErrImagePullerror is also visible in the overall Pod status view:
NAME READY STATUS RESTARTS AGE testimage-nginx-deployment-89d4fcff8-2d9pz 0/1 Pending 0 17s testimage-nginx-deployment-89d4fcff8-7kp9d 0/1 ErrImagePull 0 79s testimage-nginx-deployment-89d4fcff8-7mpkj 0/1 Pending 0 21s testimage-nginx-deployment-89d4fcff8-fszth 0/1 ErrImagePull 0 50s testimage-nginx-deployment-89d4fcff8-sjnjw 0/1 ErrImagePull 0 48s testimage-nginx-deployment-89d4fcff8-xr5kg 0/1 ErrImagePull 0 79sThe errors “x509: certificate signed by unknown authority” and “ErrImagePull” indicate that cluster is not configured with the correct certificate to connect to the private container registry. Either the certificate is missing, or it is misconfigured.
If you are experiencing errors connecting to a private registry after configuring the certificates, you can check if certificates applied in the configuration are applied to the cluster. You can check whether the certificates were applied in their configuration have been applied properly by using SSH.
- Check the folder
/etc/ssl/certs/for files named
<cert_name>is the "name" property of the certificate added in the
TkgServiceConfiguration. If the certificates match what is in the
TkgServiceConfiguration, and using a private registry still does not work, diagnose further by completing the next step.
- Run the following openssl connection test to the target server using self-signed certificates by executing the command
openssl s_client -connect hostname:port_num, where hostname is the host name/DNS name of the private registry that is using self-signed certificates, and port_num is the port number that the service is running on (usually 443 for HTTPS). You can check exactly what error is being returned by openssl when attempting to connect to the endpoint that is using self-signed certificates, and remedy the situation from there, for example, by adding the right certificates to the
TkgServiceConfiguration. If the Tanzu Kubernetes cluster is embedded with the wrong certificate, you will need to update the Tanzu Kubernetes Grid Service configuration with the correct certificates, delete the Tanzu Kubernetes cluster, then recreate it using the configuration that contains the correct certificates.