You can use an external container registry with TKG cluster pods.
Troubleshoot Errors Pulling Images from a Container Registry
If you configure the TKG with the certificates to trust, and you add the self-signed certificate to the cluster kubeconfig, you should be able to successfully pull a container image from a private registry that uses that self-signed certificate.
kubectl describe pod PODNAME
This commands shows detailed status and error messages for a given pod. An example of attempting to pull an image before adding custom certificates to the cluster:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 33s default-scheduler ... Normal Image 32s image-controller ... Normal Image 15s image-controller ... Normal SuccessfulRealizeNSXResource 7s (x4 over 31s) nsx-container-ncp ... Normal Pulling 7s kubelet Waiting test-gc-e2e-demo-ns/testimage-8862e32f68d66f727d1baf13f7eddef5a5e64bbd-v10612 Warning Failed 4s kubelet failed to get images: ... Error: ... x509: certificate signed by unknown authority
kubectl get pods
ErrImagePull
error is also visible in the overall Pod status view:
NAME READY STATUS RESTARTS AGE testimage-nginx-deployment-89d4fcff8-2d9pz 0/1 Pending 0 17s testimage-nginx-deployment-89d4fcff8-7kp9d 0/1 ErrImagePull 0 79s testimage-nginx-deployment-89d4fcff8-7mpkj 0/1 Pending 0 21s testimage-nginx-deployment-89d4fcff8-fszth 0/1 ErrImagePull 0 50s testimage-nginx-deployment-89d4fcff8-sjnjw 0/1 ErrImagePull 0 48s testimage-nginx-deployment-89d4fcff8-xr5kg 0/1 ErrImagePull 0 79sThe errors “x509: certificate signed by unknown authority” and “ErrImagePull” indicate that cluster is not configured with the correct certificate to connect to the private container registry. Either the certificate is missing, or it is misconfigured.
If you are experiencing errors connecting to a private registry after configuring the certificates, you can check if certificates applied in the configuration are applied to the cluster. You can check whether the certificates were applied in their configuration have been applied properly by using SSH.
- Check the folder
/etc/ssl/certs/
for files namedtkg-<cert_name>.pem
, where<cert_name>
is the "name" property of the certificate added in theTkgServiceConfiguration
. If the certificates match what is in theTkgServiceConfiguration
, and using a private registry still does not work, diagnose further by completing the next step. - Run the following openssl connection test to the target server using self-signed certificates by executing the command
openssl s_client -connect hostname:port_num
, wherehostname
is the host name/DNS name of the private registry that is using self-signed certificates, andport_num
is the port number that the service is running on (usually 443 for HTTPS).You can check exactly what error is being returned by openssl when attempting to connect to the endpoint that is using self-signed certificates, and remedy the situation from there, for example, by adding the right certificates to the
TkgServiceConfiguration
. If the TKG cluster is embedded with the wrong certificate, you will need to update the configuration with the correct certificates, delete the TKG cluster, then recreate it using the configuration that contains the correct certificates. - Verify that the contents of the secret's data map are double base64-encoded. Double base64-encoding is required. If the contents of the data map value are not double base6-encoded, the resulting PEM file cannot be processed.