Install Harbor in Workload Clusters Deployed by a Standalone Management Cluster

This topic explains how to deploy Harbor into a workload cluster or shared services cluster in Tanzu Kubernetes Grid.

Note

On vSphere with Tanzu you can install Harbor as a Supervisor Service Installing and Configuring Harbor Supervisor Service or on a TKG cluster.

Notary and Chartmuseum are deprecated in Harbor v2.6 and are scheduled to be removed in a future release as noted in the Harbor v2.6.0 release notes. Users should switch to Sigstore Cosign for container signing and verification.

When upgrading Harbor, VMware recommends only upgrading from N-1 or N-2 versions, to avoid database migration gaps.

Harbor

Harbor is an open-source, trusted, cloud-native container registry that stores, signs, and scans content. Tanzu Kubernetes Grid includes signed, packaged binaries for Harbor which you can deploy into a workload cluster to provide container registry services for that cluster. This Harbor package extends the open-source Docker distribution by adding functionalities usually required by users such as security and identity control and management.

Tanzu Kubernetes Grid includes signed binaries for Harbor, which you can deploy into:

A workload cluster to provide container registry services for that clusters
A shared services cluster, to provide container registry services for other workload clusters, in a deployment with a standalone management cluster.

When deployed as a shared service, Harbor is available to all of the workload clusters managed by the same standalone management cluster. To implement Harbor as a shared service, you deploy it into a special cluster that is dedicated to running shared services. Each management cluster can only have one shared service cluster.

Harbor Registry and ExternalDNS

VMware recommends installing ExternalDNS alongside the Harbor registry on infrastructures with load balancing, especially in production or other environments in which Harbor availability is important.

If the IP address to the ingress load balancer changes, ExternalDNS automatically picks up the change and re-maps the new address to the Harbor hostname. This precludes the need to manually re-map the address as described in Connect to the Harbor User Interface.

Prerequisites

You have a running vSphere with Tanzu Supervisor cluster or a standalone management cluster.
You have installed the Tanzu CLI and kubectl. For instructions, see Install the Tanzu CLI and Kubernetes CLI for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
You have installed yq v4.5 or later.
You have installed imgpkg.
You have logged in to the Tanzu CLI with the tanzu context use command.
You have installed imgpkg. For information about how to download and install imgpkg, see Install the Carvel Tools.
If you are deploying Harbor into a shared services cluster, the cluster has been created, as described in Create a Shared Services Cluster.

Note
You cannot use Harbor’s proxy cache feature for running Tanzu Kubernetes Grid v2.3 in an internet-restricted environment. You can still use a Harbor proxy cache to proxy images from prior versions of Tanzu Kubernetes Grid, and non-Tanzu images such as application images.

Prepare a Cluster for Harbor Deployment

To prepare a cluster for Harbor deployment:

Set the context of kubectl to the workload cluster or the shared services cluster. For example:
```
kubectl config use-context tkg-services-admin@tkg-services
```
If the cluster does not have a package repository with the Harbor package installed, such as the tanzu-standard repository, install one:
```
tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
```
Where:
- PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
- PACKAGE-REPO-ENDPOINT is the URL of the package repository.
  - For TKG v2.5.0 release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2024.2.1. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
If you have not already done so, install the cert-manager and Contour packages. For instructions, see Install Contour for Ingress Control.
(Optional) Install the ExternalDNS package. For instructions, see Install ExternalDNS for Service Discovery.
Proceed to Deploy Harbor into a Cluster below.

Deploy Harbor into a Cluster

Follow this procedure to deploy Harbor into a workload or shared services cluster:

Confirm that the Harbor package is available in the cluster:
```
tanzu package available list -A
```

Retrieve the version of the available package:

tanzu package available list harbor.tanzu.vmware.com -A

Create a configuration file for the Harbor package by retrieving the default configuration of the package:
```
tanzu package available get harbor.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
Where PACKAGE-VERSION is the version of package as listed by tanzu package available list but with a _ character substituted for the + character. It must also include the prefix v. For example, v2.9.1_vmware.1-tkg.1.
Note
The above method using --default-values-file-output only renders part of the configuration. To obtain a full configuration file for the Harbor package, use imgpkg to get it from the bundle. For example:
```
imgpkg pull -b projects.registry.vmware.com/tkg/packages/standard/harbor:PACKAGE-VERSION -o /tmp/harbor-package-PACKAGE-VERSION
```

Set the mandatory passwords and secrets in the harbor-data-values.yaml file by doing one of the following:

To automatically generate random passwords and secrets, run:

image_url=$(kubectl -n tkg-system get packages harbor.tanzu.vmware.com.PACKAGE-VERSION -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
imgpkg pull -b $image_url -o /tmp/harbor-package-PACKAGE-VERSION

cp /tmp/harbor-package-PACKAGE-VERSION/config/values.yaml harbor-data-values.yaml

bash /tmp/harbor-package-PACKAGE-VERSION/config/scripts/generate-passwords.sh harbor-data-values.yaml

Where PACKAGE-VERSION is the version of the Harbor package that you want to install.

For example, for the Harbor package v2.9.1, run:

image_url=$(kubectl -n tkg-system get packages harbor.tanzu.vmware.com.2.9.1+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
imgpkg pull -b $image_url -o /tmp/harbor-package-2.9.1
bash /tmp/harbor-package-2.9.1/config/scripts/generate-passwords.sh harbor-data-values.yaml

To set your own passwords and secrets, update the following entries in the harbor-data-values.yaml file:
- harborAdminPassword
- secretKey
- database.password
- core.secret
- core.xsrfKey
- jobservice.secret
- registry.secret

Specify other settings in the harbor-data-values.yaml file.
- Set the hostname setting to the hostname you want to use to access Harbor. For example, harbor.yourdomain.com.
- To use your own certificates, update the tls.crt, tls.key, and ca.crt settings with the contents of your certificate, key, and CA certificate. The certificate can be signed by a trusted authority or be self-signed. If you leave these blank, Tanzu Kubernetes Grid automatically generates a self-signed certificate.
- If you used the generate-passwords.sh script, optionally update the harborAdminPassword with something that is easier to remember.
- Non-empty values are required for the following:
  - storageClass: Under persistence.persistentVolumeClaim, for registry, jobservice.jobLog, database, redis, and trivy, set storageClass to a storage profile returned by kubectl get sc.
  Note
  With the azure-file storage class you cannot change filesystem permissions after the disk is mounted, because of an Azure issue described in “Could not change permissions” error while using Azure Files in the Azure documentation.
  - pspNames: Set pspNames to PSP values returned by kubectl get psp, for example, "vmware-system-restricted,vmware-system-privileged".
- Optionally, update other persistence settings to specify how Harbor stores data.
  
  If you need to store a large quantity of container images in Harbor, set persistence.persistentVolumeClaim.registry.size to a larger number.
To see more information about the values in the harbor-data-values.yaml file, run the below command against your target cluster:
```
tanzu package available get harbor.tanzu.vmware.com/AVAILABLE-VERSION --values-schema
```
Where AVAILABLE-VERSION is the version of the Harbor package. The --values-schema flag retrieves the valuesSchema section from the Package API resource for the Harbor package. You can set the output format, --output, for the values schema to yaml, json, or table.

For example:
```
tanzu package available get harbor.tanzu.vmware.com/2.9.1+vmware.1-tkg.1 --values-schema
```

Remove all comments in the harbor-data-values.yaml file:

yq -i eval '... comments=""' harbor-data-values.yaml

Install the package:
```
tanzu package install harbor \
--package harbor.tanzu.vmware.com \
--version AVAILABLE-PACKAGE-VERSION \
--values-file harbor-data-values.yaml \
--namespace TARGET-NAMESPACE
```
Where:
- TARGET-NAMESPACE is the namespace in which you want to install the Harbor package. For example, the my-packages or tanzu-cli-managed-packages namespace.
  - If the --namespace flag is not specified, the Tanzu CLI installs the package and its resources in the default namespace. The Harbor pods and any other resources associated with the Harbor component are created in the tanzu-system-registry namespace; do not install the Harbor package into this namespace.
  - The specified namespace must already exist, for example from running kubectl create namespace my-packages.
- AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.
For example:
```
tanzu package install harbor \
--package harbor.tanzu.vmware.com \
--version 2.9.1+vmware.1-tkg.1 \
--values-file harbor-data-values.yaml \
--namespace my-packages
```
Confirm that the harbor package has been installed:
```
tanzu package installed list -A
```
To see more details about the package, you can also run:
```
tanzu package installed get harbor --namespace PACKAGE-NAMESPACE
```
Where PACKAGE-NAMESPACE is the namespace in which the harbor package is installed.
Confirm that the harbor app has been successfully reconciled in your PACKAGE-NAMESPACE:
```
kubectl get apps -A
```
If the status is not Reconcile Succeeded, view the full status details of the harbor app. Viewing the full status can help you troubleshoot the problem.
```
kubectl get app harbor --namespace PACKAGE-NAMESPACE -o yaml
```
Where PACKAGE-NAMESPACE is the namespace in which you installed the package. If troubleshooting does not help you solve the problem, you must uninstall the package before installing it again:
```
tanzu package installed delete harbor --namespace PACKAGE-NAMESPACE
```

Confirm that the Harbor services are running by listing all of the pods in the cluster:

kubectl get pods -A

In the tanzu-system-registry namespace, you should see the harbor core, database, jobservice, notary, portal, redis, registry, and trivy services running in a pod with names similar to the following:

NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
[...]
tanzu-system-ingress    contour-6b568c9b88-h5s2r                1/1     Running   0          26m
tanzu-system-ingress    contour-6b568c9b88-mlg2r                1/1     Running   0          26m
tanzu-system-ingress    envoy-wfqdp                             2/2     Running   0          26m
tanzu-system-registry   harbor-core-557b58b65c-4kzhn            1/1     Running   0          23m
tanzu-system-registry   harbor-database-0                       1/1     Running   0          23m
tanzu-system-registry   harbor-jobservice-847b5c8756-t6kfs      1/1     Running   0          23m
tanzu-system-registry   harbor-notary-server-6b74b8dd56-d7swb   1/1     Running   2          23m
tanzu-system-registry   harbor-notary-signer-69d4669884-dglzm   1/1     Running   2          23m
tanzu-system-registry   harbor-portal-8f677757c-t4cbj           1/1     Running   0          23m
tanzu-system-registry   harbor-redis-0                          1/1     Running   0          23m
tanzu-system-registry   harbor-registry-85b96c7777-wsdnj        2/2     Running   0          23m
tanzu-system-registry   harbor-trivy-0                          1/1     Running   0          23m
[...]

Obtain the Harbor CA certificate from the harbor-tls secret in the tanzu-system-registry namespace:
```
kubectl -n tanzu-system-registry get secret harbor-tls -o=jsonpath="{.data.ca\.crt}" | base64 -d
```
Record the output for the following step
If the Harbor registry uses a self-signed CA, add it to workload clusters using the applicable procedure based on how the cluster was deployed:
- Supervisor-deployed:
  - vSphere 8: See Integrate a TKG 2 Cluster with a Private Container Registry
  - vSphere 7: See Use an External Container Registry with Tanzu Kubernetes Clusters
- Standalone management cluster-deployed:
  - New clusters: See Configure Clusters with Multiple Trusted Registries
  - Existing clusters: See Add Custom CA Certificate Trust to Existing Clusters

Important
Save a copy of the harbor-data-values.yaml file to use as a back up, and as a reference for settings such as the secret key, password, stoarge size, and so on, when upgrading Harbor in the future.

Connect to the Harbor User Interface

The Harbor UI is exposed via the Envoy service load balancer that is running in the tanzu-system-ingress namespace in the cluster. To allow users to connect to the Harbor UI, you must map the address of the Envoy service load balancer to the hostname of the Harbor service, for example, harbor.yourdomain.com.

Obtain the address of the Envoy service load balancer.
```
kubectl get svc envoy -n tanzu-system-ingress -o jsonpath='{.status.loadBalancer.ingress[0]}'
```
On vSphere without NSX Advanced Load Balancer (ALB), the Envoy service is exposed via NodePort instead of LoadBalancer, so the above output will be empty, and you can use the IP address of any worker node in the cluster instead. On vSphere with NSX ALB, the Envoy service has a Load Balancer IP address similar to 20.54.226.44.
Map the address of the Envoy service load balancer to the hostname of the Harbor service. For clusters that are running on vSphere, you must add an IP to hostname mapping in /etc/hosts or add corresponding A records in your DNS server. For example, if the IP address is 10.93.9.100, add the following to /etc/hosts:
```
10.93.9.100 harbor.yourdomain.com notary.harbor.yourdomain.com
```
On Windows machines, the equivalent to /etc/hosts/ is C:\Windows\System32\Drivers\etc\hosts.

Users can now connect to the Harbor UI by navigating to https://harbor.yourdomain.com in a Web browser and log in as user admin with the harborAdminPassword that you configured in harbor-data-values.yaml.

Push and Pull Images to and from Harbor

Now that Harbor is set up, you can push images to it to make them available for your cluster to pull.

If Harbor uses a self-signed certificate, download the Harbor CA certificate from https://harbor.yourdomain.com/api/v2.0/systeminfo/getcert and install it on your local machine, so Docker can trust this CA certificate.
- On Linux, save the certificate as /etc/docker/certs.d/harbor.yourdomain.com/ca.crt.
- On macOS, follow this procedure.
- On Windows, right-click the certificate file and select Install Certificate.
Log in to the Harbor registry with the user admin. When prompted, enter the harborAdminPassword that you set when you installed the Harbor package in the cluster.
```
docker login harbor.yourdomain.com -u admin
```
Tag an existing image that you have already pulled locally, for example, nginx:1.7.9.
```
docker tag nginx:1.7.9 harbor.yourdomain.com/library/nginx:1.7.9
```

Push the image to the Harbor registry.

docker push harbor.yourdomain.com/library/nginx:1.7.9

Now you can pull the image from the Harbor registry on any machine where the Harbor CA certificate is installed.
```
docker pull harbor.yourdomain.com/library/nginx:1.7.9
```

Update a Running Harbor Deployment

If you need to make changes to the configuration of the Harbor package after deployment, follow these steps to update your deployed Harbor package.

Update the Harbor configuration in harbor-data-values.yaml. For example, you can increase the amount of registry storage by updating the persistence.persistentVolumeClaim.registry.size value.

Update the configuration of the installed package:

tanzu package installed update harbor \
--version INSTALLED-PACKAGE-VERSION \
--values-file harbor-data-values.yaml \
--namespace INSTALLED-PACKAGE-NAMESPACE

Where:

INSTALLED-PACKAGE-VERSION is the version of the installed Harbor package.
INSTALLED-PACKAGE-NAMESPACE is the namespace in which the Harbor package is installed.

For example:

tanzu package installed update harbor \
--version 2.9.1+vmware.1-tkg.1 \
--values-file harbor-data-values.yaml \
--namespace my-packages

The Harbor package will be reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For more information about the tanzu package installed update command, see Update a Package in Install and Manage Packages. You can use this command to update the version and the configuration of an installed package.