Synchronize VMware Tanzu Application Catalog Helm Charts in Air-Gapped environments

This tutorial provides step-by-step instructions on how to synchronize either a Tanzu Application Catalog repository or just specific Helm charts with your private repository and relocate them in an offline target repository using any of the following tools:

An air-gapped environment ensures that all the applications, networks, and resources in the environment are physically isolated from external inputs to prevent potential security risks.

In this context, if you need to operate with artifacts that are not within your network, you have to mirror them to the air-gapped environment, so that your teams are able to consume them under secure conditions.

To consume Tanzu Application Catalog content in an air-gapped enterprise environment, you need an OCI compatible registry, for example Harbor installed in a private Kubernetes cluster and synchronize it with the Tanzu Application Catalog repository. This ensures that enterprise teams always have access to the most updated collection of Helm charts and container images without compromising the security of the air-gapped environment.

There are two methods to synchronize a Tanzu Application Catalog Helm Chart repository with a Harbor private repository in an air-gapped environment. Both tools are open source and therefore easily auditable. They are completely reliable and secure, since they are executed only within the air-gapped environment:

Using the charts-syncer tool to syncronize chart packages and associated container images between chart repositories. It allows to mirror several packages at once.
Using the distribution tooling for Helm dt to create reproducible and relocatable packages for Helm charts and associated containers that can be easily moved across registries without hassles. This is recommended for moving single Helm charts and its dependant images to a target registry.

Note
Some Tanzu Application Catalog Helm charts containers have been verified for use in air-gapped environments. See How can I check if a Helm chart is appropriately configured to meet specific security requirements?

What to do first

In both methods, you must first ensure:

You have access to the Tanzu Application Catalog environment. If you do not have access, contact your VMware sales representative.
You have administrator access to a pre-existing installation of Harbor, configured in line with existing enterprise requirements. If you don’t already have Harbor, you can install it using Bitnami’s Harbor Helm chart. For more information, see Harbor website. Alternatively, having any compliant OCI registry should suffice that can be used to host the relocated Helm Chart and container images.

Using the charts-syncer tool

Charts-syncer is a tool that helps you to relocate the Helm Chart repositories and associated container images from a source location to a target location. It also helps you keep them synchronized.

This section focuses on the scenario in which the source and target Helm chart/container registries are accessible from a common location, from where you can run the charts-syncer.

Understanding Charts-Syncer Tool

As part of the relocation, charts-syncer performs the following process for each Helm Chart that needs to be relocated:

Copies the Helm Chart associated container images to the target container registry.
Updates the Helm Chart container image references to point to the new location of the images. These are the changes performed in the chart code:
- Update values.yaml: This file is updated with the new container registry where the chart should pull the images from.
- Update Chart.yaml: This file will get its images annotation rewritten to point to the new relocated container images.
- Update Images.lock: If present, the Images.lock will be relocated to point to the new container images. If the file does not exist, it will be created.
Pushes the modified Helm Chart to the target Helm Chart repository or OCI registry.

To Synchronize Tanzu Application Catalog Helm Charts in Air-Gapped environments, follow the steps as explained in the sections below:

Install and configure the charts-syncer tool
Run the charts-syncer tool locally.

Install and configure the charts-syncer tool

Install the latest version of the charts-syncer CLI. You can download the latest version from https://github.com/bitnami-labs/charts-syncer/releases .
Once you have downloaded the binary, decompress, and execute it on the same Kubernetes cluster where your internal OCI registry is running. For example, Harbor, Artifactory, Nexus, and so on. See Supported Registries to learn more.
Then, configure the tool to sync both the source and the target repositories. This information is specified via a yaml file similar to the one below.
Configure the tool to sync both the source and the target repositories. This information is specified via a yaml file similar to the one below. To know all the available configuration keys see the charts-syncer file as it includes explanatory comments for each configuration key.

  source:
      repo:
        kind: HELM
        url: https://MY_REGISTRY/
      
  target:
    repo:
      kind: OCI
      url: https://HARBOR_DOMAIN/HARBOR_PROJECT/charts

  # While syncing Tanzu Application Catalog repositories we want to skip the "common" chart since it's a special case
  skipCharts:
      - common

Note https://MY_REGISTRY/ is a placeholder. Replace it with the URL of the private registry where your catalog is located.

You should provide, at least, the following configuration information in the yaml file:

source: Refers to the repository from where the Helm Charts will be fetched. In this example, source is the Tanzu Application Catalog repository.
target.repo: Refers to the repository to which Tanzu Application Catalog pushes the charts. This can be an OCI or a local repository. In this example, the charts are being pushed to an OCI repository backed by your Harbor instance. Note that traditional Helm repositories are supported as well.
HARBOR_DOMAIN and HARBOR_PROJECT: Replace the HARBOR_DOMAIN and HARBOR_PROJECT placeholders with your Harbor installation domain and the name of the project where you want to push the charts.

Note The above config file example assumes that credentials for the Helm chart repositories and container image registries have been provided using config file but they can be also specified in the configuration file by using environment variables. See the Configuration section of the charts syncer README file.

Run the charts-syncer tool locally

After updating the configuration file, enter the following command to run the charts-syncer tool:


$ charts-syncer sync -c config.yaml --latest-version-only

If authentication is required by either the source or the target repositories, a set of credentials should be provided via environment variables.

In the current example, only the target registry needs authentication.

Command invocation


$ TARGET_REPO_AUTH_USERNAME=my_target_repo_username \
  TARGET_REPO_AUTH_PASSWORD=my_target_repo_password \
  TARGET_CONTAINERS_AUTH_USERNAME=my_target_repo_username \
  TARGET_CONTAINERS_AUTH_PASSWORD=my_target_repo_password \
  charts-syncer sync -c config.yaml --latest-version-only -v3

TARGET_REPO_AUTH_USERNAME and TARGET_REPO_AUTH_PASSWORD are the credentials for the target repository to which the Helm Charts will be pushed.
TARGET_CONTAINERS_AUTH_USERNAME and TARGET_CONTAINERS_AUTH_PASSWORD are the credentials for the target container registry to which the associated container images will be pushed.

Note
In the current example, since both the Helm Chart and the container images are being pushed to the same Harbor instance, the credentials are the same. This may differ if you configure the charts-syncer tool to push charts and container images to different locations.

After the sync process is completed, you should be able to explore the relocated Helm Charts and container images through the Harbor UI.

Explore Helm Charts and Containers

As part of the relocation process, the charts-syncer tool updates the Helm Charts values.yaml files to point to the new location of the container images.

Updated YAML File

The relocated Helm Charts are now ready to be consumed by any Helm compliant client such as command line interfaces like Helm CLI proper, user interfaces like Kubeapps, or native Kubernetes controllers like flux or kapp-controller.

Appendix: Advanced use cases

Additionally, the charts-syncer tool allows users to:

Syncing signatures and metadata: charts-syncer syncs not only the images but also their signatures and metadata in the target location. Tanzu Application Catalog provides metadata in OCI format (SBOM, VEX files, CVE test results, and so on). This can be optionally disabled by using the skipArtifacts attribute.
Syncing only specific container platforms.
Syncing Helm charts and container images to different registries for instructions.
Relocating on disconnected target environment.

Using distribution tooling for Helm (dt)

dt is a set of utilities available in a standalone mode and as a Helm Plugin for making offline work with Helm charts easier.

This section explains how to install the plugin for wrapping a Helm chart in the source registry and unwrapping it in the target registry seamlessly. This is particularly useful for distributing Helm charts into air-gapped environments like those used by Federal governments.

To relocate Tanzu Application Catalog Helm Charts to Air-Gapped environments, follow the steps as explained in the sections below:

Install the dt as a Helm plugin
Wrap the Helm chart.
Unwrap the Helm chart.

Install the dt as a Helm plugin

Install the distribution tool as a plugin by executing the command below:

$ helm plugin install https://github.com/vmware-labs/distribution-tooling-for-helm

Other options for installation are available as well:

Now it is time to relocate the Helm chart by using two simple commands: wrap and unwrap.

Wrap the Helm chart

Wrapping a chart consists of packaging the chart into a tar.gz file which includes all container images, the subcharts and their container images that this chart depends on, independently of values.

The resulting file can be distributed through any medium you want (for example, USB stick) to then later be unwrapped into a destination OCI registry.

It is not necessary to download the Helm chart for wrapping it. Just by indicating where the chart is located it will sufficient for the tool to take care of packaging and downloading the assets, whenever they are as shown below. This tutorial uses PostgreSQL Photon OS-based from Tanzu Application Catalog as an example.

Remember to replace the TAC_REGISTRY placeholder with the URL of the registry where you are pulling the content from Tanzu Application Catalog. Learn more about supported registries in Tanzu Application Catalog.


$ helm dt wrap TAC_REGISTRY/charts/photon-4/postgresql:14.3.8
 »  Wrapping Helm chart "TAC_REGISTRY/charts/photon-4/postgresql:14.3.8"
    ✔  Helm chart downloaded to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/postgresql"
    ✔  Images.lock file "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/postgresql/Images.lock" does not exist
    ✔  Images.lock file written to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/postgresql/Images.lock"
    »  Pulling images into "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/postgresql/images"
       ✔  All images pulled successfully
    ✔  Helm chart wrapped to "/Users/demouser/workspace/postgresql/postgresql-14.3.8.wrap.tgz"
 🎉  Helm chart wrapped into "/Users/demouser/workspace/postgresql/postgresql-14.3.8.wrap.tgz"

If your chart and docker images include artifacts such as signatures or metadata, you can also include them in the wrap using the --fetch-artifacts flag.

dt currently supports moving artifacts that follow the conventions listed below:

Cosign or Notation keys that are associated to the digest with a .sig suffix
Metadata entries stored in a sha256-digest.metadata OCI entry


$ helm dt wrap --fetch-artifacts oci://harbor-repo.vmware.com/HARBOR_PROJECT/charts/photon-4/postgresql
 ...
 🎉  Helm chart wrapped into "/Users/demouser/workspace/distribution-tooling-for-helm/postgresql-14.3.8.wrap.tgz"

$ tar -tzf "/Users/demouser/workspace/distribution-tooling-for-helm/postgresql-14.3.8.wrap.tgz" | grep artifacts
postgresql-14.3.8/artifacts/images/postgresql/postgresql/14.3.8-photon-4-r0.sig
postgresql-14.3.8/artifacts/images/postgresql/postgresql/14.3.8-photon-4-r0.metadata
postgresql-14.3.8/artifacts/images/postgresql/postgresql/14.3.8-photon-4-r0.metadata.sig
...

Note Chart signatures are not bundled as they would be invalidated at chart unwrap because of the relocation. All the container images wrapped will maintain their signatures and metadata.

Unwrap the Helm chart

Unwrapping a Helm chart can be done either to a local folder or to a target OCI registry, being the latter the most powerful option. By unwrapping the Helm chart to a target OCI registry the dt tool will unwrap the wrapped file, proceed to push the container images into the target registry that you have specified, relocate the references from the Helm chart to the provided registry and finally push the relocated Helm chart to the registry as well.

At that moment, your Helm chart will be ready to be used from the target registry without any dependencies to the source. By default, the tool will run in safe mode and require you to confirm actions speed everything up by adding the --yes parameter to the command as shown below.

TARGET_REPO is a placeholder. Remember to replace it with the target air-gapped registry where the chart will be relocated.


$ helm dt unwrap postgresql-14.3.8.wrap.tgz TARGET_REPO/helm-plugin/ --yes
 »  Unwrapping Helm chart "postgresql-14.3.8.wrap.tgz"
    ✔  Helm chart uncompressed to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258"
    ✔  Helm chart relocated successfully
    »  The wrap includes the following 3 images:

       TARGET_REPO/helm-plugin/tac/postgresql:16.2.0-photon-4-r18
       TARGET_REPO/helm-plugin/tac/postgres-exporter:0.15.0-photon-4-r26
       TARGET_REPO/helm-plugin/tac/os-shell:4-photon-4-r21

    »  Pushing Images
       ✔  All images pushed successfully
       ✔  Chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" lock is valid

    ⠋ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (0    ⠙ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (0    ⠹ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (0    ⠸ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (1    ⠼ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (1    ⠴ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (1    ⠦ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (1    ⠧ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (1    ⠇ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (2    ⠏ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://TARGET_REPO/helm-plugin/" (2    ✔  Helm chart successfully pushed

 🎉  Helm chart unwrapped successfully: You can use it now by running "helm install oci://TARGET_REPO/helm-plugin/postgresql --generate-name"

Note
If your wrap includes bundled artifacts (if you wrapped it using the --fetch-artifacts flag), they will be also pushed to the remote registry.

Appendix: Advanced use cases

Additionally, the dt tool allows users to:

Synchronize VMware Tanzu Application Catalog Helm Charts in Air-Gapped environments

What to do first

Using the charts-syncer tool

Install and configure the charts-syncer tool

Run the charts-syncer tool locally

Appendix: Advanced use cases

Using distribution tooling for Helm (dt)

Install the dt as a Helm plugin

Wrap the Helm chart

Unwrap the Helm chart

Appendix: Advanced use cases

Useful links