Install Tanzu Application Platform through GitOps with HashiCorp Vault

Caution

Tanzu Application Platform (GitOps) is currently in beta and is intended for evaluation and test purposes only. Do not use in a production environment.

This topic tells you how to install Tanzu Application Platform (commonly known as TAP) through GitOps with secrets managed in an external secrets store. To decide which approach to use, see Choosing Secrets OPerationS (SOPS) or External Secrets Operator (ESO).

Tanzu GitOps Reference Implementation (RI) does not support changing the secrets management strategy for a cluster, for example, SOPS to ESO. However, changing between AWS Secrets Manager and HashiCorp Vault is supported. The External Secrets Operator integration in this release of Tanzu GitOps RI is verified to support Kubernetes integration with HashiCorp Vault.

Prerequisites

Before installing Tanzu Application Platform, ensure you have:

Relocate images to a registry

Before installation, you must relocate the Tanzu Application Platform images from tanzu.packages.broadcom.com to your own container image registry.

The supported registries are Harbor, Azure Container Registry, Google Container Registry, and Quay.io. See the following documentation to learn how to set up your container image registry:

To relocate images from tanzu.packages.broadcom.com to your registry:

  1. Retrieve your Broadcom registry API token:

    1. Sign in to the Broadcom Support Portal.

    2. Go to Tanzu Application Platform (TAP) and expand the VMware Tanzu Application Platform dropdown.

    3. Click the Token Download icon next to the Tanzu Application Platform version you want to download.

      Screenshot of the Tanzu Application Platform download page in the Broadcom Support Portal with the Token Download icon highlighted.

    4. Follow the instructions in the dialog box. Save the token as a variable named MY_BROADCOM_SUPPORT_ACCESS_TOKEN. For example:

      export MY_BROADCOM_SUPPORT_ACCESS_TOKEN=API-TOKEN

      Where API-TOKEN is your token from the Broadcom Support Portal.

  2. Set up environment variables for installation use by running:

    export IMGPKG_REGISTRY_HOSTNAME_0=tanzu.packages.broadcom.com
export IMGPKG_REGISTRY_USERNAME_0=MY-BROADCOM-SUPPORT-USERNAME
export IMGPKG_REGISTRY_PASSWORD_0=${MY_BROADCOM_SUPPORT_ACCESS_TOKEN}
export IMGPKG_REGISTRY_HOSTNAME_1=MY-REGISTRY
export IMGPKG_REGISTRY_USERNAME_1=MY-REGISTRY-USER
export IMGPKG_REGISTRY_PASSWORD_1=MY-REGISTRY-PASSWORD
export INSTALL_REGISTRY_USERNAME=MY-REGISTRY-USER
export INSTALL_REGISTRY_PASSWORD=MY-REGISTRY-PASSWORD
export INSTALL_REGISTRY_HOSTNAME=MY-REGISTRY
export TAP_VERSION=VERSION-NUMBER
export INSTALL_REPO=TARGET-REPOSITORY

    Where:

    • MY-BROADCOM-SUPPORT-USERNAME is the user with access to the images in tanzu.packages.broadcom.com.
    • MY-REGISTRY is your own container registry.
    • MY-REGISTRY-USER is the user with write access to MY-REGISTRY.
    • MY-REGISTRY-PASSWORD is the password for MY-REGISTRY-USER.
    • VERSION-NUMBER is your Tanzu Application Platform version. For example, 1.9.1.
    • TARGET-REPOSITORY is your target repository, a folder or repository on MY-REGISTRY that serves as the location for the installation files for Tanzu Application Platform.

    VMware recommends using a JSON key file to authenticate with Google Container Registry. In this case, the value of INSTALL_REGISTRY_USERNAME is _json_key and the value of INSTALL_REGISTRY_PASSWORD is the content of the JSON key file. For more information about how to generate the JSON key file, see Google Container Registry documentation.

  3. Install the Carvel tool imgpkg CLI.

    To query for the available versions of Tanzu Application Platform on tanzu.packages.broadcom.com, run:

    imgpkg tag list -i tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages | sort -V

  4. Relocate the images with the imgpkg CLI by running:

    imgpkg copy -b tanzu.packages.broadcom.com/tanzu-application-platform/tap-packages:${TAP_VERSION} --to-repo ${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages

(Optional) Install Tanzu Application Platform in an air-gapped environment

Complete the following steps if you install Tanzu Application Platform in an air-gapped environment:

  1. Retrieve the Tanzu Build Service version by running:

    kubectl get package -n tap-install | grep buildservice

  2. Relocate the Tanzu Build Service images to your registry by running:

    imgpkg copy -b tanzu.packages.broadcom.com/tanzu-application-platform/full-tbs-deps-package-repo:TBS-VERSION --to-repo ${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/full-tbs-deps-package-repo

    Where TBS-VERSION is the version of Tanzu Build Service you retrieved.

  3. Configure custom certificate authorities for Tanzu Developer Portal.

  4. Host a grype database in the air-gapped environment. For more information, see Use vulnerability scanning in offline and air-gapped environments.

Create a new Git repository

  1. In a hosted Git service, for example, GitHub or GitLab, create a new repository.

    This version of Tanzu GitOps RI supports authenticating to a hosted Git repository by using SSH as well as Basic Authentication.

  2. Initialize a new Git repository:

    mkdir -p $HOME/tap-gitops
cd $HOME/tap-gitops

git init
git remote add origin [email protected]:my-organization/tap-gitops.git

  3. Set up the authentication method.

    SSH
    Create a read-only deploy key for this new repository (recommended) or SSH key for an account with read access to this repository. The private portion of this key is referred to as GIT_SSH_PRIVATE_KEY.
    Basic Authentication
    Have a username with read access to the Git repository and password or personal access token for the same user.
    Important

    Only use one of ssh or Basic Authentication, not both.

Download and unpack Tanzu GitOps Reference Implementation (RI)

  1. Download Tanzu GitOps Reference Implementation for Tanzu Application Platform v1.9.1 from the Broadcom Support Portal.

  2. Unpack the downloaded TGZ file into the $HOME/tap-gitops directory by running:

    tar -xvf tanzu-gitops-ri-*.tgz -C $HOME/tap-gitops

  3. Commit the initial state:

    cd $HOME/tap-gitops

git add . && git commit -m "Initialize Tanzu GitOps RI"
git push -u origin

Create cluster configuration

  1. Seed configuration for a cluster using Vault through the provided convenience script:

    cd $HOME/tap-gitops

./setup-repo.sh CLUSTER-NAME vault

    Where:

    • CLUSTER-NAME is the name for your cluster. Typically, this is the same as your EKS cluster’s name, the name of the cluster as it appears in eksctl get clusters.
    • vault selects the Vault external Secret Store.

    For example, if the name of your cluster is iterate-green:

    cd $HOME/tap-gitops

./setup-repo.sh iterate-green vault

    This script creates the directory clusters/iterate-green/ and copies in the configuration required to sync this Git repository with the cluster and installing Tanzu Application Platform.

  2. Commit and push:

    git add . && git commit -m 'Add "iterate-green" cluster'
git push

    Saving the base configuration in an initial commit makes it easier to review customizations in the future.

Customize cluster configuration

Configuring the Tanzu Application Platform installation involves setting up two components:

  • An installation of Tanzu Application Platform.
  • An instance of Tanzu Sync, the component that implements the GitOps workflow, fetching configuration from Git and applying it to the cluster.

Follow these steps to customize your Tanzu Application Platform cluster configuration:

  1. Navigate to the created directory:

    cd clusters/CLUSTER-NAME

    For example, if the name of your cluster is iterate-green:

    cd clusters/iterate-green

  2. Define the following environment variables:

    export VAULT_ADDR=MY-VAULT-ADDR
export CLUSTER_NAME=CLUSTER-NAME
export TAP_PKGR_REPO=TAP-PACKAGE-OCI-REPOSITORY

    Where:

    • MY-VAULT-ADDR is the accessible URL of your vault instance (Vault must be reachable by the Kubernetes cluster).
    • CLUSTER-NAME is the name of the target cluster.
    • TAP-PACKAGE-OCI-REPOSITORY is the fully-qualified path to the OCI repository hosting the Tanzu Application Platform images. If they are relocated to a different registry as described in Relocate images to a registry, the value is ${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages.

Connect Vault to a Kubernetes cluster

Tanzu GitOps RI uses the Vault Kubernetes authentication method for establishing trust between the Kubernetes cluster and Vault, see Vault Kubernetes auth for more. This authentication method uses the Kubernetes Control Plane TokenReview API to authenticate the Kubernetes service accounts with Vault. For this reason, the clusters control plane must be able to communicate over the network to the Vault instance.

To configure Kubernetes authentication for Vault, you can create a new Kubernetes authentication engine instance on Vault and two IAM Roles by using the supplied script:

tanzu-sync/scripts/setup/create-kubernetes-auth.sh
Important

  • If you use an Enterprise Vault Server with namespaces, run export VAULT_NAMESPACE=MY-VAULT-NAMESPACE before using the script.
  • If you use token to access server, run export VAULT_TOKEN=MY-VAULT-TOKEN before using the script.

This creates a new vault Kubernetes authentication instance using the information for the current context in your KUBECONFIG.

Example:

vault write auth/iterate-green/config \
  kubernetes_host="MY-KUBERNETES-API-URL" \
  kubernetes_ca_cert="MY-KUBERNETES-CA-CERT" \
  ttl=1h

Grant read access to secret data

Vault secrets store all sensitive configurations, which are accessed by both Tanzu Sync and the Tanzu Application Platform installation.

Follow these steps to configure Roles in Vault:

  1. Create two Policies, one to read the Tanzu Sync secrets and another to read the Tanzu Application Platform installation secrets, by using the supplied script:

    tanzu-sync/scripts/setup/create-policies.sh

  2. Create two Roles, one to read the Tanzu Sync secrets and another to read the Tanzu Application Platform installation secrets, by using the supplied script:

    tanzu-sync/scripts/setup/create-roles.sh

Generate default configuration

You can use the following script to generate default configuration for both Tanzu Sync and Tanzu Application Platform installation:

tanzu-sync/scripts/configure.sh

The following sections tell you how to edit the configuration values to suit your specific needs.

Review and store Tanzu Sync config

Configuration for Tanzu is stored in two locations:

  • Sensitive configuration is stored in Vault.
  • Non-sensitive configuration are stored in YAML files in the Git repository.

Follow these steps to create the sensitive configuration and review the non-sensitive configuration:

Important

You must enable a key-value secret engine named secret.

  1. Save the credentials that Tanzu Sync uses to authenticate with the Git repository. There are two supported authentication methods:

    SSH
    Create a secret named secret/dev/CLUSTER-NAME/tanzu-sync/sync-git/ssh containing the following information as plaintext: 
    {
  "privatekey": "... (private key portion here) ...",
  "knownhosts": "... (known_hosts for git host here) ..."
}

    Where CLUSTER-NAME is the name of your cluster.

    For example, if the Git repository is hosted on GitHub, and the private key created in Create a new Git repository is stored in the file ~/.ssh/id_ed25519:

    export GIT_SSH_PRIVATE_KEY_FILE=~/.ssh/id_ed25519
export GIT_KNOWN_HOSTS=$(ssh-keyscan github.com)
printf '%s\n'  "$(cat <<EOF
{
    "privatekey": "$(cat $GIT_SSH_PRIVATE_KEY_FILE | awk '{printf "%s\\n", $0}')",
    "knownhosts": "$(echo $GIT_KNOWN_HOSTS | awk '{printf "%s\\n", $0}')"
}
EOF
)" | vault kv put secret/dev/${CLUSTER_NAME}/tanzu-sync/sync-git/ssh -

    Where:

    • The content of ~/.ssh/id_ed25519 is the private portion of the SSH key.
    • ssh-keyscan obtains the public keys for the SSH host.
    • awk '{printf "%s\\n", $0}' converts a multiline string into a single-line string with embedded newline chars (\n). JSON does not support multiline strings.
    Basic Authentication
    Create a secret named secret/dev/CLUSTER-NAME/tanzu-sync/sync-git/basic_auth containing the following information as plaintext: 
    {
  "username": "... (username) ...",
  "password": "... (password) ..."
}

    Where:

    • CLUSTER-NAME is the name of your cluster.
    • username is the username of a user account with read access to the Git repository.
    • password is the password or personal access token for the user.

  2. To securely store the authentication credentials required for accessing the OCI registry hosting the Tanzu Application Platform images, create a secret called dev/CLUSTER-NAME/tanzu-sync/install-registry-dockerconfig. This secret contains the following information in plaintext:

    {
  "auths": {
    "MY-REGISTRY": {
      "username": "MY-REGISTRY-USER",
      "password": "MY-REGISTRY-PASSWORD"
    }
  }
}

    Where:

    • CLUSTER-NAME is the name of your cluster.
    • MY-REGISTRY-USER is the user with write access to MY-REGISTRY.
    • MY-REGISTRY-PASSWORD is the password for MY-REGISTRY-USER.
    • MY-REGISTRY is the container registry where the Tanzu Application Platform images are located.

    For example:

    printf '%s\n'  "$(cat <<EOF
{
  "auths": {
      "${INSTALL_REGISTRY_HOSTNAME}": {
          "username": "${INSTALL_REGISTRY_USERNAME}",
          "password": "${INSTALL_REGISTRY_PASSWORD}"
      }
  }
}
EOF
)" | vault kv put secret/dev/${CLUSTER_NAME}/tanzu-sync/install-registry-dockerconfig -

  3. Review the hosted Git URL and branch used by Tanzu Sync.

    This configuration was generated by the configure.sh script. It reported:

    ...
wrote non-sensitive Tanzu Sync configuration to: tanzu-sync/app/values/tanzu-sync.yaml
...

    For example, for the iterate-green cluster, if the Git repository is hosted on GitHub at my-organization/tap-gitops on the main branch, tanzu-sync.yaml contains the following information:

    ---
git:
 url: [email protected]:my-organization/tap-gitops.git
 ref: origin/main
 sub_path: clusters/iterate-green/cluster-config

    You can review and edit these values as needed.

  4. Review the integration with External Secrets Operator.

    This configuration was generated by the configure.sh script. It reported:

    ...
wrote ESO configuration for Tanzu Sync to: tanzu-sync/app/values/tanzu-sync-vault-values.yaml
...

    For example, for the iterate-green cluster, if the Vault URL is https://vault.example.com, tanzu-sync-vault-values.yaml contains the following information:

    ---
secrets:
 eso:
  vault:
    server: https://vault.example.com
    namespace: ""
    auth:
      kubernetes:
        mountPath: iterate-green
        role: iterate-green--tanzu-sync-secrets
  remote_refs:
    sync_git:
    # TO DO: Fill in your configuration for ssh or basic authentication here. See tanzu-sync/app/config/.tanzu-managed/schema--eso.yaml for details.
    install_registry_dockerconfig:
      dockerconfigjson:
        key: secret/dev/iterate-green/tanzu-sync/install-registry-dockerconfig

    Where:

    • kubernetes.role is the IAM role that grants permission to Tanzu Application Platform installation to read its associated secrets. This role was created in the Grant read access to secret data section.
    • install_registry_dockerconfig contains the Vault secret name that contains the Docker config authentication to the OCI registry hosting the Tanzu Application Platform images created earlier.

  5. Replace any TO DO sections from line 12 in the earlier example with the relevant values.

    Configuration example for SSH authentication:

    ---
secrets:
 eso:
  vault:
    server: https://vault.example.com
    namespace: ""
    auth:
      kubernetes:
        mountPath: iterate-green
        role: iterate-green--tanzu-sync-secrets
  remote_refs:
    sync_git:
      ssh:
        private_key:
          key: secret/dev/iterate-green/tanzu-sync/sync-git/ssh
          property: privatekey
        known_hosts:
          key: secret/dev/iterate-green/tanzu-sync/sync-git/ssh
          property: knownhosts
    install_registry_dockerconfig:
      dockerconfigjson:
        key: secret/dev/iterate-green/tanzu-sync/install-registry-dockerconfig

    Configuration example for basic authentication:

    ---
secrets:
 eso:
  vault:
    server: https://vault.example.com
    namespace: ""
    auth:
      kubernetes:
        mountPath: iterate-green
        role: iterate-green--tanzu-sync-secrets
  remote_refs:
    sync_git:
      basic_auth:
        username:
          key: secret/dev/iterate-green/tanzu-sync/sync-git/basic_auth
          property: username
        password:
          key: secret/dev/iterate-green/tanzu-sync/sync-git/basic_auth
          property: password
    install_registry_dockerconfig:
      dockerconfigjson:
        key: secret/dev/iterate-green/tanzu-sync/install-registry-dockerconfig

  6. (Optional) Update Tanzu Application Platform to use the latest patch:

    Update the Tanzu Application Platform version in GIT-REPO-ROOT/clusters/CLUSTER-NAME/cluster-config/values/tap-install-values.yaml:

    tap_install:
    ...
    version:
        package_repo_bundle_tag: "1.9.1" # Populate these values with the latest patch version.
        package_version: "1.9.1"

    Where:

    • package_repo_bundle_tag is the version of Tanzu Application Platform you want to upgrade to.
    • package_version is the version of Tanzu Application Platform you want to upgrade to. This version must match package_repo_bundle_tag.
    Note

    Tanzu Application Platform (GitOps) does not provide a separate artifact for each patch version within a minor line. For example, Tanzu Application Platform v1.6.x only contains the v1.6.1 GitOps artifact.

  7. Commit the Tanzu Sync configuration.

    For example, for the “iterate-green” cluster, run:

    git add tanzu-sync/
git commit -m 'Configure Tanzu Sync on "iterate-green"'

Review and store the Tanzu Application Platform installation config

Configuration for the Tanzu Application Platform installation is stored in two places:

  • Sensitive configuration is stored in Vault.
  • Non-sensitive configuration is stored in YAML files in the Git repository.

Follow these steps to create the sensitive configuration and review the non-sensitive configuration:

  1. Create a secret named secret/dev/${CLUSTER_NAME}/tap/sensitive-values.yaml that stores the sensitive data such as username, password, private key from the tap-values.yaml file:

    printf '%s\n'  "$(cat <<EOF
---
# this document is intentionally initially blank.
EOF
)" | vault kv put secret/dev/${CLUSTER_NAME}/tap/sensitive-values.yaml -

    You can start with an empty document and edit it later on as described in the Configure and push the Tanzu Application Platform values section.

    Vault does not support storing YAML files, all secrets must be in key-value format. You must convert your sensitive-values YAML file to json before storage.

  2. Review the integration with External Secrets Operator.

    This configuration was generated by the configure.sh script. It reported:

    ...
wrote Vault configuration for TAP Install to: cluster-config/values/tap-install-vault-values.yaml
...

    For example, for the iterate-green cluster tap-install-vault-values.yaml contains the following information:

    ---
tap_install:
 secrets:
   eso:
    vault:
      server: https://vault.example.com
      namespace: ""
      auth:
        kubernetes:
          mountPath: iterate-green
          role: iterate-green--tap-install-secrets
     remote_refs:
       tap_sensitive_values:
         sensitive_tap_values_yaml:
           key: secret/dev/iterate-green/tap/sensitive-values.yaml

    Where:

    • kubernetes.role is the IAM role that grants permission to Tanzu Application Platform installation to read its associated secrets. This role was created in the Grant read access to secret data section.
    • sensitive_tap_values_yaml.key is the Vault secret name that contains the sensitive data from the tap-values.yaml file for this cluster in a YAML format.

  3. Commit the Tanzu Application Platform installation configuration.

    For example, for the iterate-green cluster, run:

    git add cluster-config/
git commit -m 'Configure installer for TAP 1.6.1 on "iterate-green"'

Configure and push the Tanzu Application Platform values

The configuration for the Tanzu Application Platform is divided into two separate locations:

Follow these steps to split the Tanzu Application Platform values:

  1. Create the file cluster-config/values/tap-values.yaml by using the Full Profile sample as a guide:

    The Tanzu Application Platform values are input configurations to the Tanzu Application Platform installation and are placed under the tap_install.values path.

    tap_install:
  values:
    # Tanzu Application Platform values go here.
    shared:
      ingress_domain: "INGRESS-DOMAIN"
    ceip_policy_disclosed: true
...

    To install Tanzu Application Platform in an offline environment, you must configure Tanzu Build Service and Grype to work in an air-gapped environment:

    ---
tap_install:
  values:
    ...
    buildservice:
      exclude_dependencies: true
    grype:
      db:
        dbUpdateUrl: INTERNAL-VULN-DB-URL

    Where INTERNAL-VULN-DB-URL is the URL that points to the internal file server.

    For more information, see Components and installation profiles.

  2. Review the contents of tap-values.yaml and move all the sensitive values into the Vault secret created in the Review and store Tanzu Application Platform installation config section.

    For example, if the iterate-green cluster is configured with the basic Out of the Box Supply Chain, this might include a passphrase for that supply chain’s GitOps flow:

    ---
tap_install:
 values:
   ...
   ootb_supply_chain_basic:
     registry:
       server: "SERVER-NAME"
       repository: "REPO-NAME"
     gitops:
       ssh_secret: "SSH-SECRET-KEY"
   ...

    To maintain the secrecy of ootb_supply_chain_basic.gitops.ssh_secret, move this value from the tap-values.yaml file:

    ---
tap_install:
 values:
   ...
   ootb_supply_chain_basic:
     registry:
       server: "SERVER-NAME"
       repository: "REPO-NAME"
   ...

    Add it to the Vault secret named secret/dev/iterate-green/tap/sensitive-values.yaml, by default, without the tap_install.values root:

    ---
...
ootb_supply_chain_basic:
 gitops:
   ssh_secret: "SSH-SECRET-KEY"
...

    When moving values, you must omit the tap_install.values root, but keep the remaining structure. All of the parent keys, such as ootb_supply_chain_basic.gitops and ssh_secret, must be copied to the sensitive value YAML.

  3. Commit and push the Tanzu Application Platform values:

    git add cluster-config/
git commit -m "Configure initial values for TAP 1.6.1"
git push

    Tanzu Sync fetches configuration from the hosted clone of the Git repository. For changes to take effect on the cluster, they must be pushed to that clone of the Git repository.

Deploy Tanzu Sync

Deploying Tanzu Sync starts the GitOps workflow that initiates the Tanzu Application Platform installation.

After deployed, Tanzu Sync periodically polls the Git repository for changes. The following deployment process is only required once per cluster:

  1. Install the Carvel tools kapp and ytt onto your $PATH:

    sudo cp $HOME/tanzu-cluster-essentials/kapp /usr/local/bin/kapp
sudo cp $HOME/tanzu-cluster-essentials/ytt /usr/local/bin/ytt

    This step is required to ensure the successful deployment of the tanzu-sync app.

  2. Ensure the Kubernetes cluster context is set to the correct cluster.

    1. List the existing contexts:

      kubectl config get-contexts

    2. Set the context to the cluster that you want to deploy:

      kubectl config use-context CONTEXT-NAME

      Where CONTEXT-NAME can be retrieved from the outputs of the previous step.

  3. Bootstrap the deployment.

    External Secrets Operator is installed from the package included in the Tanzu Application Platform package repository. That repository must be fetched from the OCI registry initially.

    1. Set the following environment variables:

      export INSTALL_REGISTRY_HOSTNAME=MY-REGISTRY
export INSTALL_REGISTRY_USERNAME=MY-REGISTRY-USER
export INSTALL_REGISTRY_PASSWORD=MY-REGISTRY-PASSWORD

      Where:

      • MY-REGISTRY is your container registry.
      • MY-REGISTRY-USER is the user with read access to MY-REGISTRY.
      • MY-REGISTRY-PASSWORD is the password for MY-REGISTRY-USER.

    2. Create a secret containing credentials to fetch from that OCI registry by using the provided script:

      tanzu-sync/scripts/bootstrap.sh

      These credentials are used exactly once to install the External Secrets Operator (ESO) package.

  4. Install Tanzu Sync and start the GitOps workflow by deploying it to the cluster using kapp and ytt.

    tanzu-sync/scripts/deploy.sh

    Depending on the profile and components included, it may take 5-10 minutes for the Tanzu Application Platform to install. During this time, kapp waits for the deployment of Tanzu Sync to reconcile successfully. This is normal.

    You can track the progress of the installation by watching the installation of those packages in a separate terminal window:

    watch kubectl get pkgi -n tap-install
check-circle-line exclamation-circle-line close-line
Scroll to top icon