You can deploy Tanzu Kubernetes Grid management clusters and Kubernetes clusters in environments that are not connected to the Internet. These procedures apply to vSphere or AWS environments.

You do not need to perform this procedure if you are using Tanzu Kubernetes Grid in a connected environment that can pull images over an external Internet connection.

General Prerequisites

Before you can deploy management clusters and Tanzu Kubernetes clusters in an Internet-restricted environment, you must have:

  • A private Docker registry such as Harbor, against which this procedure has been tested.

    • Install the Docker registry within your firewall (for vSphere). For information about how to install Harbor, see Harbor Installation and Configuration.
    • If you are using Tanzu Kubernetes Grid v1.2.0, configure the Docker registry with public SSL certificates signed by a trusted CA. Using self-signed certificates is possible, but requires you to do one of the following:

      • Disable TLS verification
      • Manually inject the certificates into every cluster node that you deploy
      • Deploy Harbor as a shared service after you complete the steps in this procedure.

      For information about how to obtain the Harbor registry certificate, see the Harbor documentation. For more information about Harbor and self-signed certificates in an Internet-restricted environment, see Using the Harbor Shared Service in Internet-Restricted Environments below.

    • If you are using Tanzu Kubernetes Grid v1.2.1 or later, you can configure the Docker registry with public SSL certificates signed by a trusted CA, or with self-signed certificates.
  • (Optional) If needed, a USB key or other portable storage medium for bringing the private registry behind an airgap after it is populated with Tanzu Kubernetes Grid images.
  • An Internet-connected Linux bootstrap machine that:
    • Can connect to your private Docker registry; for example, is inside your VPC.
    • Has the Docker client app installed.
    • Has the Tanzu Kubernetes Grid CLI installed. See Install the Tanzu Kubernetes Grid CLI to download, unpack, and install the Tanzu Kubernetes Grid CLI binary on your Internet-connected system.
    • Has yq v3 and jq installed.
    • If you intend to install one or more of the Tanzu Kubernetes Grid extensions, for example Harbor, the Carvel tools are installed. For more information, see Install the Carvel Tools on the Bootstrap Environment.

vSphere Prerequisites and Architecture

On vSphere, in addition to the general prerequisites above, you must:

Name Source Destination Service Purpose
workload-to-mgmt tkg-workload-cluster-network tkg-management-cluster-network TCP:6443 Allow workload cluster to register with management cluster
mgmt-to-workload tkg-management-cluster-network tkg-workload-cluster-network TCP:6443, 5556 Allow management network to configure workload cluster
allow-mgmt-subet tkg-management-cluster-network tkg-management-cluster-network all Allow all internal cluster communication
allow-wl-subnet tkg-workload-cluster-network tkg-workload-cluster-network all Allow all internal cluster communication
jumpbox-to-k8s Jumpbox IP tkg-management-cluster-network, tkg-workload-cluster-network TCP:6443 Allow Jumpbox to create management cluster and manage clusters.
dhcp any NSX-T: any / no NSX-T: DHCP IP DHCP Allows hosts to get DHCP addresses.
to-harbor tkg-management-cluster-network, tkg-workload-cluster-network, jumpbox IP Harbor IP HTTPS Allows components to retrieve container images
to-vcenter tkg-management-cluster-network, tkg-workload-cluster-network, jumpbox IP vCenter IP HTTPS Allows components to access vSphere to create VMs and Storage Volumes
dns-ntp-outbound tkg-management-cluster-network, tkg-workload-cluster-network, jumpbox IP DNS, NTP servers DNS, NTP Core services
ssh-to-jumpbox any Jumpbox IP SSH Access from outside to the jumpbox
deny-all any any all deny by default

vSphere Architecture

An internet-restricted Tanzu Kubernetes Grid installation on vSphere has firewalls and communication between major components as shown here.

Diagram: Airgapped TKG on vSphere

Amazon EC2 Prerequisites and Architecture

For an Internet-restricted installation on Amazon EC2, in addition to the general prerequisites above, you also need:

  • An Amazon EC2 VPC with no internet gateway ("offline VPC") configured as described below.
    • Your internet-connected bootstrap machine must be able to access IP addresses within this offline VPC.
  • An Amazon S3 bucket.
  • A Docker registry installed within your offline VPC, configured equivalently to the internet-connected registry above.
  • A Linux bootstrap VM within your offline VPC, provisioned similarly to the internet-connected machine above.
  • Before you deploy the offline management cluster, you must configure its load balancer with an internal scheme as described in Step 5: Initialize Tanzu Kubernetes Grid, below.

Offline VPC Configuration

By default, deploying Tanzu Kubernetes Grid to Amazon EC2 creates a new, internet-connected VPC. To deploy to an Internet-restricted environment, you must first create an offline VPC. For details, see the Reuse a VPC and NAT gateway(s) that already exist in your availability zone(s) instructions in the Deploy Management Clusters to Amazon EC2 topic.

After you create the offline VPC, you must add following endpoints to it:

  • Service endpoints:
    • sts
    • ssm
    • ec2
    • ec2messages
    • elasticloadbalancing
    • secretsmanager
    • ssmmessages
  • Gateway endpoint to your Amazon S3 storage bucket

To add the service endpoints to your VPC:

  1. In the AWS console, browse to VPC Dashboard > Endpoints
  2. For each of the above services
    1. Click Create Endpoint
    2. Search for the service and select it under Service Name
    3. Select your VPC and its Subnets
    4. Enable DNS Name for the endpoint
    5. Select a Security group that allows VMs in the VPC to access the endpoint
    6. Select Policy > Full Access
    7. Click Create endpoint

To add the Amazon S3 gateway endpoint to your VPC, follow the instructions in Endpoints for Amazon S3 in the AWS documentation.

  • When an offline VPC has an S3 endpoint to dedicated S3 storage, all traffic between the VPC and S3 travels via internal AWS cloud infrastructure, and never over the open Internet.

Amazon EC2 Architecture

An internet-restricted Tanzu Kubernetes Grid installation on Amazon EC2 has firewalls and communication between major components as shown here. Security Groups (SG) are automatically created between the control plane and workload domains, and between the workload components and control plane components.

Diagram: Airgapped TKG on AWS

Step 1: Prepare Environment

The following procedures apply both for the initial deployment of Tanzu Kubernetes Grid in an internet-restricted environment and to upgrading an existing internet-restricted Tanzu Kubernetes Grid deployment.

  1. On the machine with an Internet connection on which you installed the Tanzu Kubernetes Grid CLI, run the tkg get management-cluster command.

    Running a tkg command for the first time installs the necessary Tanzu Kubernetes Grid configuration files in the ~/.tkg folder on your system. The script that you create and run in subsequent steps requires the Bill of Materials (BoM) YAML files in the ~/.tkg/bom folder to be present on your machine. The scripts in this procedure use the BoM files to identify the correct versions of the different Tanzu Kubernetes Grid component images to pull.

  2. In the ~/.tkg/bom folder, temporarily remove any BoM files that either:

    • Do not correspond to your current and previously-installed versions of Tanzu Kubernetes Grid, or
    • Do not correspond to patch versions of Kubernetes that you plan to run workloads on.

    For example, if you upgraded to v1.2.0 from v1.1.3, and you want to run workloads on Kubernetes v1.19.1 (the default for Tanzu Kubernetes Grid v1.2.0), v1.18.6 (the default Kubernetes for v1.1.3), and Kubernetes v1.18.8, then you should keep the following BoM files in ~/.tkg/bom and move the rest elsewhere:

    • bom-1.2.0+vmware.1.yaml
    • bom-1.1.3+vmware.1.yaml
    • bom-1.18.8+vmware.1.yaml

    Note: BoM files named for Tanzu Kubernetes Grid versions, like bom-1.2.0+vmware.1.yaml, specify the default Kubernetes version for those Tanzu Kubernetes Grid versions and their management clusters. BoM files named for Kubernetes versions, like bom-1.18.8+vmware.1.yaml enable workload clusters to run non-default versions of Kubernetes. In all cases, the file's components.kubernetes.version field specifies the Kubernetes version that it includes.

  3. Set the IP address or FQDN of your local registry as an environment variable.

    In the following command example, replace custom-image-repository.io with the address of your private Docker registry.

    On Windows platforms, use the SET command instead of export. Include the name of the project in the value:

    export TKG_CUSTOM_IMAGE_REPOSITORY="custom-image-repository.io/yourproject"
    
  4. If your private Docker registry uses a self-signed certificate, set one of the following environment variables.

    • If you are using Tanzu Kubernetes Grid v1.2.1 or later, provide the CA certificate in base64 encoded format.

      export TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE=LS0t[...]tLS0tLQ==
      

      If you specify the CA certificate in this option, it is automatically injected into all Tanzu Kubernetes clusters that you create in this Tanzu Kubernetes Grid instance.

    • If you are using Tanzu Kubernetes Grid v1.2.0, you must disable TLS verification:

      export TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY=true
      

    On Windows platforms, use the SET command instead of export.

Step 2: Generate the publish-images Script

  1. Copy and paste the following shell script in a text editor, and save it as gen-publish-images.sh.

    #!/usr/bin/env bash
    # Copyright 2020 The TKG Contributors.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    BOM_DIR=${HOME}/.tkg/bom
    
    if [ -z "$TKG_CUSTOM_IMAGE_REPOSITORY" ]; then
        echo "TKG_CUSTOM_IMAGE_REPOSITORY variable is not defined"
        exit 1
    fi
    
    for TKG_BOM_FILE in "$BOM_DIR"/*.yaml; do
        # Get actual image repository from BoM file
        actualImageRepository=$(yq r "$TKG_BOM_FILE" imageConfig.imageRepository | tr -d '"')
    
        # Iterate through BoM file to create the complete Image name
        # and then pull, retag and push image to custom registry
        yq r --tojson "$TKG_BOM_FILE" images | jq -c '.[]' | while read -r i; do
            # Get imagePath and imageTag
            imagePath=$(jq .imagePath <<<"$i" | tr -d '"')
            imageTag=$(jq .tag <<<"$i" | tr -d '"')
    
            for tag in $(echo "${imageTag}" | tr "," "\n")
            do
                # create complete image names
                actualImage=$actualImageRepository/$imagePath:$tag
                customImage=$TKG_CUSTOM_IMAGE_REPOSITORY/$imagePath:$tag
    
                echo "docker pull $actualImage"
                echo "docker tag  $actualImage $customImage"
                echo "docker push $customImage"
                echo ""
            done
        done
    done
    
  2. Make the gen-publish-images script executable.

    chmod +x gen-publish-images.sh
    
  3. Generate a publish-images shell script that is populated with the address of your private Docker registry.

    ./gen-publish-images.sh > publish-images.sh
    
  4. Verify that the generated script contains the correct registry address.

    cat publish-images.sh
    

Step 3: Run the publish-images Script

  1. Make the publish-images script executable.

    chmod +x publish-images.sh
    
  2. Log in to your local private registry.
    docker login ${TKG_CUSTOM_IMAGE_REPOSITORY}
    
  3. Run the publish-images script to pull the required images from the public Tanzu Kubernetes Grid registry, retag them, and push them to your private registry.

    ./publish-images.sh
    

    If your registry lacks sufficient storage for all images in the publish-images script, re-generate and re-run the script after either:

  4. When the script finishes, do the following, depending on your infrastructure:

    • vSphere: Turn off your Internet connection.
    • Amazon EC2: Use the offline VPC's S3 gateway to transfer the Docker containers from the online registry to the offline registry, and the tkg CLI and dependencies from the online bootstrap machine to the offline bootstrap machine.

    You can do this from the AWS console by uploading and downloading tar archives from the online registry and machine, and to the offline registry and machine.

Step 4: Set Environment Variables

As long as the TKG_CUSTOM_IMAGE_REPOSITORY variable remains set, when you deploy clusters, Tanzu Kubernetes Grid will pull images from your local private registry rather than from the external public registry. To make sure that Tanzu Kubernetes Grid always pulls images from the local private registry, add TKG_CUSTOM_IMAGE_REPOSITORY to the ~/.tkg/config.yaml file. If your Docker registry uses self-signed certificates, also add TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY or TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE to ~/.tkg/config.yaml.

TKG_CUSTOM_IMAGE_REPOSITORY: custom-image-repository.io/yourproject
TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY: true
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: Cg==

If you specify TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE, set TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY to false and provide the CA certificate in base64 encoded format.

TKG_CUSTOM_IMAGE_REPOSITORY: custom-image-repository.io/yourproject
TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY: false
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: LS0t[...]tLS0tLQ==

Step 5: Initialize Tanzu Kubernetes Grid

  1. If your offline bootstrap machine does not have a ~/.tanzu directory, because you have not yet created a Tanzu Kubernetes Grid management cluster with it, run tanzu cluster list to create the directory.

  2. (Amazon EC2) On the offline machine, customize your AWS management cluster's load balancer template to use an internal scheme, avoiding the need for a public-facing load balancer.
    To do this, add the following into the overlay file ~/.tkg/providers/ytt/03_customizations/internal_lb.yaml:

    #@ load("@ytt:overlay", "overlay")
    #@ load("@ytt:data", "data")
    
    #@overlay/match by=overlay.subset({"kind":"AWSCluster"})
    ---
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSCluster
    spec:
    #@overlay/match missing_ok=True
      controlPlaneLoadBalancer:
    #@overlay/match missing_ok=True
         scheme: "INTERNAL"
    
  3. To initialize Tanzu Kubernetes Grid, run any tkg CLI command. For example:

    tkg init --ui
    

What to Do Next

Your Internet-restricted environment is now ready for you to deploy or upgrade Tanzu Kubernetes Grid management clusters and start deploying Tanzu Kubernetes clusters on vSphere or Amazon EC2.

You can also optionally deploy the Tanzu Kubernetes Grid extensions and use the Harbor Shared service instead of your private Docker registry.

Deploying the Tanzu Kubernetes Grid Extensions in an Internet Restricted Environment

If you are using Tanzu Kubernetes Grid in an internet-restricted environment, after you download and unpack the Tanzu Kubernetes Grid extensions bundle, you must edit the extension files so that Tanzu Kubernetes Grid pulls images from your local private Docker registry rather than from the external Internet.

  1. On a machine with an internet connection, Download and Unpack the Tanzu Kubernetes Grid Extensions Bundle.
  2. Copy the unpacked folder of extensions manifests to the machine within your firewall on which you run the tkg CLI.
  3. Use the search and replace utility of your choice to search recursively through the tkg-extensions-v1.2.0+vmware.1 folder and replace registry.tkg.vmware.run with the address of your private Docker registry, for example custom-image-repository.io/yourproject.

After making this change, when you implement the Tanzu Kubernetes Grid extensions, images will be pulled from your local private Docker registry rather than from registry.tkg.vmware.run.

To deploy the extensions, must also install the Carvel tools on the machine on which you run the tkg CLI. For more information, see Install the Carvel Tools on the Bootstrap Environment.

Using the Harbor Shared Service in Internet-Restricted Environments

For Tanzu Kubernetes Grid deployments in Internet-restricted environments, you can set up Harbor as a shared service so that your Tanzu Kubernetes Grid instance uses it instead of an external registry. As described in the procedures above, to deploy Tanzu Kubernetes Grid in an Internet-Restricted Environment, a private container registry must be running in your environment before you can deploy a management cluster. This private registry might or might not be an open-source Harbor instance. This private registry is not a Tanzu Kubernetes Grid shared service, but rather is a central registry that is available to your whole environment. This central registry is part of your own infrastructure and is not supported by VMware.

After you use this central registry to deploy a management cluster in an Internet-restricted environment, you configure Tanzu Kubernetes Grid so that Tanzu Kubernetes clusters pull images from the central registry rather than from the external Internet. If the central registry uses a trusted CA certificate, connections between Tanzu Kubernetes clusters and the registry are secure. However, if your central registry uses self-signed certificates, what to do depends on which version of Tanzu Kubernetes Grid you are using:

  • If you are using Tanzu Kubernetes Grid v1.2.0, you either need to manually inject the CA certificate into every cluster node that you deploy, or you need to enable TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY on the registry. The first option involves deploying clusters from a modified cluster plan, as documented in Plan Configuration Examples: Trust Custom CA Certificates on Cluster Nodes. In the second option, the connection between containerd and the registry is insecure, so it is inappropriate for production environments.
  • If you are using Tanzu Kubernetes Grid v1.2.1 or later, you can disable TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY and specify the TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE option. Setting this option automatically injects your self-signed certificates into your Tanzu Kubernetes clusters.

In either case, if your central registry uses self-signed certificates, after you use it to deploy a management cluster in an Internet-restricted environment, the recommended approach is to deploy the Harbor shared service in your Tanzu Kubernetes Grid instance. You can then configure Tanzu Kubernetes Grid so that Tanzu Kubernetes clusters pull images from the Harbor shared service rather than from the central registry. Tanzu Kubernetes Grid shared services implement the Tanzu connectivity API within your instance. In addition to managing connections between clusters and the shared service, the Tanzu connectivity API also automatically injects the required certificates into all cluster nodes when you deploy them. So, by deploying Harbor as a shared service in your Tanzu Kubernetes Grid instance, connections between clusters and the Harbor registry are secure, without you having to manually inject certificates into every node or to set the TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE option.

For information about how to deploy the Harbor shared service, see Deploy Harbor Registry as a Shared Service.

check-circle-line exclamation-circle-line close-line
Scroll to top icon