Prepare an Internet-Restricted Environment

You can deploy Tanzu Kubernetes Grid management clusters and workload clusters in environments that are not connected to the Internet, such as:

• Proxied environments
• air-gapped environments, with no physical connection to the Internet

This topic explains how to deploy management clusters to internet-restricted environments on vSphere or AWS. You do not need to follow these procedures if you are using Tanzu Kubernetes Grid in a connected environment that can pull images over an external Internet connection.

Note

This document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on air-gapped environments. To deploy the management and workload clusters in a specific and validated configuration in air-gapped environments, see VMware Tanzu Kubernetes Grid 2.3 Air-Gapped Reference Design and Deployment. The TKG v2.3 Reference Design is applicable to both TKG v2.3 and v2.4.

To follow this procedure, you can either use a single machine or different machines as the bootstrap machines in the online and offline environments.

(Optional) Download the Harbor OVA

If you want to install the Harbor registry, download the Harbor OVA file:

1. Go to the Broadcom Support Portal and log in with your VMware customer credentials.
2. Visit the Tanzu Kubernetes Grid downloads page.
3. In the version drop-down, select 2.3.1.
4. Scroll to the Harbor OVA, and click the download button.
5. (Optional) Verify that your downloaded files are unaltered from the original. VMware provides a SHA-1, a SHA-256, and an MD5 checksum for each download. To obtain these checksums, click Read More under the entry that you want to download. For more information, see Using Cryptographic Hashes.

General Prerequisites

Before you can deploy management clusters and workload clusters in an Internet-restricted environment, you must have:

• An Internet-connected Linux bootstrap machine that:
  • Is not inside the internet-restricted environment or can access the domains listed in Proxy Server Allowlist.
  • Has minimum 2 GB RAM, 2 vCPU and 45 GB hard disk space.
  • Has the Tanzu CLI installed. See Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters to download, unpack, and install the Tanzu CLI binary on your Internet-connected system.
• A way for cluster VMs to access images in the private registry:
  • Proxied environments: An egress proxy server that lets cluster VMs access the registry.
    • When you deploy a management cluster in this proxied environment, set TKG_*_PROXY variables in the cluster configuration file to the proxy server’s address, and set TKG_PROXY_CA_CERT to the proxy server’s CA if its certificate is self-signed. See Configure Proxies.
  • Air-gapped environments: A USB thumb drive or other medium for bringing the private registry behind an air-gap, after the registry is populated with images.
• A USB thumb drive or other portable offline storage device.

vSphere Prerequisites and Architecture

This document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on vSphere Networking in an Air-Gapped Environment. To deploy the clusters in a specific and validated configuration on vSphere networking in an Air-Gapped environment, see VMware Tanzu Kubernetes Grid on vSphere Air-Gapped Reference Design and Deploy Tanzu Kubernetes Grid on vSphere Networking in an Air-Gapped Environment in the VMware Tanzu Kubernetes Grid 2.3 Air-Gapped Reference Design and Deployment document. The TKG v2.3 Reference Design is applicable to both TKG v2.3 and v2.4.

vSphere Architecture

An internet-restricted Tanzu Kubernetes Grid installation on vSphere has firewalls and communication between major components as shown here.

Note

The following diagram describes the scenario where different bootstrap machines are used in the online and offline environments.

On vSphere, in addition to the general prerequisites above, you must:

• Upload to vSphere the OVAs from which node VMs are created. See Import the Base Image Template into vSphere in Deploy Management Clusters to vSphere.

  After the VM is created, if you cannot log in with the default username/password, reset the password using Gnu GRUB, as described in Resetting a Lost Root Password if it is Photon OS.

• Log in to the jumpbox as root, and enable remote ssh as follows:

  1. Open the file /etc/ssh/sshd_config in an editor. nano /etc/ssh/sshd_config
  2. Add a line in the Authentication section of the file that says PermitRootLogin yes. In this case the line exist, remove the “#”.
  3. Save the updated /etc/ssh/sshd_config file.
  4. Restart the SSH server using service sshd restart

• Install and configure a private Docker-compatible container registry such as Harbor, Docker, or Artifactory as follows. This registry runs outside of Tanzu Kubernetes Grid and is separate from any registry deployed as a shared service for clusters:

  • Install the registry within your firewall.
  • You can configure the container registry with SSL certificates signed by a trusted CA, or with self-signed certificates.
  • The registry must not implement user authentication. For example, if you use a Harbor registry, the project must be public, and not private.
  • To install Harbor on vSphere:
    • Download the Harbor OVA.
    • Deploy an Offline Harbor Registry on vSphere.

• Configure an offline subnet to use as the internet-restricted environment, and associate it with the jumpbox.

• Set up the DHCP server to allocate private IP’s to the new instance.

• Create a vSphere distributed switch on a data center to handle the networking configuration of multiple hosts at a time from a central place.

  • For more information, see Create a vSphere Distributed Switch.

AWS Prerequisites and Architecture

This document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on AWS in an Air-Gapped Environment. To deploy the clusters in a specific and validated configuration on AWS in an Air-Gapped environment, see VMware Tanzu Kubernetes Grid on AWS Air-Gapped Reference Design and Deploy Tanzu Kubernetes Grid on AWS in an Air-Gapped Environment in the VMware Tanzu Kubernetes Grid 2.3 Air-Gapped Reference Design and Deployment document. The TKG v2.3 Reference Design is applicable to both TKG v2.3 and v2.4.

AWS Architecture

A proxied Tanzu Kubernetes Grid installation on Amazon Web Services (AWS) has firewalls and communication between major components as shown here. Security Groups (SG) are automatically created between the control plane and workload domains, and between the workload components and control plane components.

For a proxied installation on AWS, in addition to the general prerequisites above, you also need:

• An AWS VPC with no internet gateway (“offline VPC”) configured as described below.
  • Your internet-connected bootstrap machine must be able to access IP addresses within this offline VPC. For more information, see VPC Peering.
• A private Docker-compatible container registry such as Harbor, Docker, or Artifactory installed and configured as follows. This registry runs outside of Tanzu Kubernetes Grid and is separate from any registry deployed as a shared service for clusters:
  • Install the registry within your firewall.
  • You can configure the container registry with SSL certificates signed by a trusted CA, or with self-signed certificates.
  • The registry must not implement user authentication. For example, if you use a Harbor registry, the project must be public, and not private.
  • To install Harbor:
    1. Download the Harbor OVA.
    2. Follow the Harbor Installation and Configuration instructions in the Harbor documentation.
• A Linux bootstrap VM running within your offline VPC, provisioned similarly to the internet-connected machine above.
  • The offline bootstrap VM must be able to reach cluster VMs created by Tanzu Kubernetes Grid directly, without a proxy.

After you create the offline VPC, you must add following endpoints to it (VPC endpoint enables private connections between your VPC and supported AWS services):

• Service endpoints:
  • sts
  • ssm
  • ec2
  • ec2messages
  • elasticloadbalancing
  • secretsmanager
  • ssmmessages

To add the service endpoints to your VPC:

1. In the AWS console, browse to VPC Dashboard > Endpoints.
2. For each of the above services
   1. Click Create Endpoint.
   2. Search for the service and select it under Service Name.
   3. Select your VPC and its Subnets.
   4. Enable DNS Name for the endpoint.
   5. Select a Security group that allows VMs in the VPC to access the endpoint.
   6. Select Policy > Full Access.
   7. Click Create endpoint.

Preparing an Internet-Restricted Environment

To prepare an internet-restricted environment for deploying Tanzu Kubernetes Grid, you can choose one of the following options:

• A single bootstrap machine for the online and offline environments.

• Different bootstrap machines for the online and offline environments.

Based on the option that you select, follow the steps described in the following table:

	Single Bootstrap Machine	Different Bootstrap Machines
Step 1: Install the Isolated Cluster Plugins on the Online Machine	Yes	Yes
Step 2: Download the TKG Images to the Online Machine	Yes	Yes
Step 3: Download the Tanzu CLI Binary Image for Linux to the Online Machine	No	Yes
Step 4: Download the CLI Plugins Bundle to the Online Machine	No	Yes
Step 5: Copy the Files to the Offline Machine	No	Yes
Step 6: Unpack the Files on the Offline Machine	No	Yes
Step 7: Log in to the Private Registry on the Offline Machine	Yes	Yes
Step 8: Install Tanzu CLI on the Offline Machine	No	Yes
Step 9: Install the CLI Plugins Bundle on the Offline Machine	Yes	Yes
Step 10: Upload the TKG Images to the Private Registry	Yes	Yes

Step 1: Install the Isolated Cluster Plugin on the Online Machine

To install the isolated-cluster plugin:

1. Install the isolated-cluster plugin by following the procedure described for installing a single plugin in Install Tanzu CLI Plugins in VMware Tanzu CLI Documentation.

2. Make sure the version number of the isolated-cluster plugin is compatible with the version number of the core Tanzu CLI:

   1. Run tanzu plugin list to see the version number of the Tanzu CLI Plugins.

   2. Refer to TKG, Tanzu CLI Plugin, and Tanzu Standard Package Repo Versions in About Tanzu Kubernetes Grid to confirm that your isolated-cluster plugin version is compatible with TKG v2.3.

      • The core Tanzu CLI version as listed by running tanzu version should match the Tanzu CLI version for TKG v2.3 as listed in Product Snapshot in the TKG release notes.

   3. If your isolated-cluster plugin version does not match your version of TKG version, upgrade TKG as described under Upgrading Tanzu Kubernetes Grid.

Step 2: Download the TKG Images to the Online Machine

Important

Before performing this step, ensure that the disk partition where you download the images has 45 GB of available space.

Download the image bundle on your Internet-connected Linux bootstrap machine:

tanzu isolated-cluster download-bundle --source-repo <SOURCE-REGISTRY> --tkg-version <TKG-VERSION> --ca-certificate <SECURITY-CERTIFICATE>

Where:

• SOURCE-REGISTRY is the IP address or the hostname of the registry where the images are stored.
• TKG-VERSION is the version of Tanzu Kubernetes Grid that you want to deploy in the proxied or air-gapped environment.

• SECURITY-CERTIFICATE is the security certificate of the registry where the images are stored. To bypass the security certificate validation, use --insecure, instead of --ca-certificate. Both the strings are optional. If you do not specify any value, the system validates the default server security certificate.

  The following is an example:

  tanzu isolated-cluster download-bundle --source-repo projects.registry.vmware.com/tkg --tkg-version v2.3.1
  
  

FIPS: If you are building a FIPS-enabled image, append --tkg-custom-compatibility-image-path fips/tkg-compatibility to the tanzu isolated-cluster download-bundle as described in tanzu isolated-cluster download-bundle in the Tanzu CLI Command Reference.

The image bundle in the form of TAR files, along with the publish-images-fromtar.yaml file, is downloaded to the online machine. The YAML file defines the mapping between the images and the TAR files.

Step 3: Download the Tanzu CLI Binary Image for Linux to the Online Machine

Download the Tanzu CLI image for Linux (ZIP file) to the online machine:

1. Go to the Broadcom Support Portal and log in with your VMware customer credentials.
2. Go to the Tanzu CLI download page, expand Product Downloads, select 1.1.0, and download the Tanzu CLI binary for Linux. The tanzu-cli-linux-amd64.tar.gz ZIP file is downloaded to the local machine.

Step 4: Download the CLI Plugins Bundle to the Online Machine

Download the CLI plugins bundle to the online machine:

In the Tanzu CLI, run the following command to download the CLI plugins bundle:

tanzu plugin download-bundle --group vmware-tkg/default --to-tar /tmp/plugin-bundle.tar.gz

Step 5: Copy the Files to the Offline Machine

Copy the following files to the offline machine, which is the bootstrap machine in the proxied or air-gapped environment, through a USB thumb drive or other storage medium:

• The Image TAR files
• The YAML files
• The tanzu-cli-bundle-linux-amd64.tar.gz ZIP file
• The CLI plugins bundle

Step 6: Unpack the Files on the Offline Machine

On the offline bootstrap machine, unpack the Tanzu CLI image ZIP file into the tanzu directory:

tar -xvf tanzu-cli-bundle-linux-amd64.tar.gz -C $HOME/tanzu

Step 7: Log in to the Private Registry on the Offline Machine

Log in to the private registry on the offline machine through Docker:

docker login <URL>

Where URL is the URL to the private repository where the images will be stored in the proxied or air-gapped environment.

Step 8: Install Tanzu CLI on the Offline Machine

Install Tanzu CLI on the offline machine:

sudo install tanzu-cli-linux_amd64 /usr/local/bin/tanzu

Step 9: Install the CLI Plugins Bundle on the Offline Machine

Install the CLI Plugins Bundle on the Offline Machine.

1. If the private registry in the air-gapped environment is configured with a self-signed CA certificate, add your certificate verification preferences by following the instructions in Adding Certificate Configuration for the Custom Registry.

2. Upload the CLI plugins bundle to the private repository in the air-gapped environment:

   tanzu plugin upload-bundle --tar /tmp/plugin-bundle.tar.gz --to-repo PRIVATE-REPO/tanzu_cli/plugins
   

   Where PRIVATE-REPO is the private repository where the images will be stored in the proxied or air-gapped environment. For example, registry.example.com.

3. Update the Tanzu CLI to point to the new plugin source:

   tanzu plugin source update default --uri PRIVATE-REPO/tanzu_cli/plugins/plugin-inventory:latest
   

4. Verify that the plugins are discoverable:

   tanzu plugin search
   

   tanzu plugin group search 
   

5. Install the CLI plugins:

   tanzu plugin install 
   

Step 10: Upload the TKG Images to the Private Registry

Upload the TKG images bundle to the private registry:

tanzu isolated-cluster upload-bundle --source-directory <SOURCE-DIRECTORY> --destination-repo <DESTINATION-REGISTRY> --ca-certificate <SECURITY-CERTIFICATE>

Where:

• SOURCE-DIRECTORY is the path to the location where the image TAR files are stored.
• DESTINATION-REGISTRY is the path to the private registry where the images will be hosted in the air-gapped environment.
• SECURITY-CERTIFICATE is the security certificate of the private registry where the images will be hosted in the proxied or air-gapped environment. To bypass the security certificate validation, use --insecure, instead of --ca-certificate. Both the strings are optional. If you do not specify any value, the system validates the default server security certificate.

The following is an example:

tanzu isolated-cluster upload-bundle --source-directory ./ --destination-repo hostname1 --ca-certificate /tmp/registryca.crt

What to Do Next

Configure the registry, to where you moved the images, by using the following configuration file reference variables:

• TKG_CUSTOM_IMAGE_REPOSITORY
• TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE or TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY.

For more information on these configuration variables, see Private Image Registry Configuration.

Note

You can also use the tanzu config set env.CONFIG-VARIABLE command to configure the registry and persist the environment variables when you use the Tanzu CLI in the future. For more information, see tanzu config set in the Tanzu CLI Command Reference.

Your Internet-restricted environment is now ready for you to deploy or upgrade Tanzu Kubernetes Grid management clusters and start deploying workload clusters on vSphere or AWS.

To deploy FIPS-enabled machine images, follow the procedure described in FIPS-Enabled Versions in VMware Tanzu Compliance.

To deploy the management and workload clusters in a specific and validated configuration in air-gapped environments, see VMware Tanzu Kubernetes Grid 2.3 Air-Gapped Reference Design and Deployment. The TKG v2.3 Reference Design is applicable to both TKG v2.3 and v2.4.

To deploy the management clusters using a configuration file, see Deploy Management Clusters from a Configuration File.

If you followed this procedure as a part of an upgrade, see Upgrading Tanzu Kubernetes Grid.