You can deploy Tanzu Kubernetes Grid management clusters and workload clusters in environments that are not connected to the Internet, such as:
This topic explains how to deploy management clusters to internet-restricted environments on vSphere or AWS. You do not need to follow these procedures if you are using Tanzu Kubernetes Grid in a connected environment that can pull images over an external Internet connection.
NoteThis document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on air-gapped environments. To deploy the management and workload clusters in a specific and validated configuration in air-gapped environments, see VMware Tanzu Reference Architecture Documentation. The TKG v2.1 Reference Design is applicable to both TKG v2.1 and v2.2.
To follow this procedure, you can either use a single machine or different machines as the bootstrap machines in the online and offline environments.
If you want to install the Harbor registry, download the Harbor OVA file:
Before you can deploy management clusters and workload clusters in an Internet-restricted environment, you must have:
TKG_*_PROXY
variables in the cluster configuration file to the proxy server’s address, and set TKG_PROXY_CA_CERT
to the proxy server’s CA if its certificate is self-signed. See Configure Proxies.This document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on vSphere Networking in an Air-Gapped Environment. To deploy the clusters in a specific and validated configuration on vSphere networking in an Air-Gapped environment, see VMware Tanzu Kubernetes Grid on vSphere Networking in an Air-Gapped Environment Reference Design and Deploy Tanzu Kubernetes Grid on vSphere Networking in an Air-Gapped Environment in the VMware Tanzu Kubernetes Grid 2.1 Air-Gapped Reference Design and Deployment document. The TKG v2.1 Reference Design is applicable to both TKG v2.1 and v2.2.
An internet-restricted Tanzu Kubernetes Grid installation on vSphere has firewalls and communication between major components as shown here.
NoteThe following diagram describes the scenario where different bootstrap machines are used in the online and offline environments.
On vSphere, in addition to the general prerequisites above, you must:
Upload to vSphere the OVAs from which node VMs are created. See Import the Base Image Template into vSphere in Deploy Management Clusters to vSphere.
After the VM is created, if you cannot log in with the default username/password, reset the password using Gnu GRUB, as described in Resetting a Lost Root Password if it is Photon OS.
Log in to the jumpbox as root, and enable remote ssh as follows:
PermitRootLogin yes
. In this case the line exist, remove the “#”.service sshd restart
Install and configure a private Docker-compatible container registry such as Harbor, Docker, or Artifactory as follows. This registry runs outside of Tanzu Kubernetes Grid and is separate from any registry deployed as a shared service for clusters:
Configure an offline subnet to use as the internet-restricted environment, and associate it with the jumpbox.
Set up the DHCP server to allocate private IP’s to the new instance.
Create a vSphere distributed switch on a data center to handle the networking configuration of multiple hosts at a time from a central place.
This document provides generic steps to deploy the Tanzu Kubernetes Grid management and workload clusters on AWS in an Air-Gapped Environment. To deploy the clusters in a specific and validated configuration on AWS in an Air-Gapped environment, see VMware Tanzu Kubernetes Grid on AWS Air-Gapped Reference Design and Deploy Tanzu Kubernetes Grid on AWS in an Air-Gapped Environment in the VMware Tanzu Kubernetes Grid 2.1 Air-Gapped Reference Design and Deployment document. The TKG v2.1 Reference Design is applicable to both TKG v2.1 and v2.2.
A proxied Tanzu Kubernetes Grid installation on Amazon Web Services (AWS) has firewalls and communication between major components as shown here. Security Groups (SG) are automatically created between the control plane and workload domains, and between the workload components and control plane components.
For a proxied installation on AWS, in addition to the general prerequisites above, you also need:
After you create the offline VPC, you must add following endpoints to it (VPC endpoint enables private connections between your VPC and supported AWS services):
sts
ssm
ec2
ec2messages
elasticloadbalancing
secretsmanager
ssmmessages
To add the service endpoints to your VPC:
To prepare an internet-restricted environment for deploying Tanzu Kubernetes Grid, you can choose one of the following options:
A single bootstrap machine for the online and offline environments.
Different bootstrap machines for the online and offline environments.
Based on the option that you select, follow the steps described in the following table:
Single Bootstrap Machine | Different Bootstrap Machines | |
---|---|---|
Step 1: Install the Isolated Cluster Plugin on the Online Machine | Yes |
Yes |
Step 2: Download the Images to the Online Machine | Yes |
Yes |
Step 3: Download the Tanzu CLI Image for Linux to the Online Machine | No |
Yes |
Step 4: Copy the Files to the Offline Machine | No |
Yes |
Step 5: Unpack the Files on the Offline Machine | No |
Yes |
Step 6: Install the Isolated Cluster Plugin on the Offline Machine | No |
Yes |
Step 7: Log in to the Private Registry on the Offline Machine | Yes |
Yes |
Step 8: Upload the Images to the Private Registry | Yes |
Yes |
Follow this step only if you have not installed the isolated-cluster
plugin by running the tanzu plugin sync
command on the online machine.
Install the isolated-cluster
plugin:
tanzu plugin sync
ImportantBefore performing this step, ensure that the disk partition where you download the images has 45 GB of available space.
Download the image bundle on your Internet-connected Linux bootstrap machine:
tanzu isolated-cluster download-bundle --source-repo <SOURCE-REGISTRY> --tkg-version <TKG-VERSION> --ca-certificate <SECURITY-CERTIFICATE>
Where:
SOURCE-REGISTRY
is the IP address or the hostname of the registry where the images are stored.TKG-VERSION
is the version of Tanzu Kubernetes Grid that you want to deploy in the proxied or air-gapped environment.SECURITY-CERTIFICATE
is the security certificate of the registry where the images are stored. To bypass the security certificate validation, use --insecure
, instead of --ca-certificate
. Both the strings are optional. If you do not specify any value, the system validates the default server security certificate.
The following is an example:
tanzu isolated-cluster download-bundle --source-repo projects.registry.vmware.com/tkg --tkg-version v2.2.0
FIPS: If you are building a FIPS-enabled image, append --tkg-custom-compatibility-image-path fips/tkg-compatibility
to the tanzu isolated-cluster download-bundle
as described in tanzu isolated-cluster download-bundle in the Tanzu CLI Command Reference.
The image bundle in the form of TAR files, along with the publish-images-fromtar.yaml
file, is downloaded to the online machine. The YAML file defines the mapping between the images and the TAR files.
Download the Tanzu CLI image for Linux (ZIP file) to the online machine:
Go to the Broadcom Support Portal and log in with your VMware customer credentials. 1. Visit the Tanzu Kubernetes Grid downloads page. 1. In the version drop-down, select 2.2.0. 1. Locate VMware Tanzu CLI for Linux and click the download button. The tanzu-cli-bundle-linux-amd64.tar.gz
ZIP file is downloaded to the local machine.
Copy the following files to the offline machine, which is the bootstrap machine in the proxied or air-gapped environment, through a USB thumb drive or other storage medium:
tanzu-cli-bundle-linux-amd64.tar.gz
ZIP fileOn the offline bootstrap machine, unpack the TAR files into the tanzu
directory:
tar -xvf tanzu-cli-bundle-linux-amd64.tar.gz -C $HOME/tanzu
cd $HOME/tanzu/cli
sudo install core/v0.29.0/tanzu-core-linux_amd64 /usr/local/bin/tanzu
tar -xvf tanzu-framework-plugins-standalone-linux-amd64.tar.gz
Install the isolated-cluster
plugin on the offline bootstrap machine:
tanzu plugin install isolated-cluster --local standalone-plugins/
Log in to the private registry on the offline machine through Docker:
docker login <URL>
Where URL
is the URL to the private repository where the images will be stored in the proxied or air-gapped environment.
Upload the images bundle to the offline machine:
tanzu isolated-cluster upload-bundle --source-directory <SOURCE-DIRECTORY> --destination-repo <DESTINATION-REGISTRY> --ca-certificate <SECURITY-CERTIFICATE>
Where:
SOURCE-DIRECTORY
is the path to the location where the image TAR files are stored.DESTINATION-REGISTRY
is the path to the private registry where the images will be hosted in the air-gapped environment.SECURITY-CERTIFICATE
is the security certificate of the private registry where the images will be hosted in the proxied or air-gapped environment. To bypass the security certificate validation, use --insecure
, instead of --ca-certificate
. Both the strings are optional. If you do not specify any value, the system validates the default server security certificate.The following is an example:
tanzu isolated-cluster upload-bundle --source-directory ./ --destination-repo hostname1 --ca-certificate /tmp/registryca.crt
To address the known issue Offline Upgrade Cannot Find Kubernetes v1.25.7 Packages, Fails, follow the instructions in the Knowledge Base article TKG 2.1 upgrade to TKG 2.2 fails when searching for 1.25.7 Tanzu Kubernetes Release on Air Gapped environment to check if two Kubernetes v1.25.7 packages are in your private registry and manually upload them if they are not.
Configure the registry, to where you moved the images, by using the following configuration file reference variables:
TKG_CUSTOM_IMAGE_REPOSITORY
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE
or TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY
.For more information on these configuration variables, see Private Image Registry Configuration.
Note: You can also use the tanzu config set env.CONFIG-VARIABLE
command to configure the registry and persist the environment variables when you use the Tanzu CLI in the future. For more information, see tanzu config set in VMware Tanzu CLI Reference.
Your Internet-restricted environment is now ready for you to deploy or upgrade Tanzu Kubernetes Grid management clusters and start deploying workload clusters on vSphere or AWS.
To deploy FIPS-enabled machine images, follow the procedure described in FIPS-Enabled Versions in VMware Tanzu Compliance.
To deploy the management and workload clusters in a specific and validated configuration in air-gapped environments, see VMware Tanzu Reference Architecture Documentation. The TKG v2.1 Reference Design is applicable to both TKG v2.1 and v2.2.
To deploy the management clusters using a configuration file, see Deploy Management Clusters from a Configuration File.
If you followed this procedure as a part of an upgrade, see Upgrading Tanzu Kubernetes Grid.