Before you can use the Tanzu CLI or installer interface to deploy a management cluster, you must prepare your vSphere environment. You must make sure that vSphere meets the general requirements, and import the base image templates from which Tanzu Kubernetes Grid creates cluster node VMs. Each base image template contains a version of a machine OS and a version of Kubernetes.
kubectlinstalled. See Install the Tanzu CLI and Other Tools.
kubectland other commands.
A network* with:
A set of available static virtual IP addresses for all of the clusters that you create, including both management and Tanzu Kubernetes clusters.
~/.tanzu/tkg/bom/and its name includes the Tanzu Kubernetes Grid version, for example
datecommand to see the timezone settings.
esxcli system time set.
*Or see Deploying Tanzu Kubernetes Grid in an Internet-Restricted Environment for installing without external network access.
On vSphere 7 and later, the vSphere with Tanzu feature includes a Supervisor Cluster that you can configure as a management cluster for Tanzu Kubernetes Grid. This means that on vSphere 7, you do not need to use the
tanzu management-cluster create to deploy a management cluster if vSphere with Tanzu is enabled. Deploying a Tanzu Kubernetes Grid management cluster to vSphere 7 when vSphere with Tanzu is not enabled is supported, but the preferred option is to enable vSphere with Tanzu and use the built-in Supervisor Cluster.
The Tanzu CLI works with both management clusters deployed through vSphere with Tanzu and management clusters deployed by Tanzu Kubernetes Grid on Azure, Amazon EC2, and vSphere 6.7, letting you deploy and manage workload clusters across multiple infrastructures using a single tool. For more information, see Use the Tanzu CLI with a vSphere with Tanzu Supervisor Cluster.
For information about the vSphere with Tanzu feature in vSphere 7, see vSphere with Tanzu Configuration and Management in the vSphere 7 documentation.
NOTE: On VMware Cloud on AWS and Azure VMware Solution, you cannot create a supervisor cluster, and need to deploy a management cluster to run
Each management cluster and Tanzu Kubernetes cluster that you deploy to vSphere requires one static virtual IP address for external requests to the cluster's API server. You must be able to assign this IP address, so it cannot be within your DHCP range, but it must be in the same subnet as the DHCP range.
The cluster control plane's Kube-vip pod uses this static virtual IP address to serve API requests, and the API server certificate includes the address to enable secure TLS communication. In Tanzu Kubernetes clusters, Kube-vip runs in a basic, Layer-2 failover mode, assigning the virtual IP address to one control plane node at a time. In this mode, Kube-vip does not function as a true load balancer for control plane traffic.
Tanzu Kubernetes Grid also does not use Kube-vip as a load balancer for workloads in workload clusters. Kube-vip is used solely by the cluster's API server.
To load-balance workloads on vSphere, use NSX Advanced Load Balancer, also known as Avi Load Balancer, Essentials Edition. You must deploy the NSX Advanced Load Balancer in your vSphere instance before you deploy management clusters. See Install VMware NSX Advanced Load Balancer on a vSphere Distributed Switch.
Before you can deploy a management cluster to vSphere, you must import into vSphere a base image template for the OS and Kubernetes versions that the management cluster nodes run on. VMware publishes base image templates in OVA format for each OS and Kubernetes version supported for management and workload clusters. After importing the OVA, you must convert the resulting VM into a VM template. For information about the versions of Kubernetes that each Tanzu Kubernetes Grid release supports, see the release notes for that release.
Download a Tanzu Kubernetes Grid OVA for the management cluster nodes, which can be one of:
You can also download base image templates for other OS and Kubernetes versions that you expect to create Tanzu Kubernetes clusters from, or you can download them later.
Important: Make sure you download the most recent OVA base image templates in the event of security patch releases. You can find updated base image templates that include security patches on the Tanzu Kubernetes Grid product download page.
In the vSphere Client, right-click an object in the vCenter Server inventory, select Deploy OVF template.
Follow the installer prompts to deploy a VM from the OVA.
NOTE: If you select thick provisioning as the disk format, when Tanzu Kubernetes Grid creates cluster node VMs from the template, the full size of each node's disk will be reserved. This can rapidly consume storage if you deploy many clusters or clusters with many nodes. However, if you select thin provisioning, as you deploy clusters this can give a false impression of the amount of storage that is available. If you select thin provisioning, there might be enough storage available at the time that you deploy clusters, but storage might run out as the clusters run and accumulate data.
When the OVA deployment finishes, right-click the VM and select Template > Convert to Template.
NOTE: Do not power on the VM before you convert it to a template.
In the VMs and Templates view, right-click the new template, select Add Permission, and assign the
tkg-user to the template with the
For information about how to create the user and role for Tanzu Kubernetes Grid, see Required Permissions for the vSphere Account above.
Repeat the procedure for each of the Kubernetes versions for which you downloaded the OVA file.
The vCenter Single Sign On account that you provide to Tanzu Kubernetes Grid when you deploy a management cluster must have the correct permissions in order to perform the required operations in vSphere.
It is not recommended to provide a vSphere administrator account to Tanzu Kubernetes Grid, because this provides Tanzu Kubernetes Grid with far greater permissions than it needs. The best way to assign permissions to Tanzu Kubernetes Grid is to create a role and a user account, and then to grant that user account that role on vSphere objects.
NOTE: If you are deploying Tanzu Kubernetes clusters to vSphere 7 and vSphere with Tanzu is enabled, you must set the Global > Cloud Admin permission in addition to the permissions listed below. If you intend to use Velero to back up and restore management or workload clusters, you must also set the permissions listed in Credentials and Privileges for VMDK Access in the Virtual Disk Development Kit Programming Guide.
In the vSphere Client, go to Administration > Access Control > Roles, and create a new role, for example
TKG, with the following permissions.
|vSphere Object||Required Permission|
Low level file operations
|Global (if using Velero for backup and restore)||Disable methods
|Profile-driven storage||Profile-driven storage view|
|Resource||Assign virtual machine to resource pool|
|Virtual machine|| Change Configuration > Add existing disk
Change Configuration > Add new disk
Change Configuration > Add or remove device
Change Configuration > Advanced configuration
Change Configuration > Change CPU count
Change Configuration > Change Memory
Change Configuration > Change Settings
Change Configuration > Configure Raw device
Change Configuration > Extend virtual disk
Change Configuration > Modify device settings
Change Configuration > Remove disk
Change Configuration > Toggle disk change tracking*
Edit Inventory > Create from existing
Edit Inventory > Remove
Interaction > Power On
Interaction > Power Off
Provisioning > Allow read-only disk access*
Provisioning > Allow virtual machine download*
Provisioning > Deploy template
Snapshot Management > Create snapshot*
Snapshot Management > Remove snapshot*
*Required to enable the Velero plugin, as described in Back Up and Restore Clusters. You can add these permissions when needed later.
In Administration > Single Sign On > Users and Groups, create a new user account in the appropriate domain, for example
In the Hosts and Clusters, VMs and Templates, Storage, and Networking views, right-click the objects that your Tanzu Kubernetes Grid deployment will use, select Add Permission, and assign the
tkg-user with the
TKG role to each object.
Configure the sizes of your management and Tanzu Kubernetes (workload) cluster nodes depending on cluster complexity and expected demand.
For all clusters on vSphere, you configure these with the
worker-size options to
tkg init and
tkg create cluster. Or for greater granularity, you can use the
_MEM_MIB configuration variables.
For management clusters, the installer interface Instance Type field also configures node VM sizes.
For single-worker management and workload clusters running sample applications, use the following minimum VM sizes:
In order for the Tanzu CLI to connect to vSphere from the machine on which you run it, you must provide the public key part of an SSH key pair to Tanzu Kubernetes Grid when you deploy the management cluster. If you do not already have one on the machine on which you run the CLI, you can use a tool such as
ssh-keygen to generate a key pair.
On the machine on which you will run the Tanzu CLI, run the following
ssh-keygen -t rsa -b 4096 -C "email@example.com"
Enter file in which to save the key (/root/.ssh/id_rsa):press Enter to accept the default.
Add the private key to the SSH agent running on your machine, and enter the password you created in the previous step.
.ssh/id_rsa.pubin a text editor so that you can easily copy and paste it when you deploy a management cluster.
If your vSphere environment uses untrusted, self-signed certificates to authenticate connections, you must verify the thumbprint of the vCenter Server when you deploy a management cluster. If your vSphere environment uses trusted certificates that are signed by a known Certificate Authority (CA), you do not need to verify the thumbprint.
You can use either SSH and OpenSSL or the Platform Services Controller to obtain certificate thumbprints.
You can use SSH and OpenSSL to obtain the certificate thumbprint for a vCenter Server Appliance instance.
$ ssh root@vcsa_address
opensslto view the certificate thumbprint.
openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -fingerprint -sha1 -noout
On vSphere 6.7u3, you can obtain a vCenter Server certificate thumbprint by logging into the Platform Services Controller for that vCenter Server instance. If you are deploying a management cluster to vSphere 7, there is no Platform Services Controller.
Log in to the Platform Services Controller interface.
Select Certificate Management and enter a vCenter Single Sign-On password.
For production deployments, it is strongly recommended to enable identity management for your clusters. For information about the preparatory steps to perform before you deploy a management cluster, see Enabling Identity Management in Tanzu Kubernetes Grid.
If you are using Tanzu Kubernetes Grid in an environment with an external internet connection, once you have set up identity management, you are ready to deploy management clusters to vSphere.