This topic provides hardware requirements for production deployments of VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) on vSphere with NSX-T.
A vSphere cluster is a collection of ESXi hosts and associated virtual machines (VMs) with shared resources and a shared management interface. Installing Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T requires the following vSphere clusters:
For more information on creating vSphere clusters, see Creating Clusters in the vSphere documentation.
The Tanzu Kubernetes Grid Integrated Edition Management Cluster on vSphere comprises the following components:
For more information, see Installing and Configuring NSX-T Data Center v3.0 for TKGI.
A TKGI Edge Cluster on vSphere with NSX-T is comprised of two or more NSX-T Edge Nodes. The maximum supported number of Edge Nodes per TKGI Edge Cluster is 10.
For more information, see Installing and Configuring NSX-T Data Center v3.0 for TKGI.
The Tanzu Kubernetes Grid Integrated Edition Compute Cluster on vSphere comprises the following components:
For more information, see Installing and Configuring NSX-T Data Center v3.0 for TKGI.
The Tanzu Kubernetes Grid Integrated Edition Management Plane comprises the following components:
Depending on your design choice, TKGI management components can be deployed in the Tanzu Kubernetes Grid Integrated Edition Management Cluster on the standard vSphere network or in the Tanzu Kubernetes Grid Integrated Edition Compute Cluster on the NSX-T-defined virtual network. For more information, see NSX-T Deployment Topologies for Tanzu Kubernetes Grid Integrated Edition.
For each vSphere cluster defined for Tanzu Kubernetes Grid Integrated Edition, the following configurations are required to support production workloads:
The vSphere Distributed Resource Scheduler (DRS) is enabled. For more information, see Creating a DRS Cluster in the vSphere documentation.
The DRS custom automation level is set to Partially Automated or Fully Automated. For more information, see Set a Custom Automation Level for a Virtual Machine in the vSphere documentation.
vSphere high-availability (HA) is enabled. For more information, see Creating and Using vSphere HA Clusters in the vSphere documentation.
vSphere HA Admission Control (AC) is configured to support one ESXi host failure. For more information, see Configure Admission Control in the vSphere documentation.
Specifically:The recommended production deployment (RPD) topology represents the VMware-recommended configuration to run production workloads in Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.
Note: The RPD differs depending on whether you are using vSAN or not.
The RPD for Tanzu Kubernetes Grid Integrated Edition with vSAN storage requires 12 ESXi hosts. The diagram below shows the topology for this deployment.
The following subsections describe configuration details for the RPD with vSAN topology.
The RPD with vSAN topology includes a Management/Edge Cluster with the following characteristics:
The RPD with vSAN topology includes three Compute Clusters with the following characteristics:
The RPD with vSAN topology requires the following storage configuration:
For more information on using vSAN with Tanzu Kubernetes Grid Integrated Edition, see PersistentVolume Storage Options on vSphere.
The RPD with vSAN topology can be scaled as follows to accommodate future growth requirements:
The RPD for Tanzu Kubernetes Grid Integrated Edition without vSAN storage requires nine ESXi hosts. The diagram below shows the topology for this deployment.
The following subsections describe configuration details for the RPD of Tanzu Kubernetes Grid Integrated Edition without vSAN.
The RPD without vSAN includes a Management/Edge Cluster with the following characteristics:
The RPD without vSAN topology includes three Compute Clusters with the following characteristic:
The RPD without vSAN topology requires the following storage configuration:
The RPD without vSAN topology can be scaled as follows to accommodate future growth requirements:
The minimum production deployment (MPD) topology represents the baseline requirements for running Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.
Note: The MPD topology for Tanzu Kubernetes Grid Integrated Edition applies to both vSAN and non-vSAN environments.
The diagram below shows the topology for this deployment.
The following subsections describe configuration details for an MPD of Tanzu Kubernetes Grid Integrated Edition.
The MPD topology for Tanzu Kubernetes Grid Integrated Edition requires the following minimum configuration:
Note: For an MPD deployment, each ESXi host must have four physical network interface controllers (PNICs). In addition, while a Tanzu Kubernetes Grid Integrated Edition deployment requires a minimum of three nodes, Tanzu Kubernetes Grid Integrated Edition upgrades require four ESXi hosts to ensure full survivability of the NSX-T Manager appliance.
When configuring vSphere for an MPD topology for Tanzu Kubernetes Grid Integrated Edition, keep in mind the following requirements:
For more information on defining DRS anti-affinity rules, see Virtual Machine Storage DRS Rules in the vSphere documentation.
When planning an MPD topology for Tanzu Kubernetes Grid Integrated Edition, keep in mind the following:
For more information, see Create Management Plane and Create IP Blocks and Pool for Compute Plane in Installing and Configuring NSX-T Data Center v3.0 for TKGI.
The following tables list the VMs and their sizes for deployments of Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.
The following table lists the resource requirements for NSX-T infrastructure and Tanzu Kubernetes Grid Integrated Edition Management Plane VMs.
VM | CPU Cores | Memory (GB) | Ephemeral Disk (GB) |
---|---|---|---|
BOSH Director | 2 | 8 | 103 |
Harbor Registry | 2 | 8 | 167 |
Ops Manager | 1 | 8 | 160 |
TKGI API | 2 | 8 | 64 |
TKGI Database | 2 | 8 | 64 |
NSX-T Manager 1 | 6 | 24 | 200 |
NSX-T Manager 2 | 6 | 24 | 200 |
NSX-T Manager 3 | 6 | 24 | 200 |
vCenter Appliance | 4 | 16 | 290 |
TOTAL | 31 | 128 | 1.38 TB |
Note: The NSX-T Manager resource requirements are based on the medium size VM, which is the minimum recommended form factor for NSX-T v3.0 and later. For more information, see NSX Manager VM and Host Transport Node System Requirements.
If you expect the cluster workload to run a large number of pods continuously, then increase the size of persistent disk storage allocated to the TKGI Database VM as follows:
Number of Pods | Persistent Disk Requirements (GB) |
---|---|
1,000 pods | 20 |
5,000 pods | 100 |
10,000 pods | 200 |
50,000 pods | 1,000 |
The following table lists the resource requirements for each VM in the Edge Cluster.
VM | CPU Cores | Memory (GB) | Ephemeral Disk (GB) |
---|---|---|---|
NSX-T Edge Node 1 | 8 | 32 | 120 |
NSX-T Edge Node 2 | 8 | 32 | 120 |
TOTAL | 16 | 64 | 240 |
Note: NSX-T 3.0 Edge Nodes can be deployed on Intel and AMD-based hosts.
The following table lists sizing information for Kubernetes cluster node VMs. The size and resource consumption of these VMs are configurable in the Plans section of the Tanzu Kubernetes Grid Integrated Edition tile.
VM | CPU Cores | Memory (GB) | Ephemeral Disk (GB) | Persistent Disk |
---|---|---|---|---|
Control&nbsb;Plane Nodes | 1 to 16 | 1 to 64 | 8 to 256 | 1 GB to 32 TB |
Worker Nodes | 1 to 16 | 1 to 64 | 8 to 256 | 1 GB to 32 TB |
For illustrative purposes, consider the following example control plane node and a worker node requirements:
VM | CPU Cores | Memory (GB) | Ephemeral Disk (GB) | Persistent Disk (GB) |
---|---|---|---|---|
Control Plane Node | 2 | 8 | 64 | 128 |
Worker Node | 4 | 16 | 64 | 256 |
The following are the requirements for an example environment consisting of two Kubernetes clusters, each cluster consisting of three of the control plane nodes and five of the worker nodes described above:
VM | CPU Cores | Memory (GB) | Ephemeral Disk | Persistent Disk |
---|---|---|---|---|
Control Plane Nodes (3 nodes) |
6 | 24 | 192 GB | 384 GB |
Worker Nodes (5 nodes) |
20 | 80 | 320 GB | 1280 GB |
TOTAL (2 clusters) |
52 | 208 | 1.0 TB | 3.4 TB |
The following tables list the hardware requirements for RDP and MPD topologies for Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.
The following table lists the hardware requirements for the RPD with vSAN topology.
VM | VM Count | CPU Cores (with HT) | Memory (GB) | NICS | Shared Persistent Disk (TB) |
---|---|---|---|---|---|
Management/ Edge Cluster | 3 | 16 | 98 | 2x 10GbE | 1.5 |
Compute cluster1 (AZ1) | 3 | 6 | 48 | 2x 10GbE | 192 |
Compute cluster2 (AZ2) | 3 | 6 | 48 | 2x 10GbE | 192 |
Compute cluster3 (AZ3) | 3 | 6 | 48 | 2x 10GbE | 192 |
Note: The CPU Cores values assume the use of hyper-threading (HT).
The following table lists the hardware requirements for the RPD without vSAN topology.
VM | VM Count | CPU Cores (with HT) | Memory (GB) | NICS | Shared Persistent Disk (TB) |
---|---|---|---|---|---|
Management/ Edge Cluster | 3 | 16 | 98 | 2x 10GbE | 1.5 |
Compute cluster1 (AZ1) | 2 | 10 | 70 | 2x 10GbE | 192 |
Compute cluster2 (AZ2) | 2 | 10 | 70 | 2x 10GbE | 192 |
Compute cluster3 (AZ3) | 2 | 10 | 70 | 2x 10GbE | 192 |
Note: The CPU Cores values assume the use of hyper-threading (HT).
The following table lists the hardware requirements for the MPD topology with a single (collapsed) cluster for all Management, Edge, and Compute nodes.
VM | VM Count | CPU Cores (with HT) | Memory (GB) | NICS | Shared Persistent Disk (TB) |
---|---|---|---|---|---|
Collapsed Cluster | 3 | 32 | 236 | 4x 10GbE or 2x pNICs* | 5.9 |
*If necessary, you can use two pNICs. For more information, see Deploy a Fully Collapsed vSphere Cluster NSX-T on Hosts Running N-VDS Switches in the NSX-T Data Center documentation.
To add hardware capacity to your Tanzu Kubernetes Grid Integrated Edition environment on vSphere, do the following:
Add one or more ESXi hosts to the vSphere compute cluster. For more information, see the VMware vSphere documentation.
Prepare each newly added ESXi host so that it becomes an ESXi transport node for NSX-T. For more information, see Configure vSphere Networking for ESXi Hosts in Installing and Configuring NSX-T Data Center v3.0 for TKGI.