Deploy a Workload Cluster to Specialized Hardware

Tanzu Kubernetes Grid supports deploying workload clusters to specific types of GPU-enabled hosts and edge devices on vSphere 7.0+.

Deploy a GPU-Enabled Workload Cluster

To use a node with a GPU in a vSphere workload cluster, you must enable PCI passthrough mode. This allows the cluster to access the GPU directly, bypassing the ESXi hypervisor, which provides a level of performance that is similar to the performance of the GPU on a native system. When using PCI passthrough mode, each GPU device is dedicated to a virtual machine (VM) in the vSphere workload cluster.

Note: To add GPU enabled nodes to existing clusters, use the tanzu cluster node-pool set command.



To create a workload cluster of GPU-enabled hosts, follow these steps to enable PCI passthrough, build a custom machine image, create a cluster configuration file and Tanzu Kubernetes release, deploy the workload cluster, and install a GPU operator using Helm.

  1. Add the ESXi hosts with the GPU cards to your vSphere Client.

  2. Enable PCI passthrough and record the GPU IDs as follows:

    1. In your vSphere Client, select the target ESXi host in the GPU cluster.
    2. Select Configure > Hardware > PCI Devices.
    3. Select the All PCI Devices tab.
    4. Select the target GPU from the list.
    5. Click Toggle Passthrough.
    6. Under General Information, record the Device ID and Vendor ID (highlighted in green in the image below). The IDs are the same for identical GPU cards. You will need these for the cluster configuration file.

    vSphere Client interface showing list of PCI devices. Underneath the list, the location of the device ID and vendor ID is highlighted by a green box.

  3. Create a custom machine image for your cluster that uses Ubuntu 20.04 for the operating system, EFI for the boot option, and vmx-17 for the VM hardware version by following the procedure in Build a Linux Image.

  4. Create a Tanzu Kubernetes release (TKr) for the image by following the steps in Create a TKr for the Linux Image.

  5. Create a workload cluster configuration file using the template in Workload Cluster Template and include the following variables:

    VSPHERE_WORKER_CUSTOM_VMX_KEYS: 'pciPassthru.allowP2P=true,pciPassthru.RelaxACSforP2P=true,pciPassthru.use64bitMMIO=true,pciPassthru.64bitMMIOSizeGB=<GPU-SIZE>'


    • <VENDOR-ID> and <DEVICE-ID> is the Vendor ID and Device ID you recorded in a previous step. For example, if the Vendor ID is 10DE and the Device ID is 1EB8, the value is "0x10DE:0x1EB8". Note: You can only use one type of GPU per VM. For example, you cannot use both the NVIDIA V100 and NVIDIA Tesla T4 on a single VM, but you can use multiple GPUs with the same Vendor ID and Device ID.
    • <GPU-SIZE> is the total GB of framebuffer memory of all GPUs in the cluster rounded to the next power-of-two. For example, if you have two 40GB GPUs, the total is 80GB, then rounded to the next power-of-two is 128GB, so the value is pciPassthru.64bitMMIOSizeGB=128.
    • <BOOLEAN> is false if you are using the NVIDIA Tesla T4 GPU and true if you are using the NVIDIA V100 GPU.
    • WORKER_ROLLOUT_STRATEGY is RollingUpdate if you have extra PCI devices which can be used by the worker nodes during upgrades, otherwise use OnDelete.

    Note: The tanzu CLI does not allow updating the WORKER_ROLLOUT_STRATEGY spec on the MachineDeployment. If the cluster upgrade is stuck due unavailable PCI devices, VMware suggests editing the MachineDeployment strategy using the kubectl CLI. The rollout strategy is defined at spec.strategy.type.

    For a complete list of variables you can configure for GPU-enabled clusters, see GPU-Enabled Clusters in Configuration File Variable Reference.

  6. Create the workload cluster by running:

    tanzu cluster create -f CLUSTER-CONFIG-NAME --tkr TKR-NAME

    Where CLUSTER-CONFIG-NAME and TKR-NAME are the names of the cluster configuration file and TKr file you created in the previous steps.

  7. Add the NVIDIA Helm repository:

    helm repo add nvidia \
    && helm repo update
  8. Install the NVIDIA GPU Operator:

    helm install --kubeconfig=./KUBECONFIG  --wait --generate-name -n gpu-operator --create-namespace nvidia/gpu-operator

    Where KUBECONFIG is the name and location of the kubeconfig for your workload cluster. For more information, see Retrieve Workload Cluster kubeconfig.

    For information about the parameters in this command, see Install the GPU Operator in the NVIDIA documentation.

  9. Ensure the NVIDIA GPU Operator is running:

    kubectl --kubeconfig=./KUBECONFIG  get pods -A

    The output is similar to:

    NAMESPACE         NAME                                                              READY   STATUS     RESTARTS   AGE
    gpu-operator      gpu-feature-discovery-4p6rs                                       0/1     Init:0/1   0          10s
    gpu-operator      gpu-operator-1656477030-node-feature-discovery-master-56457lp8r   1/1     Running    0          50s
    gpu-operator      gpu-operator-1656477030-node-feature-discovery-worker-9g2cm       1/1     Running    0          50s
    gpu-operator      gpu-operator-1656477030-node-feature-discovery-worker-l296w       1/1     Running    0          50s
    gpu-operator      gpu-operator-6688b48999-zssxv                                     1/1     Running    0          50s
    gpu-operator      nvidia-container-toolkit-daemonset-r6nzz                          0/1     Init:0/1   0          10s
    gpu-operator      nvidia-dcgm-exporter-m2vt8                                        0/1     Init:0/1   0          10s
    gpu-operator      nvidia-device-plugin-daemonset-tp6qx                              0/1     Init:0/1   0          10s

Testing Your GPU Cluster

To test your GPU-enabled cluster, create a pod manifest for the cuda-vector-add example from the Kubernetes documentation and deploy it. The container will download, run, and perform a CUDA calculation with the GPU.

  1. Create a file named cuda-vector-add.yaml and add the following:

    apiVersion: v1
    kind: Pod
     name: cuda-vector-add
     restartPolicy: OnFailure
       - name: cuda-vector-add
         image: ""
    1 # requesting 1 GPU
  2. Apply the file:

    kubectl apply -f cuda-vector-add.yaml
  3. Run:

    kubectl get po cuda-vector-add

    The output is similar to:

    cuda-vector-add   0/1     Completed   0          91s
  4. Run:

    kubectl logs cuda-vector-add

    The output is similar to:

    [Vector addition of 50000 elements]
    Copy input data from the host memory to the CUDA device
    CUDA kernel launch with 196 blocks of 256 threads
    Copy output data from the CUDA device to the host memory
    Test PASSED

Deploy a Workload Cluster to an Edge Device

Tanzu Kubernetes Grid v1.6+ supports deploying workload clusters to VMware SD-WAN Edge devices.

Topology: You can run edge workload clusters in production with a single control plane node and just one or two hosts. However, while this uses less CPU, memory, and network bandwidth, you do not have the same resiliency and recovery characteristics of standard production Tanzu Kubernetes Grid clusters. For more information, see VMware Tanzu Edge Solution Reference Architecture 1.0.

Local Registry: To minimize communication delays and maximize resilience, each edge cluster should have its own local Harbor container registry. For an overview of this architecture, see Container Registry in Architecture Overview. To install a local Harbor registry on the edge device, see the Knowledge Base article Building a Harbor Appliance (OVA) to Bootstrap Thick Edge Clusters and Airgap Environments with Tanzu Kubernetes Grid 1.6 (89416).

Timeouts: In addition, when an edge workload cluster has its management cluster remote in a main datacenter, you may need to adjust certain timeouts to allow the management cluster enough time to connect with the workload cluster machines. To adjust these timeouts, see Extending Timeouts for Edge Clusters to Handle Higher Latency below.

Extending Timeouts for Edge Clusters to Handle Higher Latency

If your management cluster is remotely managing workload clusters running on edge devices or managing more than 20 workload clusters, you can adjust specific timeouts so the Cluster API does not block or prune machines that may be temporarily offline or taking longer than 12 minutes to communicate with their remote management cluster, particularly if your infrastructure is underprovisioned.

There are three settings you can adjust to give your edge devices additional time to communicate with their control plane:

  • MHC_FALSE_STATUS_TIMEOUT: Extend the default 12m to, for example, 40m in the workload cluster configuration file to prevent the MachineHealthCheck controller from recreating the machine if its Ready condition remains False for more than 12 minutes. For more information about machine health checks, see Configure Machine Health Checks for Tanzu Kubernetes Clusters.

  • NODE_STARTUP_TIMEOUT: Extend the default 20m to, for example, 60m in the workload cluster configuration file to prevent the MachineHealthCheck controller from blocking new machines from joining the cluster because they took longer than 20 minutes to start up, which it considers unhealthy.

  • etcd-dial-timeout-duration: Extend the default 10m to, for example, 40s in the capi-kubeadm-control-plane-controller-manager manifest to prevent etcd clients on the management cluster from prematurely failing while scanning the health of etcd on the workload clusters. The management cluster uses its ability to connect with etcd as a yardstick for machine health. For example:

    1. In a terminal, run:

      kubectl edit  capi-kubeadm-control-plane-controller-manager -n capi-system
    2. Change the value for --etcd-dial-timeout-duration:

      - args:
           - --leader-elect
           - --metrics-bind-addr=localhost:8080
           - --feature-gates=ClusterTopology=false
           - --etcd-dial-timeout-duration=40s
           - /manager
check-circle-line exclamation-circle-line close-line
Scroll to top icon