By using Kubernetes with VMware Cloud Director, you can provide a multi-tenant Kubernetes service to your tenants.

Container Service Extension

Kubernetes Container Clusters is the VMware Cloud Director Container Service Extension plug-in for VMware Cloud Director. To create Kubernetes clusters, service providers and tenants must use the Kubernetes Container Clusters plug-in. You can download the latest compatible Kubernetes Container Clusters plug-in from the VMware Cloud Director download page for the relevant VMware Cloud Director version, and upload the plug-in to the VMware Cloud Director Service Provider Admin Portal. To enable tenants to create Kubernetes clusters, you must publish the plug-in to the tenant organizations.

Both service providers and tenants must use the Container Service Extension version 3.1 to create native and VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) clusters. You must complete the Container Service Extension 3.1 server setup and publish a Container Service Extension native placement policy to one or more organization VDCs.

vSphere with Tanzu in VMware Cloud Director

You can use vSphere with Tanzu in VMware Cloud Director to create provider virtual data centers (VDCs) backed by Supervisor Clusters. A host cluster with enabled vSphere with Tanzu is called a Supervisor Cluster. You can set restrictions on the uses of the resources and limit the available resources, including number of Kubernetes clusters per organization, user, or group. For more information, see Manage Quotas on the Resource Consumption of an Organization.

To use vSphere with Tanzu in VMware Cloud Director, first, you must enable the vSphere with Tanzu functionality on a vSphere 7.0 or later cluster, and configure that cluster as a Supervisor Cluster. See the vSphere with Kubernetes Configuration and Management guide in the vSphere documentation. The vCenter Server instance that you want to use can have both host clusters and Supervisor Clusters.

To create clusters,Tanzu Kubernetes you must publish a provider VDC Kubernetes policy to an organization and apply the organization VDC Kubernetes policy during the creation. Native and TKGI clusters do not use the provider and organization VDC Kubernetes policies.

Kubernetes Cluster Types

  • Native clusters - The Kubernetes Container Clusters plug-in manages the clusters with native Kubernetes runtime. These clusters are with reduced High Availability function with a single control plane node, they offer fewer persistent volume choices and no networking automation. However, they might come at a lower cost. For native Kubernetes cluster deployment, you must set up a Container Service Extension server. See the CSE Server Management chapter in the Container Service Extension (CSE) documentation.
  • Tanzu Kubernetes Grid, informally known as TKG - Starting with VMware Cloud Director 10.3.1, you can create Tanzu Kubernetes Grid clusters. Tanzu Kubernetes Grid supports VMware hardened and signed upstream compatible Kubernetes, single control plane node, independent disk-based dynamic provisioning of Persistent Volumes, and L4 load balancer automation. For more information on Tanzu Kubernetes Grid clusters, see the VMware Tanzu Kubernetes Grid documentation.
  • VMware Tanzu® Kubernetes Grid™ Service clusters, informally known as TKGS - You can use the vSphere with Tanzu runtime option to create vSphere with Tanzu managed Tanzu Kubernetes Grid Service clusters. Tanzu Kubernetes Grid Service supports VMware hardened and signed upstream compatible Kubernetes, multiple control plane nodes, First Class Disk-based dynamic and static provisioning of Persistent Volumes, and L4 load balancer automation. This option offers more features, however, it might be more expensive. For more information, see the vSphere with Tanzu Configuration and Management guide in the vSphere documentation.
  • TKGI clusters - VMware Tanzu Kubernetes Grid Integrated Edition is a purpose-built container solution to operationalize Kubernetes for multi-cloud enterprises and service providers. Some of its capabilities are high availability, auto-scaling, health-checks, self-healing, and rolling upgrades for Kubernetes clusters. For more information on TKGI clusters, see the VMware Tanzu Kubernetes Grid Integrated Edition documentation.

Workflow for Tanzu Kubernetes Cluster Creation

  1. Add a vCenter Server 7.0 or later instance with an enabled vSphere with Tanzu functionality to VMware Cloud Director. See Attach a vCenter Server Instance Alone or Together with an NSX Manager Instance.
  2. Verify the network settings on each Supervisor Cluster to enable them to run Kubernetes workloads.
    Important: The IP address ranges for the Ingress CIDRs and Services CIDR parameters must not overlap with IP addresses 10.96.0.0/12 and 192.168.0.0/16 which are the default vSphere values for the services and pods parameters. See the configuration parameters for Tanzu Kubernetes clusters information in the vSphere with Kubernetes Configuration and Management guide.
    Note: Starting with VMware Cloud Director 10.2.2, if you modify the network settings of the Supervisor Cluster after the initial setup, you must refresh the vCenter Server instance to adjust the automatic firewall policies and NAT rules that block the access to the Tanzu Kubernetes cluster from outside the organization virtual data center in which the cluster is created.
  3. Create a provider VDC backed by a Supervisor Cluster. See Create a Provider Virtual Data Center.

    Alternatively, you can add a Supervisor Cluster to an existing provider VDC. If you have a vSphere 6.7 or earlier environment, you can also upgrade the environment to version 7.0 and enable vSphere with Tanzu on an existing cluster.

    Provider VDCs backed by a Supervisor Cluster appear with a Kubernetes icon next to their name in the grid that lists all provider VDCs.

  4. (Optional) VMware Cloud Director generates automatically a default provider VDC Kubernetes policy for provider VDCs backed by a Supervisor Cluster. You can create additional provider VDC Kubernetes policies for Tanzu Kubernetes clusters. See Create a Provider VDC Kubernetes Policy.
  5. Publish a Provider VDC Kubernetes Policy to an Organization VDC from the Provider VDCs tab or Add an Organization VDC Kubernetes Policy from the Organization VDCs tab.
  6. Publish the Kubernetes Container Clusters plug-in to service providers. See Publish or Unpublish a Plug-in from an Organization. If you want to enable tenants to create Kubernetes clusters, you must publish the Kubernetes Container Clusters plug-in to those organizations. For more information about managing VMware Cloud Director plug-ins, see Managing Plug-Ins.
  7. If you want to grant tenants the rights to create and manage Tanzu Kubernetes clusters, you must publish the vmware:tkgcluster Entitlement rights bundle to any organizations that you want to work with clusters. After sharing the rights bundle, you must add the Edit: Tanzu Kubernetes Guest Cluster right to the roles you want to create and modify Tanzu Kubernetes clusters. If you want the users also to delete clusters, you must add the Full Control: Tanzu Kubernetes Guest Cluster right to the roles. In addition, you can assign the administrator rights to users that you want to view all Tanzu Kubernetes clusters in an organization or users that you want to manage clusters across sites. For information about the rights and access levels for Runtime Defined Entities (RDEs), see Managing Defined Entities.
  8. Grant access to tenants or system administrators by creating Access Control List (ACL) entries. For more information on sharing Runtime Defined Entities (RDEs), see Sharing Defined Entities.
  9. Create a Tanzu Kubernetes Cluster

Workflow for Native and TKGI Cluster Creation

  1. Publish the Kubernetes Container Clusters plug-in to service providers. See Publish or Unpublish a Plug-in from an Organization. If you want to enable tenants to create Kubernetes clusters, you must publish the Kubernetes Container Clusters plug-in to those organizations. For more information about managing VMware Cloud Director plug-ins, see Managing Plug-Ins.
  2. Set up a Container Service Extension server and publish the Container Service Extension native placement policy or TKGI enablement metadata to the organization VDC. For more information about setting up the CSE server, see the CSE Server Management chapter in the Container Service Extension (CSE) documentation.
  3. If you want to grant tenants the rights to create and manage native clusters, you must publish the cse:nativeCluster Entitlement rights bundle to any organizations that you want to work with native clusters. After sharing the rights bundle, you must add the Edit CSE:NATIVECLUSTER right to the roles you want to create and modify native clusters. If you want the users also to delete clusters, you must add the Full Control CSE:NATIVECLUSTER right to the roles. In addition, you can assign the administrator rights to users that you want to view all Tanzu Kubernetes clusters in an organization or users that you want to manage clusters across sites. For information about the rights and access levels for Runtime Defined Entities (RDEs), see Managing Defined Entities.
  4. If you want to grant tenants the rights to create and manage TKGI clusters, you must publish the {cse}:PKS DEPLOY RIGHT to the specific organizations, and add the {cse}:PKS DEPLOY RIGHT right to the roles you want to create and manage TKGI clusters. The {cse}:PKS DEPLOY RIGHT is created during the Container Service Extension server install.
  5. For native clusters, grant access to tenants or system administrators by creating Access Control List (ACL) entries. For more information on sharing Runtime Defined Entities (RDEs), see Sharing Defined Entities.
  6. Create a Native Kubernetes Cluster or Create a VMware Tanzu Kubernetes Grid Integrated Edition Cluster.

Workflow for Tanzu Kubernetes Grid Cluster Creation

Starting with VMware Cloud Director 10.3.1, you can create Tanzu Kubernetes Grid clusters.

  1. Publish the Kubernetes Container Clusters plug-in to service providers. See Publish or Unpublish a Plug-in from an Organization. If you want to enable tenants to create Kubernetes clusters, you must publish the Kubernetes Container Clusters plug-in to those organizations. For more information about managing VMware Cloud Director plug-ins, see Managing Plug-Ins.
  2. If you want to grant tenants the rights to create and manage native clusters, you must publish the cse:nativeCluster Entitlement rights bundle to any organizations that you want to work with Tanzu Kubernetes Grid clusters. After sharing the rights bundle, you must add the Edit CSE:NATIVECLUSTER right to the roles you want to create and modify Tanzu Kubernetes Grid clusters. If you want the users also to delete clusters, you must add the Full Control CSE:NATIVECLUSTER right to the roles. In addition, you can assign the administrator rights to users that you want to view all Tanzu Kubernetes clusters in an organization or users that you want to manage clusters across sites. For information about the rights and access levels for Runtime Defined Entities (RDEs), see Managing Defined Entities.
  3. Create a VMware Tanzu Kubernetes Grid Cluster.