The Cluster v1beta1 API lets you provision a Cluster based on a default ClusterClass definition.

ClusterClass API v1beta1

The Kubernetes Cluster API is a suite of tools which provide for the declarative provisioning, upgrading, and operating of Kubernetes clusters. ClusterClass is an evolution of the Cluster API that lets you define templates for managing the life cycle of sets of clusters. TKG 2.0 on Supervisor supports ClusterClass using the v1beta1 API.

TKG 2.0 on Supervisor ships with a default ClusterClass definition named tanzukubernetescluster. The tanzukubernetescluster ClusterClass provides the template for TKG 2.0 cluster creation on Supervisor using the v1beta API. The tanzukubernetescluster ClusterClass is available in all user namespaces. To create a cluster based on this ClusterClass, reference it in the Cluster specification. Refer to the v1beta examples for guidance.

Default ClusterClass tanzukubernetescluster

The default tanzukubernetescluster ClusterClass is immutable. It is provided here as reference.
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: tanzukubernetescluster
  annotations:
    run.tanzu.vmware.com/resolve-tkr: ""
spec:
  controlPlane:
    metadata:
      annotations:
        run.tanzu.vmware.com/resolve-os-image: os-name=photon
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
      kind: KubeadmControlPlaneTemplate
      name: "tkc-control-plane"
    machineInfrastructure:
      ref:
        kind: VSphereMachineTemplate
        apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
        name: "tkc-control-plane"
    machineHealthCheck:
      maxUnhealthy: 100%
      nodeStartupTimeout: 2h0m0s
      unhealthyConditions:
        - status: Unknown
          timeout: 5m0s
          type: Ready
        - status: "False"
          timeout: 12m0s
          type: Ready
  infrastructure:
    ref:
      apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereClusterTemplate
      name: "tkc-infrastructure"
  workers:
    machineDeployments:
      - class: node-pool  
        template:
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-name=photon            
          bootstrap:
            ref:
              kind: KubeadmConfigTemplate
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              name: "tkc-md"
          infrastructure:
            ref:
              kind: VSphereMachineTemplate
              apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
              name: "tkc-md"
        machineHealthCheck:
          maxUnhealthy: 100%
          nodeStartupTimeout: 2h0m0s
          unhealthyConditions:
            - status: Unknown
              timeout: 5m0s
              type: Ready
            - status: "False"
              timeout: 12m0s
              type: Ready
  variables:
    - name: TKR_DATA
      required: false
      schema:
        openAPIV3Schema:
          type: object
          additionalProperties:
            type: object
            properties:
              kubernetesSpec:
                type: object
                properties:
                  version:
                    type: string
                  imageRepository:
                    type: string
                  etcd:
                    type: object
                    properties:
                      imageTag:
                        type: string
                  coredns:
                    type: object
                    properties:
                      imageTag:
                        type: string
              osImageRef:
                type: object
                properties:
                  name:
                    type: string
              labels:
                type: object
                additionalProperties:
                  type: string
    - name: vmClass
      required: true
      schema:
        openAPIV3Schema:
          type: string
    - name: storageClass
      required: true
      schema:
        openAPIV3Schema:
          type: string
    - name: storageClasses
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: string
    - name: defaultStorageClass
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: extensionCert
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            contentSecret:
              type: object
              properties:
                name:
                  type: string
                key:
                  type: string
    - name: clusterEncryptionConfigYaml
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: defaultRegistrySecret
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            namespace:
              type: string
            name:
              type: string
            data:
              type: string
    - name: ntp
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: user
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            passwordSecret:
              type: object
              properties:
                name:
                  type: string
                key:
                  type: string
            sshAuthorizedKey:
              type: string
    - name: nodePoolTaints
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              key:
                type: string
              value:
                type: string
              effect:
                type: string
              timeAdded:
                type: integer
    - name: nodePoolLabels
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              key:
                type: string
              value:
                type: string
    - name: proxy
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            httpProxy:
              type: string
            httpsProxy:
              type: string
            noProxy:
              type: array
              items:
                type: string
    - name: trust
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            additionalTrustedCAs:
              type: array
              items:
                type: object
                properties:
                  name:
                    type: string
    - name: controlPlaneVolumes
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              storageClass:
                type: string
              mountPath:
                type: string
              name:
                type: string
              capacity:
                type: object
                properties:
                  storage:
                    type: string                   
    - name: nodePoolVolumes
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              storageClass:
                type: string
              mountPath:
                type: string
              name:
                type: string
              capacity:
                type: object
                properties:
                  storage:
                    type: string

ClusterClass Variables for Customizing a Cluster

You customize a Cluster based on the tanzukubernetescluster ClusterClass using variables. Variables are defined using name-values pairs. The syntax must conform to the openAPIV3Schema.

Two variables are required to provision a Cluster using the v1beta1 API:
  • VM class
  • Storage class
Additional variables are available for customizing a Cluster, such as:
  • Proxy
  • TLS Certificates
  • SSH keys

The following sections list all variables that are available with the default tanzukubernetescluster ClusterClass.

Important: A valid key name must consist only of alphanumeric characters, a dash (such as key-name), an underscore (such as KEY_NAME) or a dot (such as key.name). You cannot use a space in a key name.

clusterEncryptionConfigYaml

Use the clusterEncryptionConfigYaml variable to configure cluster encryption.

clusterEncryptionConfigYaml
String which is a YAML file that provides encryption configuration details.
You can configure the encryption of data in the etcd database using the kube-apiserver Encryption Configuration package. See Encrypting Secret Data at Rest in the Kubernetes documentation.
...
    variables:
    #clusterEncryptionConfigYaml specifies the base64 encoded 
    #EncryptionConfiguration YAML
    #the YAML contains a base64 encryption configuration for the cluster identity
    #the key is generated randomly
    - name: clusterEncryptionConfigYaml
      value: string which is name of the EncryptionConfiguration YAML 

controlPlaneCertificateRotation

Use the controlPlaneCertificateRotation variable to configure the system to rotate the TLS certificates for control plane nodes by triggering a rollout of these certificates before they expire. If configured, control plane certificate rotation is available for all new and existing control plane nodes.
controlPlaneCertificateRotation
Boolean for activating the feature and number of days before expiration to rotate the certificates (minimum of 7 days before expiration). See Automatically rotating certificates using Kubeadm Control Plane provider for more information.
...
    variables:
    - name: controlPlaneCertificateRotation.activate
      value: true (default is false which means not enabled)
    - name: controlPlaneCertificateRotation.daysBefore
      value: EXPIRY-DAYS 

controlPlaneVolumes

Use the controlPlaneVolumes variable to configure persistent volumes for control plane nodes.
controlPlaneVolumes
Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string.
...
    variables:
      #controlPlaneVolumes is an optional set of PVCs to create and
      #attach to each node
      - name: controlPlaneVolumes
        value:
          #name of the PVC to be used as the suffix (node.name)
          - name: NAME
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /dir/path
            #storageClass is the storage class to use for the PVC
            storageClass: tkgs-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi           

defaultRegistrySecret

The defaultRegistrySecret variable can be used to configure the embedded Harbor Registry as the default container registry for the cluster.
Note: This variable is reserved for use with the embedded Harbor Registry.
defaultRegistrySecret
Object that includes a public key, certificate name, and namespace for a default container registry.
If you enable the embedded Harbor Registry on Supervisor, the defaultRegistrySecret variable specifies the registry service certificate that the Cluster trusts. The certificate secret is labeled with managed-by: vmware-vRegistry.
...
    variables:
    - name: defaultRegistrySecret
      value:
        #data holds the base64 encoded data.ca\.crt content
        #data.ca\.crt is already encoded, so raw cert data is encoded twice
        data: LS0tLS1CRUdJTiBDRVJU...S0tRU5EIENFUlRJRklDQVRFL
        #name specifies the name of the registry cert secret 
        name: harbor-ca-key-pair
        #namespace specifies the ns of the registry cert secret
        namespace: svc-harbor-domain-c9

defaultStorageClass

Use the defaultStorageClass variable to configure a default storage class for the cluster.

defaultStorageClass
String that identifies which storage class to use as the default storage class, often required by certain applications such as Helm charts and Tanzu Packages.
...
    variables:
    - name: defaultStorageClass
      value: tkg2-storage-profile

extensionCert

Use the extensionCert variable to configure a TLS certificate.

extensionCert
Object containing a contentSecret object containing name and key strings. The contentSecret references a Kubernetes secret object that has been created for a TLS certificate.
...
    variables:
    #extensionCert specifies the cert and key for Extensions Controller
    #self-signed issuer and certificates must be created in advance
    - name: extensionCert
      value: 
        contentSecret:
          #name specifies the name of secret
          name: string
          #key specifies the content of tls\.crt in the secret's data map
          key: string

nodePoolLabels

Use the nodePoolLabels variable to configure labels for worker nodes.

nodePoolLabels
Array of one or more objects, each object containing a key/value pair, both of which are strings.
Labels let you organize system objects according to your requirements for ease of querying and reporting. Refer to the Kubernetes labels documentation for usage details.

nodePoolTaints

Use the nodePoolTaints variable to apply taints to worker nodes.

nodePoolTaints
Array of objects, each object contains a taint that applies to worker nodes.
Each taint object includes a key (string), a value (string), and an effect (string). The system populates the timeAdded field on creation or update.

nodePoolVolumes

Use the nodePoolVolumes variable to specify persistent volumes for cluster nodes.

nodePoolVolumes
Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string.
...
    variables:
      #nodePoolVolumes is an optional set of PVCs to create and
      #attach to each node; use for high-churn components like containerd
      - name: nodePoolVolumes
        value: |
          #name of the PVC to be used as the suffix (node.name)
          - name: etcd
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /var/lib/containerd
            #storageClass is the storage class to use for the PVC
            storageClass: tkg2-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi           

ntp

Use the ntp variable to configure an NTP server for the cluster.

ntp
String which is the FQDN or IP address of an NTP server.
The NTP variable specifies the domain name of the NTP server such as shown in the example. The NTP server is injected into the Cluster variable at cluster creation. After cluster creation, you manage the server name rotation or any update by manually updating the cluster variable.
...
    variables:
    - name: ntp
      value: time1.vmware.com

proxy

Use the proxy variable to configure a proxy server for the cluster.

proxy
Object with parameters that reference a proxy server for outbound cluster connections.
The required proxy parameters are httpProxy, httpsProxy, and noProxy. All three fields are required if you include the proxy variable in the Cluster definition.
The fields httpProxy and httpsProxy take strings values that reference the URI of a proxy server that is configured to manage outbound HTTP and HTTPS connections from the TKG cluster. You can connect to the proxy server using HTTP. HTTPS connections are not supported.
The noProxy field is an array of strings. Obtain the noProxy values from the SupervisorWorkload Network. You must include in the noProxy field the Namespace Network, Ingress, and Egress subnets.
You do not need to include the Services subnet in the noProxy field. The TKG cluster does not interact with this subnet.
You do not need to include the clusterNetwork.services.cidrBlocks and the clusterNetwork.pods.cidrBlocks in the noProxy field. These endpoints are automatically not proxied for you.
You do not need to include localhost and 127.0.0.1 in the noProxy field. These endpoints are automatically not proxied for you.
...
    variables:
    #proxy specifies a proxy server to be used for the cluster
    #if omitted no proxy is configured
    - name: proxy
      value:
        #httpProxy is the proxy URI for HTTP connections
        #to endpoints outside the cluster
        httpProxy: http://<user>:<pwd>@<ip>:<port>
        #httpsProxy is the proxy URL for HTTPS connections 
        #to endpoints outside the cluster
        httpsProxy: http://<user>:<pwd>@<ip>:<port>
        #noProxy is the list of destination domain names, domains, 
        #IP addresses, and other network CIDRs to exclude from proxying
        #must include Supervisor Pod, Egress, Ingress CIDRs
        noProxy: [array of strings, comma-separated]

storageClass

Use the storageClass variable to configure a storage class for the cluster.

storageClass
String that is the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned.
...
    variables:
    - name: storageClass
      value: tkg2-storage-profile 
Use the following command to list available storage classes:
kubectl describe namespace VSPHERE-NAMESPACE-NAME
Or, if you have vSphere administrator privileges:
kubectl describe storageclasses

storageClasses

Use the storageClasses variable to configure an array of storage classes for the cluster.

storageClasses
Array of one or more strings, each string being the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned.
...
    variables:
    - name: storageClasses
      value: [tkg2-storage-profile, tkg2-storage-profile-latebinding] 
Use the following command to list available storage classes:
kubectl describe namespace VSPHERE-NAMESPACE-NAME
Or, if you have vSphere administrator privileges:
kubectl describe storageclasses

TKR_DATA

Use the TKR_DATA variable to specify TKR information.

TKR_DATA
Object that you use to specify the TKR version and other details.
The version is a string in the TKR NAME format to be used by cluster nodes.
Only TKRs that do not have the legacy-tkr label are compatible with TKG on vSphere 9 Supervisor. See Using Tanzu Kubernetes Releases with TKG Clusters on Supervisor.
The default operating system is PhotonOS. Use annotations to specify the Ubuntu TKR.

trust

Use the trust variable to specify one or more trusted CA certificates for the cluster.

trust
Object for adding TLS certificates to the Cluster, either additional CAs or end certificates.
The value is additionalTrustedCAs which holds an array of strings. For example:
#trust-example
    variables:
      - name: trust
        value:
          additionalTrustedCAs:
          - name: additional-ca-1
          - name: additional-ca-2
          - name: additional-ca-N
The value of each string is the name given to a Kubernetes secret data map that holds the contents of the additional trusted certificate as a double base64-encoded string that conforms to the PEM format. For example:
#secret-example
apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09
kind: Secret
metadata:
  name: cluster01-user-trusted-ca-secret
  namespace: tkgs-cluster-ns
type: Opaque
A common use case for the trust variable is to integrate a v1beta1 Cluster with a private container registry. See Integrate a TKGS Cluster with a Private Container Registry.

user

Use the user variable to specify cluster user credentials.

user
Object that includes a passwordSecret object, with name and key strings, and sshAuthorizedKey string. You can use this variable to add a user's SSH key to cluster nodes for remote SSH access.
The User variable specifies the SSH login credentials including password and authorized keys. The user name by default is vmware-system-user. The password needs to be hashed and stored in a secret in the same namespace where the Cluster is provisioned. The passwordSecret object references this secret. For example, on Linux you can generate a safe hash using mkpasswd --method=SHA-512 --rounds=4096. See Including users and groups for details.
...
    variables:
    #user specifies an authorized user and credentials
    - name: user
      value:
        #passwordSecret is an object that contains a Kubernetes secret and key
        passwordSecret:
          #name specifies the secret name
          name: string
          #key specifies the key value pair in the secret's data map
          key: string
        sshAuthorizedKey: string that is the base64-encoded public key

vmClass

Use the vmClass variable to configure the VM class for cluster nodes.

vmClass
Required string that maps to the name of a VM class that is bound to the vSphere Namespace where the TKG cluster is provisioned.
vmClass is the name of the VirtualMachineClass which describes the virtual hardware settings to be used for cluster nodes. The VirtualMachineClass controls the CPU and memory available to the node and the requests and limits on those resources. See Using Virtual Machine Classes with TKG Clusters on Supervisor.
You can only use VM classes that are bound to the vSphere Namespace where the TKG cluster is being provisioned. Use the command kubectl get virtualmachineclassbinding to list bound classes.
You can define the vmClass variable at different scopes so that you can use different VM classes for control plane nodes and node pool worker nodes.
For example, here an inline vmClass variable overrides the primary vmClass variable for this specific machineDeployment topology.
...
    workers:
      machineDeployments:
      - class: tkg-worker
        name: compute
        replicas: 3
        variables:
          overrides:
          - name: vmClass
            value: guaranteed-large