The Cluster v1beta1 API lets you provision a Cluster based on a default ClusterClass definition.

ClusterClass API v1beta1

The Kubernetes Cluster API is a suite of tools which provide for the declarative provisioning, upgrading, and operating of Kubernetes clusters. ClusterClass is an evolution of the Cluster API that lets you define templates for managing the life cycle of sets of clusters. TKG 2 on Supervisor supports ClusterClass using the v1beta1 API.

TKG 2 on Supervisor ships with a default ClusterClass definition named tanzukubernetescluster. The tanzukubernetescluster ClusterClass provides the template for TKG 2 cluster creation on Supervisor using the v1beta API. The tanzukubernetescluster ClusterClass is available in all user namespaces. To create a cluster based on this ClusterClass, reference it in the Cluster specification. Refer to the v1beta examples for guidance.

Default ClusterClass tanzukubernetescluster

The default tanzukubernetescluster ClusterClass is immutable. It is provided here as reference. 
apiVersion: cluster.x-k8s.io/v1beta1
kind: ClusterClass
metadata:
  name: tanzukubernetescluster
  annotations:
    run.tanzu.vmware.com/resolve-tkr: ""
spec:
  controlPlane:
    metadata:
      annotations:
        run.tanzu.vmware.com/resolve-os-image: os-name=photon
    ref:
      apiVersion: controlplane.cluster.x-k8s.io/v1beta1
      kind: KubeadmControlPlaneTemplate
      name: "tkc-control-plane"
    machineInfrastructure:
      ref:
        kind: VSphereMachineTemplate
        apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
        name: "tkc-control-plane"
    machineHealthCheck:
      maxUnhealthy: 100%
      nodeStartupTimeout: 2h0m0s
      unhealthyConditions:
        - status: Unknown
          timeout: 5m0s
          type: Ready
        - status: "False"
          timeout: 12m0s
          type: Ready
  infrastructure:
    ref:
      apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
      kind: VSphereClusterTemplate
      name: "tkc-infrastructure"
  workers:
    machineDeployments:
      - class: node-pool  
        template:
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-name=photon            
          bootstrap:
            ref:
              kind: KubeadmConfigTemplate
              apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
              name: "tkc-md"
          infrastructure:
            ref:
              kind: VSphereMachineTemplate
              apiVersion: vmware.infrastructure.cluster.x-k8s.io/v1beta1
              name: "tkc-md"
        machineHealthCheck:
          maxUnhealthy: 100%
          nodeStartupTimeout: 2h0m0s
          unhealthyConditions:
            - status: Unknown
              timeout: 5m0s
              type: Ready
            - status: "False"
              timeout: 12m0s
              type: Ready
  variables:
    - name: TKR_DATA
      required: false
      schema:
        openAPIV3Schema:
          type: object
          additionalProperties:
            type: object
            properties:
              kubernetesSpec:
                type: object
                properties:
                  version:
                    type: string
                  imageRepository:
                    type: string
                  etcd:
                    type: object
                    properties:
                      imageTag:
                        type: string
                  coredns:
                    type: object
                    properties:
                      imageTag:
                        type: string
              osImageRef:
                type: object
                properties:
                  name:
                    type: string
              labels:
                type: object
                additionalProperties:
                  type: string
    - name: vmClass
      required: true
      schema:
        openAPIV3Schema:
          type: string
    - name: storageClass
      required: true
      schema:
        openAPIV3Schema:
          type: string
    - name: storageClasses
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: string
    - name: defaultStorageClass
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: extensionCert
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            contentSecret:
              type: object
              properties:
                name:
                  type: string
                key:
                  type: string
    - name: clusterEncryptionConfigYaml
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: defaultRegistrySecret
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            namespace:
              type: string
            name:
              type: string
            data:
              type: string
    - name: ntp
      required: false
      schema:
        openAPIV3Schema:
          type: string
    - name: user
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            passwordSecret:
              type: object
              properties:
                name:
                  type: string
                key:
                  type: string
            sshAuthorizedKey:
              type: string
    - name: nodePoolTaints
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              key:
                type: string
              value:
                type: string
              effect:
                type: string
              timeAdded:
                type: integer
    - name: nodePoolLabels
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              key:
                type: string
              value:
                type: string
    - name: proxy
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            httpProxy:
              type: string
            httpsProxy:
              type: string
            noProxy:
              type: array
              items:
                type: string
    - name: trust
      required: false
      schema:
        openAPIV3Schema:
          type: object
          properties:
            additionalTrustedCAs:
              type: array
              items:
                type: object
                properties:
                  name:
                    type: string
    - name: controlPlaneVolumes
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              storageClass:
                type: string
              mountPath:
                type: string
              name:
                type: string
              capacity:
                type: object
                properties:
                  storage:
                    type: string                   
    - name: nodePoolVolumes
      required: false
      schema:
        openAPIV3Schema:
          type: array
          items:
            type: object
            properties:
              storageClass:
                type: string
              mountPath:
                type: string
              name:
                type: string
              capacity:
                type: object
                properties:
                  storage:
                    type: string

ClusterClass Variables for Customizing a Cluster

You customize a Cluster based on the tanzukubernetescluster ClusterClass using variables. Variables are defined using name-values pairs. The syntax must conform to the openAPIV3Schema.

Two variables are required to provision a Cluster using the v1beta1 API:
  • VM class
  • Storage class
Additional variables are available for customizing a Cluster, such as:
  • Proxy
  • TLS Certificates
  • SSH keys

The following lists all variables that are available with the default tanzukubernetescluster.

TKR_DATA
Object that you use to specify the TKR version and other details.
The version is a string in the TKR NAME format to be used by cluster nodes.
The *-zshippable TKR is compatible with TKG 2. The default operating system is PhotonOS. Use annotations to specify the Ubuntu TKR.
vmClass
Required string that maps to the name of a VM class that is bound to the vSphere Namespace where the TKG cluster is provisioned.
vmClass is the name of the VirtualMachineClass which describes the virtual hardware settings to be used for cluster nodes. The VirtualMachineClass controls the CPU and memory available to the node and the requests and limits on those resources. See Using Virtual Machine Classes with TKG 2 Clusters on Supervisor.
You can only use VM classes that are bound to the vSphere Namespace where the TKG cluster is being provisioned. Use the command kubectl get virtualmachineclassbinding to list bound classes.
You can define the vmClass variable at different scopes so that you can use different VM classes for control plane nodes and node pool worker nodes.
For example, here an inline vmClass variable overrides the primary vmClass variable for this specific machineDeployment topology. 
...
    workers:
      machineDeployments:
      - class: tkg-worker
        name: compute
        replicas: 3
        variables:
          overrides:
          - name: vmClass
            value: guaranteed-large
storageClass
String that is the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned. 
...
    variables:
    - name: storageClass
      value: tkg2-storage-profile
Use the following command to list available storage classes: 
kubectl describe namespace VSPHERE-NAMESPACE-NAME
Or, if you have vSphere administrator privileges: 
kubectl describe storageclasses
storageClasses
Array of one or more strings, each string being the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned. 
...
    variables:
    - name: storageClasses
      value: [tkg2-storage-profile, tkg2-storage-profile-latebinding]
Use the following command to list available storage classes: 
kubectl describe namespace VSPHERE-NAMESPACE-NAME
Or, if you have vSphere administrator privileges: 
kubectl describe storageclasses
defaultStorageClass
String that identifies which storage class to use as the default storage class, often required by certain applications such as Helm charts and Tanzu Packages. 
...
    variables:
    - name: defaultStorageClass
      value: tkg2-storage-profile
extensionCert
Object containing a contentSecret object containing name and key strings. The contentSecret references a Kubernetes secret object that has been created for a TLS certificate. 
...
    variables:
    #extensionCert specifies the cert and key for Extensions Controller
    #self-signed issuer and certificates must be created in advance
    - name: extensionCert
      value: 
        contentSecret:
          #name specifies the name of secret
          name: string
          #key specifies the content of tls\.crt in the secret's data map
          key: string
clusterEncryptionConfigYaml
String which is a YAML file that provides encryption configuration details.
You can configure the encryption of data in the etcd database using the kube-apiserver Encryption Configuration package. See Encrypting Secret Data at Rest in the Kubernetes documentation. 
...
    variables:
    #clusterEncryptionConfigYaml specifies the base64 encoded 
    #EncryptionConfiguration YAML
    #the YAML contains a base64 encryption configuration for the cluster identity
    #the key is generated randomly
    - name: clusterEncryptionConfigYaml
      value: string which is name of the EncryptionConfiguration YAML
defaultRegistrySecret
Object that includes a public key, certificate name, and namespace for a default container registry.
If you enable the Harbor Registry on Supervisor, the defaultRegistrySecret variable specifies the registry service certificate that the Cluster trusts. The certificate secret is labeled with managed-by: vmware-vRegistry. On Cluster creation, a defaultRegistry certificate is injected into the defaultRegistrySecret variable. After cluster creation, you manage the certificate rotation or any update by manually updating the variable. 
...
    variables:
    - name: defaultRegistrySecret
      value:
        #data holds the base64 encoded data.ca\.crt content
        #data.ca\.crt is already encoded, so raw cert data is encoded twice
        data: LS0tLS1CRUdJTiBDRVJU...S0tRU5EIENFUlRJRklDQVRFL
        #name specifies the name of the registry cert secret 
        name: harbor-ca-key-pair
        #namespace specifies the ns of the registry cert secret
        namespace: svc-harbor-domain-c9
ntp
String which is the FQDN or IP address of an NTP server.
The NTP variable specifies the domain name of the NTP server such as shown in the example. The NTP server is injected into the Cluster variable at cluster creation. After cluster creation, you manage the server name rotation or any update by manually updating the cluster variable. 
...
    variables:
    - name: ntp
      value: time1.vmware.com
user
Object that includes a passwordSecret object, with name and key strings, and sshAuthorizedKey string. You can use this variable to add a user's SSH key to cluster nodes for remote SSH access.
The User variable specifies the SSH login credentials including password and authorized keys. The user name by default is vmware-system-user. The password needs to be hashed and stored in a secret in the same namespace where the Cluster is provisioned. The passwordSecret object references this secret. For example, on Linux you can generate a safe hash using mkpasswd --method=SHA-512 --rounds=4096. See Including users and groups for details. 
...
    variables:
    #user specifies an authorized user and credentials
    - name: user
      value:
        #passwordSecret is an object that contains a Kubernetes secret and key
        passwordSecret:
          #name specifies the secret name
          name: string
          #key specifies the key value pair in the secret's data map
          key: string
        sshAuthorizedKey: string that is the base64-encoded public key
nodePoolTaints
Array of objects, each object contains a taint that applies to worker nodes.
Each taint object includes a key (string), a value (string), and an effect (string). The system populates the timeAdded field on creation or update.
nodePoolLabels
Array of one or more objects, each object containing a key/value pair, both of which are strings.
Labels let you organize system objects according to your requirements for ease of querying and reporting. Refer to the Kubernetes labels documentation for usage details.
proxy
Object with parameters that reference a proxy server for outbound cluster connections.
The required proxy parameters are httpProxy, httpsProxy, and noProxy. All three fields are required if you include the proxy variable in the Cluster definition.
The fields httpProxy and httpsProxy take strings values that reference the URI of a proxy server that is configured to manage outbound HTTP and HTTPS connections from the TKG cluster. You can connect to the proxy server using HTTP. HTTPS connections are not supported.
The noProxy field is an array of strings. Obtain the noProxy values from the SupervisorWorkload Network. You must include in the noProxy field the Namespace Network, Ingress, and Egress subnets.
You do not need to include the Services subnet in the noProxy field. The TKG cluster does not interact with this subnet.
You do not need to include the clusterNetwork.services.cidrBlocks and the clusterNetwork.pods.cidrBlocks in the noProxy field. These endpoints are automatically not proxied for you.
You do not need to include localhost and 127.0.0.1 in the noProxy field. These endpoints are automatically not proxied for you. 
...
    variables:
    #proxy specifies a proxy server to be used for the cluster
    #if omitted no proxy is configured
    - name: proxy
      value:
        #httpProxy is the proxy URI for HTTP connections
        #to endpoints outside the cluster
        httpProxy: http://<user>:<pwd>@<ip>:<port>
        #httpsProxy is the proxy URL for HTTPS connections 
        #to endpoints outside the cluster
        httpsProxy: http://<user>:<pwd>@<ip>:<port>
        #noProxy is the list of destination domain names, domains, 
        #IP addresses, and other network CIDRs to exclude from proxying
        #must include Supervisor Pod, Egress, Ingress CIDRs
        noProxy: [array of strings, comma-separated]
trust
Object for adding TLS certificates to the Cluster, either additional CAs or end certificates.
The value is additionalTrustedCAs which is an array of strings. Each string is the name of Kubernetes secret that holds the contents of the additional trusted certificate in the PEM format base64-encoded string.
To configure a trust variable:
  1. Create a Kubernetes secret definition YAML file with the following contents.
    apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09
kind: Secret
metadata:
  name: cluster01-user-trusted-ca-secret
  namespace: tkg2-cluster-ns
type: Opaque

    The content of the secret's data map is a user-defined name for the certificate (additional-ca-1) whose value is a base64-encoded certificate. Name the secret CLUSTER-NAME-user-trusted-ca-secret, where CLUSTER-NAME is the name of the Cluster. This secret must be created in the same vSphere Namespace as the Cluster.

  2. Create the Kubernetes secret using kubectl -f apply additional-ca-1.yaml.
  3. Verify secret creation.
    kubeclt get secret -n tkg2-cluster-ns cluster01-user-trusted-ca-secret
NAME                                             TYPE     DATA   AGE
cluster01-user-trusted-ca-secret                   Opaque   12     2d22h
  4. Include the trust variable in the Cluster spec that references the name value of the data map for the secret, which in this example is additional-ca-1.
    spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 193.0.0.0/16
    serviceDomain: cluster.local
    services:
      cidrBlocks:
      - 198.201.0.0/16
  topology:
    class: tanzukubernetescluster
    controlPlane:
      metadata: {}
      replicas: 3
    variables:
    - name: storageClass
      value: tkg2-storage-profile
    - name: trust
      value:
        additionalTrustedCAs:
        - name: additional-ca-1
controlPlaneVolumes
Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string. 
...
    variables:
      #controlPlaneVolumes is an optional set of PVCs to create and
      #attach to each node; use for high-churn components like etcd
      - name: controlPlaneVolumes
        value: |
          #name of the PVC to be used as the suffix (node.name)
          - name: etcd
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /var/lib/etcd
            #storageClass is the storage class to use for the PVC
            storageClass: tkg2-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi
nodePoolVolumes
Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string. 
...
    variables:
      #nodePoolVolumes is an optional set of PVCs to create and
      #attach to each node; use for high-churn components like containerd
      - name: nodePoolVolumes
        value: |
          #name of the PVC to be used as the suffix (node.name)
          - name: etcd
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /var/lib/containerd
            #storageClass is the storage class to use for the PVC
            storageClass: tkg2-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi