The builtin-generic-v3.2.0 ClusterClass introduces a new variable schema. You can use the following ClusterClass variables to customize the cluster you deploy.

vmClass

Use the vmClass variable to configure the VM class for cluster nodes.

Use the vmClass variable to configure the VM class for cluster nodes.

vmClass is a required string that maps to the name of a VM class that is bound to the vSphere Namespace where the TKG cluster is provisioned.

The name of the VirtualMachineClass describes the virtual hardware settings to be used for cluster nodes. The VirtualMachineClass controls the CPU and memory available to the node and the requests and limits on those resources. See Using VM Classes with TKG Service Clusters.

You can only use VM classes that are associated with the vSphere Namespace where the TKG cluster is being provisioned. Use the command kubectl get virtualmachineclass to list bound classes.

Example

Note:

The vmClass variable can be overridden at the control plane or on a per node pool level to set the VirtualMachineClass used to provision the specific nodes.

This is a required input and it is mandatory.

# Setting the variable at the cluster level causes the same VMClass 
# to be used by all control plane and node pool nodes.
variables:
  - name: vmClass
    value: best-effort-medium

storageClass

Use the storageClass variable to configure a storage class for the cluster.

storageClass is a string that is the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG service cluster is provisioned. It is a required input.

Use the following command to list available storage classes: 
kubectl describe namespace VSPHERE-NAMESPACE-NAME
Or, if you have vSphere administrator privileges: 
kubectl describe storageclasses

Example

Note: This variable can be overridden at the control plane or on a per node pool level to set the storage profile for a specific node.

This is a required input and is mandatory.

variables:
  - name: storageClass
    value: tkgs-storage-profile

volumes

Use the volumes variable to configure additional disks for the nodes of the Cluster.

volumes is an optional array of objects, each of which includes name, storageClass, mountPath, and capacity.

Field Name Expected Type Example
name string kubelet-override
mountPath string /var/lib/kubelet

or

C:\var\lib\kubelet
storageClass string tkgs-storage-profile
capacity quantity 20Gi

or

50G

Example

Note:

The volumes variable can be overridden at the control plane or on a per node pool level to set the volume mount for specific node groups.

The following example sets a custom volume mount for the containerd directory of the nodes of the cluster.

variables:
- name: volumes
  value:
      #name of the PVC to be used as the suffix (node.name)
    - name: containerd
      # mountPath is the directory where the volume device is mounted
      # takes the form /dir/path (for Linux nodes) and C:\foo\bar (for Windows nodes)
      mountPath: /var/lib/containerd
      # storageClass is the storage class to use for the PVC
      storageClass: tkgs-storage-profile
      # capacity is the PVC storage capacity
      capacity: 40Gi

Use the following example for the volume overrides for containerd and kubelet directories for a Windows node pool.

variables:
- name: volumes
  value:
    - name: containerd-override
      mountPath: C:\ProgramData\containerd
      storageClass: tkgs-storage-profile
      capacity: 40Gi
    - name: kubelet-override
      mountPath: C:\var\lib\kubelet
      storageClass: tkgs-storage-profile
      capacity: 40Gi

node

Use the node variable to configure different properties, such as labels or taints on the Kubernetes node objects of the cluster.

The variable includes these two sub-sections.

labels

An optional set of key value pairs that are added as Kubernetes labels to the node. The key value pairs follow the same format as documented in Kubernetes upstream documentation for labels.

taints

An array of objects, each object contains a Kubernetes taints that applies to the node. Each object in the array has the following properties:
Field Name Expected Type Example Required
key string foo true
value string bar

or

""		 true
effect One of the following:
  • NoExecute
  • NoSchedule
  • PreferNoSchedule
 NoSchedule true

Example

Note: The node variable can be set at the cluster level and can be overridden at the control plane or on a per node pool level to set values for specific node groups.
The following example adds three label pairs and two taints to the nodes. 
variables:
- name: node
  value:
    labels:
      tenant: tenant-foo
      organization: engineering
      managed: ""
    taints:
    - key: key1
      value: value1
      effect: NoSchedule
    - key: key2
      value: value2
      effect: NoExecute

kubernetes

Use the kubernetes variable to configure optional settings for the cluster.

The variable includes these sub-sections.

ceritificateRotation

Use this section to configure the system to rotate the TLS certificates for control plane nodes by triggering a rollout of these certificates before they expire. Control plane certificate rotation is available for all new and existing control plane nodes. See Automatically rotating certificates using Kubeadm Control Plane provider for more information.

The following fields are exposed in this section:
Field Name Expected Type Example Default Required
enabled boolean false true No
renewalDaysBeforeExpiry integer 100 90 No

To avoid automatic certificate rotation on the control plane nodes, explicitly set enabled to false.

endpointFQDNs

An array of one or more fully qualified domain names (FQDN). The Kubernetes API certificate that is generated includes each of the FQDNs that have been supplied through this field. The system populates the kubeconfig with the first FQDN from the list, and assumes it is resolvable. If you want to use a different FQDN from the list, you can manually edit the generated kubeconfig file with the specific FQDN from the variable list.

security

Use this section to configure Kubernetes specific security settings, for example, podSecurityStandard.

podSecurityStandard
Use this section to configure cluster-wide pod security. With TKr v1.26 and later, by default pod security (PSA) restrictions are enforced at the namespace level using annotation labels. See Configure PSA for TKR 1.25 and Later.
Alternatively, you can use the podSecurityStandard variable to configure cluster-wide PSA when you provision or update a v1beta1 cluster.
The table lists the fields available in this section.
Field Name Expected Type Example Required

deactivated

boolean false No
audit One of the following:
  • privileged
  • baseline
  • restricted
 restricted No
auditVersion
  • latest – Sets the audit version to latest.
  • v<k8s-major>.<k8s-minor> – For example, v1.28 sets the audit version enforcement to the Kubernetes version 1.28.
 latest No

enforce

One of the following:
  • privileged
  • baseline
  • restricted
 restricted No
enforceVersion
  • latest – Sets the version to latest.
  • v<k8s-major>.<k8s-minor> – For example, v1.28 sets the enforce version enforcement to the Kubernetes version 1.28.
 v1.27 No
warn One of the following:
  • privileged
  • baseline
  • restricted
 restricted No
warnVersion
  • latest – Sets the version to latest.
  • v<k8s-major>.<k8s-minor> – For example, v1.28 sets the warn version enforcement to the Kubernetes version 1.28.
 v1.29 No

exemptions

object namespaces: ["foo-ns", "bar-ns"] No
Note: System namespaces are excluded from pod security, including kube-system, tkg-system, and vmware-system-cloud-provider.
Examples: 
# The following example shows the defaults.
variables:
- name: podSecurityStandard
  value:
    enforce: "restricted"
    enforceVersion: "latest"

# The following example provides audit logs and warnings to identify workloads that are not following current pod hardening best practices, but only enforces a minimally restrictive policy ("baseline") which prevents known privilege escalations.
variables:
- name: podSecurityStandard
  value:
    audit: "restricted"
    warn: "restricted"
    enforce: "baseline"

# The following example enforces restricted policy, except on a specific namespace.
variables:
- name: podSecurityStandard
  value:
    audit: "restricted"
    warn: "restricted"
    enforce: "restricted"
    exemptions:
      namesaces: ["privileged-workload-ns"]

# The following example restricts enforcement to a particular TKr version.
variables:
- name: podSecurityStandard
  value: 
    auditVersion: "v1.26"
    warnVersion: "v1.26"
    enforceVersion: "v1.26"

Example

Note: Set the kubernetes variable at the cluster level. Overriding at the control plane or on a per node pool level does not yield different results since these properties are applicable to the cluster as a whole.
Use the following as an example: 
variables:
- name: kubernetes
  value:
    certificateRotation:
      enabled: true
      renewalDaysBeforeExpiry: 99
    endpointFQDNs:
    - fqdn1.cluster.internal
    - fqdn2.engineering.org
    security:
      podSecurityStandard:
        deactivated: false
        audit: privileged
        auditVersion: v1.27
        enforce: restricted
        enforceVersion: latest
        warn: baseline
        warnVersion: latest

vsphereOptions

vsphereOptions is an optional variable to set vSphere specific settings for the cluster.

This variable has a single sub-section.

persistentVolumes

This section supports options related to the persistent volumes on the cluster. It supports propagating the storage classes and volume snapshots classes to be available in the cluster as well as setting the default values for both options.

Multiple child fields under this section include the following.
availableStorageClasses
The list of storage classes that should be made available in the cluster. Each value in this list should be the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the cluster is provisioned. This is an optional field and accepts an array of strings. When the field is empty, all storage profiles associated with the vSphere Namespace will be made available in the cluster. Use the command kubectl get storageclass in the cluster to list the storage classes available for use.
defaultStorageClass
The default storage class for the provisioned cluster. This value should be present in the list of available storage classes, if the availableStorageClasses field is set. This is an optional field and accepts a string. The output of the command kubectl get storageclass will show if a storage class has been marked as default. The annotation storage.kubernetes.io/is-default-class denotes whether a class is default.
availableSnapshotClasses
The list of volume snapshot classes available for the provisioned cluster. This is an optional field and accepts an array of strings. When the field is empty, no volume snapshot classes are available in the cluster. Use the command kubectl get volumesnapshotclass in the cluster to list the volume snapshot classes available for use.
defaultVolumeSnapshotClass
The default volume snapshot class for the provisioned cluster. This value should be present in the list of available volume snapshot classes, if the availableVolumeSnapshotClasses field is set. This is an optional field and accepts a string. The output of the command kubectl get volumesnapshotclass will show if a volume snapshot class has been marked as default. The annotation snapshot.storage.kubernetes.io/is-default-class: "true" denotes whether a class is default.

Example

Note: Set the vsphereOptions variable at the cluster level. Overriding at the control plane or on a per node pool level does not yield different results since these properties are applicable to the cluster as a whole.
Use the following as an example: 
variables:
- name: vsphereOptions
  value:
    persistentVolumes:
      availableStorageClasses:
      - storage-class-foo
      - storage-class-bar
      defaultStorageClass: storage-class-bar
      availableVolumeSnapshotClasses:
      - vol-snapclass-foo
      - vol-snapclass-bar
      defaultVolumeSnapshotClass: vol-snapclass-bar

osConfiguration

osConfiguration is an optional variable used to define the OS level configurations for the cluster nodes.

The variable includes the following subsections.

ntp

This is an optional array of the domain name of the NTP server. The NTP server is added into the cluster variable at cluster creation. After the cluster creation, you manage the server name rotation or any other updates by manually updating the cluster variable.

If not specified, a default is selected during the time of cluster creation.

systemProxy

This is an optional input to reference a proxy server for outbound cluster connections.

Multiple child fields under this section include the following.

http
A required input that references the URI of a proxy server that is configured to manage outbound HTTP connections from the TKG cluster. You can connect to the proxy server using HTTP. If an authenticated proxy is used, the format should be http: http://<user>:<pwd>@<ip>:<port>.
https
A required input that references the URI of a proxy server that is configured to manage outbound HTTPS connections from the TKG cluster. HTTPS connections are not supported. If an authenticated proxy is used, the format should be https: http://<user>:<pwd>@<ip>:<port>.
noProxy
A required input that takes as input an array of strings. Obtain the values from the SupervisorWorkload Network. The Namespace Network, Ingress, and Egress subnets must be included in the field. You do not need to include the Services subnet in the noProxy field. The TKG cluster does not interact with this subnet. You do not need to include the clusterNetwork.services.cidrBlocks and the clusterNetwork.pods.cidrBlocks in the noProxy field. These endpoints are automatically not proxied for you.

trust

This is an optional input object for adding TLS certificates to the cluster, either additional CAs or end certificates.

It has a single sub-section.

additionalTrustedCAs
An optional array of input objects with the marker caCert containing two mutually exclusive top level keys.
  • secretRef This can be used to provide the name of the secret and the key from the data map field which contains the value of the CA or end certificate. The secret should be present in the same namespace as the cluster, and must be created in advance. The value of each string is the user-defined name for the data map field in the Kubernetes secret that contains the CA certificate in PEM format that is double base64-encoded. For example: 
    #secret-example
apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09
kind: Secret
metadata:
  name: custom-ca-secret
  namespace: cluster-ns
type: Opaque
  • content This can be used to provide certificate content in plain text. This should only be used if the secret data is viewable on the cluster CR.

user

An optional input to specify user credentials. It specifies the SSH login credentials including username, password, and authorized keys. You can use this to add a user's SSH key to cluster nodes for remote SSH access.

The password needs to be hashed and stored in a secret in the same namespace where the cluster is provisioned. The passwordSecret object references this secret. For example, on Linux you can generate a safe hash using mkpasswd --method=SHA-512 --rounds=4096. See Including users and groups for details.

If this is not specified, a default user with the vmware-system-user username is added for SSH access.

Example

Note: Set the osConfiguration variable at the cluster level. Overriding at the control plane or on a per node pool level does not yield different results since these properties are applicable to the cluster as a whole.
Use the following as an example: 
variables:
- name: osConfiguration
  value:
    ntp:
      servers:
        - ntp.test
    systemProxy:
      http: http://1.2.3.4:2139
      https: http://4.3.2.1:2139
      noProxy:
        - no.proxy.test1
        - no.proxy.test2
    trust:
      additionalTrustedCAs:
        - caCert:
            secretRef:
              key: trust-ca-test1
              name: "trust-ca-secret-1"
        - caCert:
            # For existing clusters which are auto rebased to new CC from version 3.2 onwards
            # the name falls back to <cluster-name>-user-trusted-ca-secret.
            secretRef:
              key: trust-ca-test2
              name: ""
        - caCert:
            # Plain text CA cert can be added using the `content` key
            content: |-
              -----BEGIN CERTIFICATE-----
              MIIEczCCA1ugAwIBAgIBADANBgkqhkiG9w0BAQQFAD..AkGA1UEBhMCR0Ix
              EzARBgNVBAgTClNvbWUtU3RhdGUxFDASBgNVBAoTC0..0EgTHRkMTcwNQYD
              VQQLEy5DbGFzcyAxIFB1YmxpYyBQcmltYXJ5IENlcn..XRpb24gQXV0aG9y
              aXR5MRQwEgYDVQQDEwtCZXN0IENBIEx0ZDAeFw0wMD..TUwMTZaFw0wMTAy
              MDQxOTUwMTZaMIGHMQswCQYDVQQGEwJHQjETMBEGA1..29tZS1TdGF0ZTEU
              MBIGA1UEChMLQmVzdCBDQSBMdGQxNzA1BgNVBAsTLk..DEgUHVibGljIFBy
              aW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxFD..AMTC0Jlc3QgQ0Eg
              THRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCg..Tz2mr7SZiAMfQyu
              vBjM9OiJjRazXBZ1BjP5CE/Wm/Rr500PRK+Lh9x5eJ../ANBE0sTK0ZsDGM
              ak2m1g7oruI3dY3VHqIxFTz0Ta1d+NAjwnLe4nOb7/..k05ShhBrJGBKKxb
              8n104o/5p8HAsZPdzbFMIyNjJzBM2o5y5A13wiLitE..fyYkQzaxCw0Awzl
              kVHiIyCuaF4wj571pSzkv6sv+4IDMbT/XpCo8L6wTa..sh+etLD6FtTjYbb
              rvZ8RQM1tlKdoMHg2qxraAV++HNBYmNWs0duEdjUbJ..XI9TtnS4o1Ckj7P
              OfljiQIDAQABo4HnMIHkMB0GA1UdDgQWBBQ8urMCRL..5AkIp9NJHJw5TCB
              tAYDVR0jBIGsMIGpgBQ8urMCRLYYMHUKU5AkIp9NJH..aSBijCBhzELMAkG
              A1UEBhMCR0IxEzARBgNVBAgTClNvbWUtU3RhdGUxFD..AoTC0Jlc3QgQ0Eg
              THRkMTcwNQYDVQQLEy5DbGFzcyAxIFB1YmxpYyBQcm..ENlcnRpZmljYXRp
              b24gQXV0aG9yaXR5MRQwEgYDVQQDEwtCZXN0IENBIE..DAMBgNVHRMEBTAD
              AQH/MA0GCSqGSIb3DQEBBAUAA4IBAQC1uYBcsSncwA..DCsQer772C2ucpX
              xQUE/C0pWWm6gDkwd5D0DSMDJRqV/weoZ4wC6B73f5..bLhGYHaXJeSD6Kr
              XcoOwLdSaGmJYslLKZB3ZIDEp0wYTGhgteb6JFiTtn..sf2xdrYfPCiIB7g
              BMAV7Gzdc4VspS6ljrAhbiiawdBiQlQmsBeFz9JkF4..b3l8BoGN+qMa56Y
              It8una2gY4l2O//on88r5IWJlm1L0oA8e4fR2yrBHX..adsGeFKkyNrwGi/
              7vQMfXdGsRrXNGRGnX+vWDZ3/zWI0joDtCkNnqEpVn..HoX
              -----END CERTIFICATE-----
    user:
      passwordSecret:
        key: user-secret-key-test
        name: user-secret-name-test
      sshAuthorizedKey: sshAuthorizedKeyTest...
      user: customuser

resourceConfiguration

resourceConfiguration is an optional variable can be used to configure resource requirements for the Kubernetes components.

This variable has a single sub-section.

systemReserved

Supports specifying the CPU and memory resources reserved for system processes. This information is passed as a kubelet argument to the Kubelet process on the nodes, specifically --system-reserved.

For more information about the flag, see the upstream Kubernetes kubelet docs.

The fields under this section include the following.

cpu
Describes the number of CPU cores reserved for system processes. Inputs should be of type Quantity. For a detailed explanation of CPU resource units in Kubernetes, refer to Meaning of CPU.

An example input value of 1 describes 1 virtual CPU core.

memory
Describes the memory resources reserved for system processes. Inputs should be of type Quantity. For a detailed explanation of memory resource units in Kubernetes, refer to Meaning of Memory.
automatic
Configures the automatic calculation of system reserved resources. The default value is true.

If this section is left unset, these values are calculated automatically.

Example

Note: The resourceConfiguration variable can be overridden at the control plane or on a per node pool level to set the volume mount for specific node groups.
Use the following as an example: 
variables:
- name: resourceConfiguration
  value:
    systemReserved:
      cpu: 1
      memory: 4G
      automatic: false

kubeAPIServerFQDNs

kubeAPIServerFQDNs is a deprecated field and is only present for backwards compatibility.

Use kubernetes.endpointFQDNs instead.

TKR_DATA

TKR_DATA is a deprecated field and is only present for backwards compatibility.

This value is computed internally and does need to be set.