Cluster v1beta1 API

The Cluster v1beta1 API lets you provision a Cluster based on a default ClusterClass definition.

ClusterClass API v1beta1

The Kubernetes Cluster API is a suite of tools which provide for the declarative provisioning, upgrading, and operating of Kubernetes clusters. ClusterClass is an evolution of the Cluster API that lets you define templates for managing the life cycle of sets of clusters. TKG Service supports ClusterClass using the v1beta1 API.

TKG Service ships with a default ClusterClass definition named tanzukubernetescluster. The tanzukubernetescluster ClusterClass provides the template for cluster creation using the v1beta API. The tanzukubernetescluster ClusterClass is available in all user namespaces. To create a cluster based on this ClusterClass, reference it in the Cluster specification. Refer to the v1beta examples for guidance.

Default ClusterClass tanzukubernetescluster

The default tanzukubernetescluster ClusterClass is immutable. It may be updated with each release of the TKG Service.

To view the default tanzukubernetescluster ClusterClass that ships with your TKG Service instance, complete the following steps:

1. Log in to Supervisor.

   kubectl vsphere login --server=IP-or-FQDN --vsphere-username [email protected]

2. Switch context the vSphere Namespace where a TKGS cluster is provisioned.

   kubeclt config use-context VSPEHRE-NS

3. Get the default tanzukubernetescluster ClusterClass.

   kubectl get clusterclass tanzukubernetescluster -o yaml

4. Optionally you can write the output of the default ClusterClass to a file named tkc-dcc.yaml.

   kubectl get clusterclass tanzukubernetescluster -o yaml > tkc-dcc.yaml

ClusterClass Variables for Customizing a Cluster

You customize a Cluster based on the tanzukubernetescluster ClusterClass using variables. Variables are defined using name-values pairs. The syntax must conform to the openAPIV3Schema.

Two variables are required to provision a Cluster using the v1beta1 API:

• VM class
• Storage class

Additional variables are available for customizing a Cluster, such as:

• Proxy
• TLS Certificates
• SSH keys

The following sections list all variables that are available with the default tanzukubernetescluster ClusterClass.

Important: A valid key name must consist only of alphanumeric characters, a dash (such as key-name), an underscore (such as KEY_NAME) or a dot (such as key.name). You cannot use a space in a key name.

clusterEncryptionConfigYaml

Use the clusterEncryptionConfigYaml variable to configure cluster encryption.

clusterEncryptionConfigYaml

String which is a YAML file that provides encryption configuration details.

You can configure the encryption of data in the etcd database using the kube-apiserver Encryption Configuration package. See Encrypting Secret Data at Rest in the Kubernetes documentation.

...
    variables:
    #clusterEncryptionConfigYaml specifies the base64 encoded 
    #EncryptionConfiguration YAML
    #the YAML contains a base64 encryption configuration for the cluster identity
    #the key is generated randomly
    - name: clusterEncryptionConfigYaml
      value: string which is name of the EncryptionConfiguration YAML 

controlPlaneCertificateRotation

Use the controlPlaneCertificateRotation variable to configure the system to rotate the TLS certificates for control plane nodes by triggering a rollout of these certificates before they expire. Control plane certificate rotation is available for all new and existing control plane nodes.

controlPlaneCertificateRotation

Boolean for activating the feature and number of days before expiration to rotate the certificates. See Automatically rotating certificates using Kubeadm Control Plane provider for more information.

...
    variables:
    - name: controlPlaneCertificateRotation
      value: 
        activate: true
        daysBefore: 90

Where:

• activate is a boolean for activating the feature; default is true

• daysBefore is the duration in the number of days before expiration; default is 90 days; minimum is 7 days. To forcefully rotate certs, set this value to be greater than the number of days remaining when the certificate expires. Reset it back to the default once the certificates have been rotated.

Note: This variable is updated for vSphere 8 Update 3 ( TKG Service 3.0).

controlPlaneVolumes

Use the controlPlaneVolumes variable to configure persistent volumes for control plane nodes.

controlPlaneVolumes

Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string.

...
    variables:
      #controlPlaneVolumes is an optional set of PVCs to create and
      #attach to each node
      - name: controlPlaneVolumes
        value:
          #name of the PVC to be used as the suffix (node.name)
          - name: NAME
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /dir/path
            #storageClass is the storage class to use for the PVC
            storageClass: tkgs-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi           

defaultRegistrySecret

The defaultRegistrySecret variable configures the default container registry for the cluster.

Note: This variable is reserved for use with the embedded Harbor Registry.

defaultRegistrySecret

Object that includes a public key, certificate name, and namespace for the default container registry.

If you enable the embedded Harbor Registry on Supervisor, the defaultRegistrySecret variable specifies the registry service certificate that the Cluster trusts. The certificate secret is labeled with managed-by: vmware-vRegistry. On Cluster creation, a defaultRegistry certificate is injected into the defaultRegistrySecret variable. After cluster creation, you manage the certificate rotation or any update by manually updating the variable.

...
    variables:
    - name: defaultRegistrySecret
      value:
        #data holds the base64 encoded data.ca\.crt content
        #data.ca\.crt is already encoded, so raw cert data is encoded twice
        data: LS0tLS1CRUdJTiBDRVJU...S0tRU5EIENFUlRJRklDQVRFL
        #name specifies the name of the registry cert secret 
        name: harbor-ca-key-pair
        #namespace specifies the ns of the registry cert secret
        namespace: svc-harbor-domain-c9

defaultStorageClass

Use the defaultStorageClass variable to configure a default storage class for the cluster.

defaultStorageClass

String that identifies which storage class to use as the default storage class, often required by certain applications such as Helm charts and Tanzu Packages.

...
    variables:
    - name: defaultStorageClass
      value: tkg2-storage-profile

extensionCert

Use the extensionCert variable to configure a TLS certificate.

extensionCert

Object containing a contentSecret object containing name and key strings. The contentSecret references a Kubernetes secret object that has been created for a TLS certificate.

...
    variables:
    #extensionCert specifies the cert and key for Extensions Controller
    #self-signed issuer and certificates must be created in advance
    - name: extensionCert
      value: 
        contentSecret:
          #name specifies the name of secret
          name: string
          #key specifies the content of tls\.crt in the secret's data map
          key: string

kubeAPIServerFQDNs

Use the kubeAPIServerFQDNs variable to configure a cluster with an FQDN.

kubeAPIServerFQDNs

Array of one or more fully qualified domain names (FQDN).

The Kubernetes API certificate that is generated will include each of the FQDNs that you have specified in the kubeAPIServerFQDNs variable. The system will populate the kubeconfig with the first FQDN from the list, and assume it is resolvable. If you want to use a different FQDN from the list, you can manually edit the generated kubeconfig file with the desired FQDN from the variable list.

See v1beta1 Example: Cluster with FQDN for details.

nodePoolLabels

Use the nodePoolLabels variable to configure labels for worker nodes.

nodePoolLabels

Array of one or more objects, each object containing a key/value pair, both of which are strings.

Labels let you organize system objects according to your requirements for ease of querying and reporting. Refer to the Kubernetes labels documentation for usage details.

nodePoolTaints

Use the nodePoolTaints variable to apply taints to worker nodes.

nodePoolTaints

Array of objects, each object contains a taint that applies to worker nodes.

Each taint object includes a key (string), a value (string), and an effect (string). The system populates the timeAdded field on creation or update.

nodePoolVolumes

Use the nodePoolVolumes variable to specify persistent volumes for cluster nodes.

nodePoolVolumes

Optional array of objects, each of which includes name, storageClass, and mountPath, each of which are strings, and an optional capacity object that includes a storage string.

...
    variables:
      #nodePoolVolumes is an optional set of PVCs to create and
      #attach to each node; use for high-churn components like containerd
      - name: nodePoolVolumes
        value: |
          #name of the PVC to be used as the suffix (node.name)
          - name: etcd
            #mountPath is the directory where the volume device is mounted
            #takes the form /dir/path
            mountPath: /var/lib/containerd
            #storageClass is the storage class to use for the PVC
            storageClass: tkgs-storage-profile
            #capacity is the PVC storage capacity
            capacity:
              #storage sets the capacity for the disk volume
              #if not specified defaults to storageClass capacity
              storage: 4Gi           

ntp

Use the ntp variable to configure an NTP server for the cluster.

ntp

String which is the FQDN or IP address of an NTP server.

The NTP variable specifies the domain name of the NTP server such as shown in the example. The NTP server is injected into the Cluster variable at cluster creation. After cluster creation, you manage the server name rotation or any update by manually updating the cluster variable.

...
    variables:
    - name: ntp
      value: time1.vmware.com

podSecurityStandard

Use the podSecurityStandard variable to configure cluster-wide pod security.

Note: This variable is new for vSphere 8 Update 3 ( TKG Service 3.0).

podSecurityStandard

With TKr v1.26 and later, by default pod security (PSA) restrictions are enforced at the namespace level using annotation labels. See Configure PSA for TKR 1.25 and Later.

Alternatively you can use the podSecurityStandard variable to configure cluster-wide PSA when you provision or update a v1beta1 cluster.

The podSecurityStandard variable can be implemented as follows:

...
variables:
- name: podSecurityStandard
  value: 
    deactivated: DEACTIVATED
    audit: AUDIT-PROFILE
    enforce: ENFORCE-PROFILE
    warn: WARN-PROFILE
    auditVersion: AUDIT-VERSION
    enforceVersion: ENFORCE-VERSION
    warnVersion: WARN-VERSION
    exemptions: 
      namespaces: [EXEMPT-NS]

Where:

• The DEACTIVATED value is false (default) to apply cluster-wide PSA and true otherwise.
• The *-PROFILE value is the PSA profile for each mode, which can be "privileged", "baseline", or "restricted" (default).
• The *-VERSION value is the Kubernetes version for each mode, such as "v1.26". The value "latest" is the default.
• The EXEMPT-NS value is an comma-separated list of namespaces to exclude from PSA control.

Note: System namespaces are excluded from pod security, including kube-system, tkg-system, and vmware-system-cloud-provider.

If you do not implement the podSecurityStandard variable, the default PSA behavior is preserved. If you include the podSecurityStandard variable in the cluster specification, the variable settings will control, including its defaults unless you override them.

The following example shows the defaults.

...
    variables:
      - name: podSecurityStandard
        value:
          enforce: "restricted"
          enforce-version: "latest"

The following example provides audit logs and warnings to identify workloads that are not following current pod hardening best practices, but only enforces a minimally restrictive policy ("baseline") which prevents known privilege escalations.

...
    variables:
      - name: podSecurityStandard
        value:
          audit: "restricted"
          warn: "restricted"
          enforce: "baseline"

The following example enforces restricted policy, except on a specific namespace.

...
    variables:
      - name: podSecurityStandard
        value:
          audit: "restricted"
          warn: "restricted"
          enforce: "restricted"
          exemptions:
            namesaces: ["privileged-workload-ns"]

The following example restricts enforcement to a particular TKr version.

...
variables:
  - name: podSecurityStandard
    value: 
      audit-version: "v1.26"
      warn-version: "v1.26"
      enforce-version: "v1.26"

For more examples, see pod security standards in the Kubernetes documentation.

proxy

Use the proxy variable to configure a proxy server for the cluster.

proxy

Object with parameters that reference a proxy server for outbound cluster connections.

The required proxy parameters are httpProxy, httpsProxy, and noProxy. All three fields are required if you include the proxy variable in the Cluster definition.

The fields httpProxy and httpsProxy take strings values that reference the URI of a proxy server that is configured to manage outbound HTTP and HTTPS connections from the TKG cluster. You can connect to the proxy server using HTTP. HTTPS connections are not supported.

The noProxy field is an array of strings. Obtain the noProxy values from the SupervisorWorkload Network. You must include in the noProxy field the Namespace Network, Ingress, and Egress subnets.

You do not need to include the Services subnet in the noProxy field. The TKG cluster does not interact with this subnet.

You do not need to include the clusterNetwork.services.cidrBlocks and the clusterNetwork.pods.cidrBlocks in the noProxy field. These endpoints are automatically not proxied for you.

You do not need to include localhost and 127.0.0.1 in the noProxy field. These endpoints are automatically not proxied for you.

...
    variables:
    #proxy specifies a proxy server to be used for the cluster
    #if omitted no proxy is configured
    - name: proxy
      value:
        #httpProxy is the proxy URI for HTTP connections
        #to endpoints outside the cluster
        httpProxy: http://<user>:<pwd>@<ip>:<port>
        #httpsProxy is the proxy URL for HTTPS connections 
        #to endpoints outside the cluster
        httpsProxy: http://<user>:<pwd>@<ip>:<port>
        #noProxy is the list of destination domain names, domains, 
        #IP addresses, and other network CIDRs to exclude from proxying
        #must include Supervisor Pod, Egress, Ingress CIDRs
        noProxy: [array of strings, comma-separated]

storageClass

Use the storageClass variable to configure a storage class for the cluster.

storageClass

String that is the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned.

...
    variables:
    - name: storageClass
      value: tkgs-storage-profile 

Use the following command to list available storage classes:

kubectl describe namespace VSPHERE-NAMESPACE-NAME

Or, if you have vSphere administrator privileges:

kubectl describe storageclasses

storageClasses

Use the storageClasses variable to configure an array of storage classes for the cluster.

storageClasses

Array of one or more strings, each string being the name of a vSphere storage profile that has been assigned to the vSphere Namespace where the TKG cluster is provisioned.

...
    variables:
    - name: storageClasses
      value: [tkg2-storage-profile, tkg2-storage-profile-latebinding] 

Use the following command to list available storage classes:

kubectl describe namespace VSPHERE-NAMESPACE-NAME

Or, if you have vSphere administrator privileges:

kubectl describe storageclasses

TKR_DATA

Use the TKR_DATA variable to specify TKR information.

TKR_DATA

Object that you use to specify the TKR version and other details.

The version is a string in the TKR NAME format to be used by cluster nodes.

Only TKRs that do not have the legacy-tkr label are compatible with TKG on vSphere 9 Supervisor. See Using Kubernetes Releases with TKG Service Clusters.

The default operating system is PhotonOS. Use annotations to specify the Ubuntu TKR.

trust

Use the trust variable to specify one or more trusted CA certificates for the cluster.

trust

Object for adding TLS certificates to the Cluster, either additional CAs or end certificates.

The value is additionalTrustedCAs which holds an array of strings. For example:

#trust-example
    variables:
      - name: trust
        value:
          additionalTrustedCAs:
          - name: additional-ca-1
          - name: additional-ca-2
          - name: additional-ca-N

The value of each string is the user-defined name for the data map field in the Kubernetes secret that contains the CA certificate in PEM format that is double base64-encoded. For example:

#secret-example
apiVersion: v1
data:
  additional-ca-1: TFMwdExTMUNSGlSzZ3Jaa...VVNVWkpRMEMwdExTMHRDZz09
kind: Secret
metadata:
  name: cluster01-user-trusted-ca-secret
  namespace: tkgs-cluster-ns
type: Opaque

A common use case for the trust variable is to integrate a v1beta1 Cluster with a private container registry. See Integrate TKG Service Clusters with a Private Container Registry

user

Use the user variable to specify cluster user credentials.

user

Object that includes a passwordSecret object, with name and key strings, and sshAuthorizedKey string. You can use this variable to add a user's SSH key to cluster nodes for remote SSH access.

The User variable specifies the SSH login credentials including password and authorized keys. The user name by default is vmware-system-user. The password needs to be hashed and stored in a secret in the same namespace where the Cluster is provisioned. The passwordSecret object references this secret. For example, on Linux you can generate a safe hash using mkpasswd --method=SHA-512 --rounds=4096. See Including users and groups for details.

...
    variables:
    #user specifies an authorized user and credentials
    - name: user
      value:
        #passwordSecret is an object that contains a Kubernetes secret and key
        passwordSecret:
          #name specifies the secret name
          name: string
          #key specifies the key value pair in the secret's data map
          key: string
        sshAuthorizedKey: string that is the base64-encoded public key

vmClass

Use the vmClass variable to configure the VM class for cluster nodes.

vmClass

Required string that maps to the name of a VM class that is bound to the vSphere Namespace where the TKG cluster is provisioned.

vmClass is the name of the VirtualMachineClass which describes the virtual hardware settings to be used for cluster nodes. The VirtualMachineClass controls the CPU and memory available to the node and the requests and limits on those resources. See Using VM Classes with TKG Service Clusters.

You can only use VM classes that are associated with the vSphere Namespace where the TKG Service cluster is being provisioned. Use the command kubectl get virtualmachineclass to list bound classes.

You can define the vmClass variable at different scopes so that you can use different VM classes for control plane nodes and node pool worker nodes.

For example, here an inline vmClass variable overrides the primary vmClass variable for this specific machineDeployment topology.

...
    workers:
      machineDeployments:
      - class: tkg-worker
        name: compute
        replicas: 3
        variables:
          overrides:
          - name: vmClass
            value: guaranteed-large