Configuration File to Cluster Class Variable Translations

This topic explains how variables in workload cluster configuration files translate to variable settings within a class-based Cluster objects and their ClusterBootstrap and other subordinate objects in Tanzu Kubernetes Grid (TKG). For an example Cluster object and its subordinate objects, see Example Cluster Object and Its Subordinate Objects below.

Overview: Configuration File and Cluster Class Variables

Configuration file variables and class-based cluster variables differ:

  • Configuration file variables are:

  • Class-based cluster variable settings are nested YAML settings within Cluster or ClusterBootstrap object specs:

    • Cluster object settings are:

      • Most of the cluster settings available, and can be changed arbitrarily.
      • Defined by the ClusterClass object set as the cluster’s spec.topology.class value.
      • Set in the controlPlane, workers or variables blocks under spec.topology.
      • Example setting:

        variables:
        - name: imageRepository
          value:
            host: stg-project.vmware.com
        
    • ClusterBootstrap settings are a few one-time settings for container networking and other low-level infrastructure that cannot be changed in an existing cluster.

When you run the tanzu cluster create command with a cluster configuration file as described in Create a Class-Based Cluster, the Tanzu CLI performs this translation.

Node Configuration

The table below lists variables for configuring the control plane and worker nodes, and the operating system that the node instances run. For information about the variables in the Configuration File Variables column, see Node Configuration in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables. The example code in this column applies to Cluster objects on vSphere; object structure on AWS and Azure may differ.

Configuration File Variables Class-based Cluster Object Structure
OS_NAME: ubuntu
OS_VERSION: 20.04
OS_ARCH: amd64
metadata:
  annotations:
    osInfo: ubuntu,20.04,amd64
    
CONTROL_PLANE_MACHINE_COUNT: 3
    
controlPlane:
  ...
  replicas: 3
    
CONTROLPLANE_SIZE: large
SIZE: large
Values translate to machine settings based on infrastructure.
SIZE applies to both control plane and workers.
- name: controlPlane
  value:
    machine:
      diskGiB: 40
      memoryMiB: 16384
      numCPUs: 4
    
CONTROL_PLANE_NODE_SEARCH_DOMAINS: corp.local, example.com
    
- name: controlPlane
  value:
    ...
    network:
      ...
      searchDomains:
      - corp.local
      - example.com
    
CONTROL_PLANE_NODE_LABELS: 'key1=value1,key2=value2'
    
- name: controlPlane
  value:
    ...
    nodeLabels:
    - key: key1
      value: value1
    - key: key2
      value: value2
    
WORKER_MACHINE_COUNT: 6
    
- name: worker
  value:
    count: 6
    
WORKER_SIZE: extra-large
SIZE: extra-large
Values translate to machine settings based on infrastructure.
SIZE applies to both control plane and workers.
- name: worker
  value:
    ...
    machine:
      diskGiB: 80
      memoryMiB: 32768
      numCPUs: 8
    
WORKER_NODE_SEARCH_DOMAINS: corp.local, example.com
- name: worker
  value:
    ...
    network:
      searchDomains:
      - corp.local
      - example.com
    
CUSTOM_TDNF_REPOSITORY_CERTIFICATE: "YPdeNjLW[...]"
- name: customTDNFRepository
  value:
    certificate: YPdeNjLW[...]
    

Cluster Autoscaler

When ENABLE_AUTOSCALER is true, the Tanzu CLI creates a Deployment object for Cluster Autoscaler and adds Cluster Autoscaler annotations to the Cluster object. For information about the variables in the Configuration File Variables column below, see Cluster Autoscaler in Configuration File Variable Reference.

Configuration File Variables Class-based Cluster and Autoscaler Deployment Object Structure
AUTOSCALER_SCALE_DOWN_DELAY_AFTER_ADD: 10m
AUTOSCALER_SCALE_DOWN_DELAY_AFTER_DELETE: 10s
AUTOSCALER_SCALE_DOWN_DELAY_AFTER_FAILURE: 3m
AUTOSCALER_SCALE_DOWN_UNNEEDED_TIME: 10m
AUTOSCALER_MAX_NODE_PROVISION_TIME: 15m
AUTOSCALER_MAX_NODES_TOTAL: 0
    
In Cluster Autoscaler Deployment object:
spec:
  containers:
  - args:
  ...
  - --scale-down-delay-after-add=10m
  - --scale-down-delay-after-delete=10s
  - --scale-down-delay-after-failure=3m
  - --scale-down-unneeded-time=10m
  - --max-node-provision-time=15m
  - --max-nodes-total=0
AUTOSCALER_MAX_SIZE_0: 12
AUTOSCALER_MIN_SIZE_0: 8
AUTOSCALER_MAX_SIZE_1: 10
AUTOSCALER_MIN_SIZE_1: 6
AUTOSCALER_MAX_SIZE_2: 8
AUTOSCALER_MIN_SIZE_2: 4
    
If AUTOSCALER_MAX_SIZE_* or AUTOSCALER_MIN_SIZE_* are not set, the annotations settings take the values of WORKER_MACHINE_COUNT_*.
In the Cluster object, under spec.topology.workers:
machineDeployments:
- class: tkg-worker
  metadata:
    annotations:
      cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "12"
      cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "8"
      ...
  name: md-0
- class: tkg-worker
  metadata:
    annotations:
      cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "10"
      cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "6"
      ...
  name: md-1
...

Proxy and Private Image Registry

This section lists variables for using proxies and private image registries, for example in an internet-restricted deployment.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure Notes
TKG_CUSTOM_IMAGE_REPOSITORY: example.com/yourproject
TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY: true
- name: imageRepository
  value:
    host: example.com/yourproject
    tlsCertificateValidation:
      enabled: false
tldCertificateValidation.enabled reverses the boolean setting for TKG_CUSTOM_IMAGE_REPOSITORY_SKIP_TLS_VERIFY and is not written if its value is true.
TKG_HTTP_PROXY_ENABLED: true
TKG_HTTP_PROXY: http://proxy.example.com:80
TKG_HTTPS_PROXY: https://proxy.example.com:3128
TKG_NO_PROXY: .noproxy.example.com,noproxy.example.com,192.168.0.0/24
- name: proxy
  value:
    httpProxy: http://proxy.example.com:80
    httpsProxy: https://proxy.example.com:3128
    noProxy:
    - .noproxy.example.com
    - noproxy.example.com
    - 192.168.0.0/24
    - [...]
Internally, the Tanzu CLI appends to the noProxy list values that are not set in a configuration file, as described in [Proxy Configuration](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/2.2/tkg-deploy-mc/mgmt-deploy-config-ref.html#proxies).
TKG_PROXY_CA_CERT: "LS0tLSBL[...]"
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE: “MIIEpgIH[...]"
- name: trust
  value:
    - name: proxy
      data: LS0tLSBL[...]
    - name: imageRepository
      data: MIIEpgIH[...]
Values are base64-encoded CA certificates.
TKG_PROXY_CA_CERT takes precedence over TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE, and trust imageRespository value is not written if trust proxy value is set.
ADDITIONAL_IMAGE_REGISTRY_1: "example.com/otherregistry-1"
ADDITIONAL_IMAGE_REGISTRY_1_SKIP_TLS_VERIFY: false
ADDITIONAL_IMAGE_REGISTRY_1_CA_CERTIFICATE: "LS0tLSBL[...]"
ADDITIONAL_IMAGE_REGISTRY_2: "example.com/otherregistry-2"
ADDITIONAL_IMAGE_REGISTRY_2_SKIP_TLS_VERIFY: true
ADDITIONAL_IMAGE_REGISTRY_3: "example.com/otherregistry-3"
ADDITIONAL_IMAGE_REGISTRY_3_SKIP_TLS_VERIFY: false
ADDITIONAL_IMAGE_REGISTRY_3_CA_CERTIFICATE: "MIIEpgIH[...]"
- name: additionalImageRegistries
  value:
  - caCert: LS0tLSBL[...]
    host: example.com/otherregistry-1
    skipTlsVerify: false
  - host: example.com/otherregistry-2
    skipTlsVerify: true
  - caCert: MIIEpgIH[...]
    host: example.com/otherregistry-3
    skipTlsVerify: false

Common Variables

The following table lists variables that are common to all target platforms.

For information about the variables in the Configuration File Variables column, see Common Variables for All Target Platforms in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure
CLUSTER_NAME: my-cluster
NAMESPACE: default
metadata:
  name: my-cluster
  namespace: default
CLUSTER_PLAN: dev
metadata:
  annotations:
    tkg/plan: dev
INFRASTRUCTURE_PROVIDER: vsphere
topology:
    class: tkg-vsphere-default-v1.0.0
    
CLUSTER_API_SERVER_PORT: 6443
- name: apiServerPort
  value: 6443
CNI: antrea
- name: cni
  value: antrea
ENABLE_AUDIT_LOGGING: true
- name: auditLogging
      value:
        enabled: true
CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 100.96.0.0/11
    services:
      cidrBlocks:
      - 100.64.0.0/13
CONTROLPLANE_CERTIFICATE_ROTATION_ENABLED: true
CONTROLPLANE_CERTIFICATE_ROTATION_DAYS_BEFORE: 65
- name: controlPlaneCertificateRotation
  value:
    activate: true
    daysBefore: 65

Antrea CNI

This section lists variables for configuring a cluster’s Antrea container network interface (CNI). The class-based cluster variables reside in the AntreaConfig object that is referenced under the ClusterBootstrap object’s spec.cni.refName property.

Class-based clusters support many Antrea configuration options that do not have corresponding TKG configuration file variables. For all Antrea configuration options that are settable for class-based Cluster objects, see Feature Gates and other topics in the Antrea documentation.

In the AntreaConfig Object Structure column, all settings are located in the AntreaConfig object definition, under spec.antrea.config.

Configuration File Variables AntreaConfig Object Structure
ANTREA_DISABLE_UDP_TUNNEL_OFFLOAD: false
ANTREA_ENABLE_USAGE_REPORTING: false
ANTREA_KUBE_APISERVER_OVERRIDE: "https://192.168.77.100:6443"
ANTREA_NO_SNAT: false
ANTREA_TRAFFIC_ENCAP_MODE: "encap"
ANTREA_TRANSPORT_INTERFACE: "eth0"
ANTREA_TRANSPORT_INTERFACE_CIDRS: "10.0.0.2/24"
disableUdpTunnelOffload: false
enableUsageReporting: false
kubeAPIServerOverride: https://192.168.77.100:6443
noSNAT: false
trafficEncapMode: encap
transportInterface: eth0
transportInterfaceCIDRs:
- 10.0.0.2/24
ANTREA_EGRESS: true
ANTREA_IPAM: false
ANTREA_MULTICAST: false
ANTREA_NETWORKPOLICY_STATS: true
ANTREA_NODEPORTLOCAL: true
ANTREA_SERVICE_EXTERNALIP: false
ANTREA_POLICY: true
ANTREA_TRACEFLOW: true
featureGates:
  Egress: true
  AntreaIPAM: false
  Multicast: false
  NetworkPolicyStats: true
  NodePortLocal: true
  ServiceExternalIP: false
  AntreaPolicy: true
  AntreaTraceflow: true
ANTREA_PROXY: true
ANTREA_PROXY_ALL: false
ANTREA_PROXY_LOAD_BALANCER_IPS: true
ANTREA_PROXY_NODEPORT_ADDRS: "100.70.70.12"
ANTREA_PROXY_SKIP_SERVICES: 10.11.1.2,kube-system/kube-dns
antreaProxy:
  nodePortAddresses:
  - 100.70.70.12
  proxyAll: false
  proxyLoadBalancerIPs: true
  skipServices:
  - 10.11.1.2
  - kube-system/kube-dns
ANTREA_FLOWEXPORTER: false
ANTREA_FLOWEXPORTER_ACTIVE_TIMEOUT: "60s"
ANTREA_FLOWEXPORTER_COLLECTOR_ADDRESS: "flow-aggregator.svc:4739:tls"
ANTREA_FLOWEXPORTER_POLL_INTERVAL: "5s"
ANTREA_FLOWEXPORTER_IDLE_TIMEOUT: "15s"
flowExporter:
  activeFlowTimeout: 60s
  collectorAddress: flow-aggregator.svc:4739:tls
  idleFlowTimeout: 15s
  pollInterval: 5s
ANTREA_NODEPORTLOCAL_ENABLED: true
ANTREA_NODEPORTLOCAL_PORTRANGE: 61000-62000
nodePortLocal:
  enabled: true
  portRange: 61000-62000
ANTREA_EGRESS_EXCEPT_CIDRS: "10.0.0.0/6"
egress:
  exceptCIDRs:
  - 10.0.0.0/6
ANTREA_MULTICAST_INTERFACES: "eth0"
multicastInterfaces:
- eth0

vSphere

This section lists variables for deploying workload clusters to vSphere. For information about the variables in the Configuration File Variables column below, see vSphere in Configuration File Variable Reference. Settings listed in the Class-based Object Structure column are in the objects of the kind listed.

Configuration File Variables Object Structure
TKG_IP_FAMILY: ipv4
VSPHERE_CONTROL_PLANE_ENDPOINT: 10.237.177.161
VSPHERE_REGION: my-region
VSPHERE_ZONE: my-zone
kind: VSphereCPIConfig
spec:
  vsphereCPI:
    ipFamily: ipv4
    vmNetwork:
      excludeExternalSubnetCidr: 10.237.177.161/32
      excludeInternalSubnetCidr: 10.237.177.161/32
    zone: my-zone

kind: Cluster
metadata:
  annotations:
    tkg.tanzu.vmware.com/cluster-controlplane-endpoint: 10.237.177.161
spec:
  topology:
    variables:
    - name: apiServerEndpoint
      value: 10.237.177.161
VSPHERE_DATACENTER: /dc0
VSPHERE_DATASTORE: /dc0/datastore/sharedVmfs-0
VSPHERE_FOLDER: /dc0/vm/folder0
VSPHERE_NETWORK: /dc0/network/VM Network
VSPHERE_RESOURCE_POOL: /dc0/host/cluster0/Resources/rp0
VSPHERE_SERVER: 10.237.179
VSPHERE_STORAGE_POLICY_ID: my-local-sp
VSPHERE_TEMPLATE: /dc0/vm/ubuntu-2004-kube-v1.25.7+vmware.2-tkg.1
VSPHERE_TLS_THUMBPRINT: B7:15:2D:D8:E5:FC:BE:C6:85:7A:03:98:0E:43:7E:59:A9:CF:1D:2F
kind: ClusterBootstrap
spec:
  additionalPackages:
  - refName: tkg-storageclass*
    valuesFrom:
      inline:
        VSPHERE_STORAGE_POLICY_ID: my-local-sp

kind: Cluster
spec:
  topology:
    variables:
    - name: vcenter
      value:
        datacenter: /dc0
        datastore: /dc0/datastore/sharedVmfs-0
        folder: /dc0/vm/folder0
        network: /dc0/network/VM Network
        resourcePool: /dc0/host/cluster0/Resources/rp0
        server: 10.237.179.190
        storagePolicyID: my-local-sp
        template: /dc0/vm/ubuntu-2004-kube-v1.25.7+vmware.2-tkg.1
        tlsThumbprint: B7:15:2D:D8:E5:FC:BE:C6:85:7A:03:98:0E:43:7E:59:A9:CF:1D:2F
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa NzaC1yc2EA[...]==
kind: Cluster
spec:
  topology:
    variables:
    - name: user
      value:
        sshAuthorizedKeys:
        - ssh-rsa NzaC1yc2EA[...]==
VSPHERE_CONTROL_PLANE_DISK_GIB: "30"
VSPHERE_CONTROL_PLANE_MEM_MIB: "2048"
VSPHERE_CONTROL_PLANE_NUM_CPUS: "2"
VSPHERE_WORKER_DISK_GIB: "50"
VSPHERE_WORKER_MEM_MIB: "4096"
VSPHERE_WORKER_NUM_CPUS: "4"
kind: Cluster
spec:
  topology:
    variables:
    - name: controlPlane
      value:
        machine:
          diskGiB: 40
          memoryMiB: 8192
          numCPUs: 2
    - name: worker
      value:
        count: 1
        machine:
          diskGiB: 40
          memoryMiB: 8192
          numCPUs: 2
NTP_SERVERS: time.google.com
kind: Cluster
spec:
  topology:
    variables:
    - name: ntpServers
      value:
      - time.google.com
VSPHERE_AZ_0: rack1
VSPHERE_AZ_1: rack2
VSPHERE_AZ_2: rack3
kind: Cluster
spec:
  topology:
    workers:
      machineDeployments:
      - class: tkg-worker
        failureDomain: rack1
        name: md-0
      - class: tkg-worker
        failureDomain: rack2
        name: md-1
      - class: tkg-worker
        failureDomain: rack3
        name: md-2

AWS

This section lists variables for deploying workload clusters to AWS. For information about the variables in the Configuration File Variables column below, see AWS in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_PROFILE
    
N/A
These settings are not stored in the Cluster object.
AWS_REGION: us-east-1
    
- name: region
  value: us-east-1
    
AWS_SSH_KEY_NAME: aws-tkg-clusteradmin
    
- name: sshKeyName
  value: aws-tkg-clusteradmin
    
AWS_LOAD_BALANCER_SCHEME_INTERNAL: false
    
- name: loadBalancerSchemeInternal
  value: false
BASTION_HOST_ENABLED: true
    
- name: bastion
  value:
    enabled: true
    
AWS_NODE_AZ: us-east-1a
AWS_PRIVATE_NODE_CIDR: 10.0.0.0/24
AWS_PRIVATE_SUBNET_ID: subnet-0a7d376dde53c77ed
AWS_PUBLIC_NODE_CIDR: 10.0.1.0/24
AWS_PUBLIC_SUBNET_ID: subnet-0794d50f57e9801b6
AWS_NODE_AZ_1: us-west-2b
AWS_PRIVATE_NODE_CIDR_1: 10.0.2.0/24
AWS_PRIVATE_SUBNET_ID_1: subnet-0c338780824d1c59d
AWS_PUBLIC_NODE_CIDR_1: 10.0.3.0/24
AWS_PUBLIC_SUBNET_ID_1: subnet-0addabd635d02ba97
AWS_NODE_AZ_2: ap-southeast-3
AWS_PRIVATE_NODE_CIDR_2: 10.0.4.0/24
AWS_PRIVATE_SUBNET_ID_2: subnet-00b9638e419a6187b
AWS_PUBLIC_NODE_CIDR_2: 10.0.5.0/24
AWS_PUBLIC_SUBNET_ID_2: subnet-0ed174ef16a2f43aa
- name: network
  value:
    subnets:
    - az: us-east-1a
      private:
        cidr: 10.0.0.0/24
        id: subnet-0a7d376dde53c77ed
      public:
        cidr: 10.0.1.0/24
        id: subnet-0794d50f57e9801b6
    - az: us-west-2b
      private:
        cidr: 10.0.2.0/24
        id: subnet-0c338780824d1c59d
      public:
        cidr: 10.0.3.0/24
        id: subnet-0addabd635d02ba97
    - az: ap-southeast-3
      private:
        cidr: 10.0.4.0/24
        id: subnet-00b9638e419a6187b
      public:
        cidr: 10.0.5.0/24
        id: subnet-0ed174ef16a2f43aa
    
AWS_VPC_CIDR: 10.0.0.0/16
    
- name: network
  value:
    ...
    vpc:
      cidr: 10.0.0.0/16
    
AWS_VPC_ID: vpc-0ce8bdfea218
    
- name: network
  value:
    ...
    vpc:
      existingID: vpc-0ce8bdfea218
    
AWS_SECURITY_GROUP_BASTION: sg-1
AWS_SECURITY_GROUP_APISERVER_LB: sg-2
AWS_SECURITY_GROUP_LB: sg-3
AWS_SECURITY_GROUP_CONTROLPLANE: sg-4
AWS_SECURITY_GROUP_NODE: sg-5
    
- name: network
  value:
    ...
    securityGroupOverrides:
      bastion: sg-1
      apiServerLB: sg-2
      lb: sg-3
      controlPlane: sg-4
      node: sg-5
    
AWS_IDENTITY_REF_NAME: my-aws-id
AWS_IDENTITY_REF_KIND: AWSClusterRoleIdentity
    
- name: identityRef
  value:
    name: my-aws-id
    kind: AWSClusterRoleIdentity
    
NODE_MACHINE_TYPE: m5.large
AWS_NODE_OS_DISK_SIZE_GIB: 80
    
In a multi-AZ deployment, you can also set NODE_MACHINE_TYPE_1 and NODE_MACHINE_TYPE_2
In the Cluster object, under spec.topology.workers:
machineDeployments:
- class: tkg-worker
  name: md-0
  value:
    instanceType: m5.large
    rootVolume:
      sizeGiB: 80
    
CONTROL_PLANE_MACHINE_TYPE: t3.large
AWS_CONTROL_PLANE_OS_DISK_SIZE_GIB: 80
    
- name: controlPlane
  value:
    instanceType: t3.large
    rootVolume:
      sizeGiB: 80
    

Microsoft Azure

This section lists variables for deploying workload clusters to Microsoft Azure. For information about the variables in the Configuration File Variables column below, see Microsoft Azure in Configuration File Variable Reference.

In the Class-based Cluster Object Structure column, all name / value pair settings are located in the Cluster object definition under spec.topology.variables.

Configuration File Variables Class-based Cluster Object Structure
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID
N/A
These settings are not exposed in the Cluster object.
AZURE_LOCATION: eastus2
- name: location
  value: eastus2
AZURE_RESOURCE_GROUP: my-azure-rg
- name: resourceGroup
  value: my-azure-rg
AZURE_SUBSCRIPTION_ID: c789uce3-aaaa-bbbb-cccc-a51b6b0gb405
- name: subscriptionID
  value: c789uce3-aaaa-bbbb-cccc-a51b6b0gb405
AZURE_ENVIRONMENT: AzurePublicCloud
- name: environment
  value: AzurePublicCloud
AZURE_SSH_PUBLIC_KEY_B64: "c3NoLXJzYSBB [...] vdGFsLmlv"
- name: sshPublicKey
  value: c3NoLXJzYSBB [...] vdGFsLmlv
		
AZURE_FRONTEND_PRIVATE_IP: 10.0.0.100
Set if AZURE_ENABLE_PRIVATE_CLUSTER is true.
- name: frontendPrivateIP
  value: 10.0.0.100
    
AZURE_CUSTOM_TAGS: "foo=bar, plan=prod"
- name: customTags
  value: "foo=bar, plan=prod"
    
AZURE_ENABLE_ACCELERATED_NETWORKING: true
- name: acceleratedNetworking
  value:
    enabled: true
    
AZURE_ENABLE_PRIVATE_CLUSTER: false
- name: privateCluster
  value:
    enabled: false
    
AZURE_VNET_CIDR: 10.0.0.0/16
AZURE_VNET_NAME: my-azure-vnet
AZURE_VNET_RESOURCE_GROUP: my-azure-vnet-rg
- name: network
  value:
    vnet:
      cidrBlocks:
      - 10.0.0.0/16
      name: my-azure-vnet
      resourceGroup: my-azure-vnet-rg
AZURE_IDENTITY_NAME: my-azure-id
AZURE_IDENTITY_NAMESPACE: default
    
- name: identityRef
  value:
    name: my-azure-id
    namespace: default  
AZURE_CONTROL_PLANE_DATA_DISK_SIZE_GIB: 256
AZURE_CONTROL_PLANE_OS_DISK_SIZE_GIB: 128
AZURE_CONTROL_PLANE_OS_DISK_STORAGE_ACCOUNT_TYPE: Premium_LRS
AZURE_ENABLE_CONTROL_PLANE_OUTBOUND_LB: true
AZURE_CONTROL_PLANE_OUTBOUND_LB_FRONTEND_IP_COUNT: 1
AZURE_CONTROL_PLANE_SUBNET_CIDR: 10.0.0.0/24
AZURE_CONTROL_PLANE_SUBNET_NAME: my-azure-cp-subnet
AZURE_CONTROL_PLANE_MACHINE_TYPE: Standard_D2s_v3
    
- name: controlPlane
  value:
    dataDisks:
    - sizeGiB: 256
    osDisk:
      sizeGiB: 128
      storageAccountType: Premium_LRS
    outboundLB:
      enabled: true
      frontendIPCount: 1
    subnet:
      cidr: 10.0.0.0/24
      name: my-azure-cp-subnet
    vmSize: Standard_D2s_v3
    
AZURE_ENABLE_NODE_DATA_DISK: true
AZURE_NODE_DATA_DISK_SIZE_GIB: 256
AZURE_NODE_OS_DISK_SIZE_GIB: 128
AZURE_NODE_OS_DISK_STORAGE_ACCOUNT_TYPE: Premium_LRS
AZURE_ENABLE_NODE_OUTBOUND_LB: true
AZURE_NODE_OUTBOUND_LB_FRONTEND_IP_COUNT: 1
AZURE_NODE_OUTBOUND_LB_IDLE_TIMEOUT_IN_MINUTES: 4
AZURE_NODE_SUBNET_CIDR: 10.0.1.0/24
AZURE_NODE_SUBNET_NAME: my-azure-worker-subnet
AZURE_NODE_MACHINE_TYPE: Standard_D2s_v3
    
- name: worker
  value:
    dataDisks:
    - sizeGiB: 256
    osDisk:
      sizeGiB: 128
      storageAccountType: Premium_LRS
    outboundLB:
      enabled: true
      frontendIPCount: 1
      idleTimeoutInMinutes: 4
    subnet:
      cidr: 10.0.1.0/24
      name: my-azure-worker-subnet
    vmSize: Standard_D2s_v3
    

NSX Advanced Load Balancer

This section lists variables for configuring NSX Advanced Load Balancer (ALB) in TKG.

Configuration File Variables Class-based Cluster Object Structure
AVI_CONTROL_PLANE_HA_PROVIDER
topology:
  variables:
    - name: aviAPIServerHAProvider
      value: true
    

Example Cluster Object and Its Subordinate Objects

When you pass a cluster configuration file to the --file flag of tanzu cluster create, the command converts your cluster configuration file into a cluster spec file. See below for an example Cluster object and its subordinate objects generated by tanzu cluster create from the following configuration file:

CLUSTER_NAME: example-cluster
CLUSTER_PLAN: dev
NAMESPACE: default
CNI: antrea
VSPHERE_NETWORK: /dc0/network/VM Network
VSPHERE_SSH_AUTHORIZED_KEY: ssh-rsa AAAACCCza2EBBB[...]ADGQAg/POl6vyWOmQ==
VSPHERE_USERNAME: Administrator@vsphere.local
VSPHERE_PASSWORD: 1234567AbC!
VSPHERE_SERVER: 10.XXX.XXX.71
VSPHERE_DATACENTER: /dc0
VSPHERE_RESOURCE_POOL: /dc0/host/cluster0/Resources/example-tkg
VSPHERE_DATASTORE: /dc0/datastore/vsanDatastore
VSPHERE_FOLDER: /dc0/vm/example-tkg
VSPHERE_TLS_THUMBPRINT: ""
VSPHERE_INSECURE: true
VSPHERE_CONTROL_PLANE_ENDPOINT: 10.XXX.XXX.75
AVI_CONTROL_PLANE_HA_PROVIDER: false
ENABLE_AUDIT_LOGGING: false
ENABLE_DEFAULT_STORAGE_CLASS: true
CLUSTER_CIDR: 100.96.0.0/11
SERVICE_CIDR: 100.64.0.0/13
ENABLE_AUTOSCALER: false

Resulting Cluster object and its subordinate objects:

apiVersion: cpi.tanzu.vmware.com/v1alpha1
kind: VSphereCPIConfig
metadata:
  name: example-cluster
  namespace: default
spec:
  vsphereCPI:
    mode: vsphereCPI
    tlsCipherSuites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    vmNetwork:
      excludeExternalSubnetCidr: 10.XXX.XXX.75/32
      excludeInternalSubnetCidr: 10.XXX.XXX.75/32
---
apiVersion: csi.tanzu.vmware.com/v1alpha1
kind: VSphereCSIConfig
metadata:
  name: example-cluster
  namespace: default
spec:
  vsphereCSI:
    config:
      datacenter: /dc0
      httpProxy: ""
      httpsProxy: ""
      insecureFlag: true
      noProxy: ""
      region: null
      tlsThumbprint: ""
      useTopologyCategories: false
      zone: null
    mode: vsphereCSI
---
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: ClusterBootstrap
metadata:
  annotations:
    tkg.tanzu.vmware.com/add-missing-fields-from-tkr: TKR-NAME
  name: example-cluster
  namespace: default
spec:
  additionalPackages:
  - refName: metrics-server*
  - refName: secretgen-controller*
  - refName: pinniped*
  cpi:
    refName: vsphere-cpi*
    valuesFrom:
      providerRef:
        apiGroup: cpi.tanzu.vmware.com
        kind: VSphereCPIConfig
        name: example-cluster
  csi:
    refName: vsphere-csi*
    valuesFrom:
      providerRef:
        apiGroup: csi.tanzu.vmware.com
        kind: VSphereCSIConfig
        name: example-cluster
  kapp:
    refName: kapp-controller*
---
apiVersion: v1
kind: Secret
metadata:
  name: example-cluster
  namespace: default
stringData:
  password: 1234567AbC!
  username: Administrator@vsphere.local
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  annotations:
    osInfo: ubuntu,20.04,amd64
    tkg.tanzu.vmware.com/cluster-controlplane-endpoint: 10.XXX.XXX.75
    tkg/plan: dev
  labels:
    tkg.tanzu.vmware.com/cluster-name: example-cluster
  name: example-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 100.96.0.0/11
    services:
      cidrBlocks:
      - 100.64.0.0/13
  topology:
    class: tkg-vsphere-default-CLUSTER-CLASS-VERSION
    controlPlane:
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=ubuntu
      replicas: 1
    variables:
    - name: cni
      value: antrea
    - name: controlPlaneCertificateRotation
      value:
        activate: true
        daysBefore: 90
    - name: auditLogging
      value:
        enabled: false
    - name: podSecurityStandard
      value:
        audit: restricted
        deactivated: false
        warn: restricted
    - name: apiServerEndpoint
      value: 10.XXX.XXX.75
    - name: aviAPIServerHAProvider
      value: false
    - name: vcenter
      value:
        cloneMode: fullClone
        datacenter: /dc0
        datastore: /dc0/datastore/vsanDatastore
        folder: /dc0/vm/example-tkg
        network: /dc0/network/VM Network
        resourcePool: /dc0/host/cluster0/Resources/example-tkg
        server: 10.XXX.XXX.71
        storagePolicyID: ""
        tlsThumbprint: ""
    - name: user
      value:
        sshAuthorizedKeys:
        - ssh-rsa AAAACCCza2EBBB[...]ADGQAg/POl6vyWOmQ==
    - name: controlPlane
      value:
        machine:
          diskGiB: 40
          memoryMiB: 8192
          numCPUs: 2
    - name: worker
      value:
        machine:
          diskGiB: 40
          memoryMiB: 4096
          numCPUs: 2
    - name: security
      value:
        fileIntegrityMonitoring:
          enabled: false
        imagePolicy:
          pullAlways: false
          webhook:
            enabled: false
            spec:
              allowTTL: 50
              defaultAllow: true
              denyTTL: 60
              retryBackoff: 500
        kubeletOptions:
          eventQPS: 50
          streamConnectionIdleTimeout: 4h0m0s
        systemCryptoPolicy: default
    version: KUBERNETES-VERSION
    workers:
      machineDeployments:
      - class: tkg-worker
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=ubuntu
        name: md-0
        replicas: 1
        strategy:
          type: RollingUpdate

The example object spec file above includes the following placeholder text:

  • TKR-NAME in tkg.tanzu.vmware.com/add-missing-fields-from-tkr: TKR-NAME: Set by tanzu cluster create to a compatible Tanzu Kubernetes release (TKr), depending on your configuration. For example, tkg.tanzu.vmware.com/add-missing-fields-from-tkr: v1.25.7---vmware.2-tkg.1. v1.25.7---vmware.2-tkg.1 is the default TKr for this Tanzu Kubernetes Grid release.
  • KUBERNETES-VERSION in version: KUBERNETES-VERSION: Set by tanzu cluster create to a compatible Kubernetes version, depending on your configuration. For example, version: v1.25.7+vmware.2-tkg.1. v1.25.7+vmware.2-tkg.1 is the default Kubernetes version for this Tanzu Kubernetes Grid release.
  • CLUSTER-CLASS-VERSION in class: tkg-vsphere-default-CLUSTER-CLASS-VERSION: Is the version of the default class, which is set by tanzu cluster create. For example, class: tkg-vsphere-default-v1.0.0.
check-circle-line exclamation-circle-line close-line
Scroll to top icon