This topic explains where Tanzu Kubernetes (workload) cluster plan configuration values come from, and what is the order of precedence for their multiple sources. It also explains how you can customize the dev and prod plans for workload clusters on each cloud infrastructure, and how you can use ytt overlays to customize cluster plans and clusters, and create new custom plans, while preserving original configuration code.

Where Cluster Configuration Values Come From

When the tanzu CLI creates a cluster, it combines configuration values from the following:

  • Live input at invocation
    • CLI input
    • UI input, when deploying a management cluster with the installer
  • Environment variables
  • ~/.config/tanzu/tkg/cluster-config.yaml or other file passed to the CLI --file option
  • Cluster plan YAML configuration files in ~/.config/tanzu/tkg/providers, as described in Plan Configuration Files below.
  • Other, non-plan YAML configuration files under ~/.config/tanzu/tkg/providers

Live input applies configuration values that are unique to each invocation, environment variables persist them over a terminal session, and configuration files and overlays persist them indefinitely. You can customize clusters through any of these sources, with recommendations and caveats described below.

See Configuration Precedence Order for how the tanzu CLI derives specific cluster configuration values from these multiple sources where they may conflict.

Plan Configuration Files

The ~/.config/tanzu/tkg/providers directory contains workload cluster plan configuration files in the following subdirectories, based on the cloud infrastructure that deploys the clusters:

Clusters deployed by... ~/.config/tanzu/tkg/providers Directory
Management cluster on vSphere /infrastructure-vsphere
vSphere 7 Supervisor cluster /infrastructure-tkg-service-vsphere
Management cluster on Amazon EC2 /infrastructure-aws
Management cluster on Azure /infrastructure-azure

These plan configuration files are named cluster-template-definition-PLAN.yaml. The configuration values for each plan come from these files and from the files that they list under spec.paths:

  • Config files that ship with the tanzu CLI
  • Custom files that users create and add to the spec.paths list
  • ytt Overlays that users create or edit to overwrite values in other configuration files

Files to Edit, Files to Leave Alone

To customize cluster plans via YAML, you edit files under ~/.config/tanzu/tkg/providers/, but you should avoid changing other files.

Files to Edit

Workload cluster plan configuration file paths follow the form ~/.config/tanzu/tkg/providers/infrastructure-INFRASTRUCTURE/VERSION/cluster-template-definition-PLAN.yaml, where:

  • INFRASTRUCTURE is vsphere, aws, or azure.
  • VERSION is the version of the Cluster API Provider module that the configuration uses.
  • PLAN is dev, prod, or a custom plan as created in the New nginx Workload Plan example.

Each plan configuration file has a spec.paths section that lists source files and ytt directories that configure the cluster plan. For example:

spec:
  paths:
    - path: providers/infrastructure-aws/v0.5.5/ytt
    - path: providers/infrastructure-aws/ytt
    - path: providers/ytt
    - path: bom
      filemark: text-plain
    - path: providers/config_default.yaml

These files are processed in the order listed. If the same configuration field is set in multiple files, the last-processed setting is the one that the tanzu CLI uses.

To customize your cluster configuration, you can:

  • Create new configuration files and add them to the spec.paths list.
    • This is the easier method.
  • Modify existing ytt overlay files as described in ytt Overlays below.
    • This is the more powerful method, for people who are comfortable with ytt.

Files to Leave Alone

VMware discourages changing the following files under ~/.config/tanzu/tkg/providers, except as directed:

  • base-template.yaml files, in ytt directories

    • These configuration files use values from the Cluster API provider repos for vSphere, AWS, and Azure under Kubernetes SIGs, and other upstream, open-source projects, and they are best kept intact.
    • Instead, create new configuration files or see Clusters and Cluster Plans in Customizing Clusters and Plans with ytt Overlays below to set values in the overlay.yaml file in the same ytt directory.
  • ~/.config/tanzu/tkg/providers/config_default.yaml - Append only

    • This file contains system-wide defaults for Tanzu Kubernetes Grid on all cloud infrastructures.
    • Do not modify existing values in this file, but you can append a User Customizations section at the end.
    • Instead of changing values in this file, customize cluster configurations in files that you pass to the --file option of tanzu cluster create and tanzu management-cluster create.
  • ~/.config/tanzu/tkg/providers/config.yaml

    • The tanzu CLI uses this file as a reference for all providers present in the /providers directory, and their default versions.

Configuration Precedence Order

When the tanzu CLI creates a cluster, it reads in configuration values from multiple sources that may conflict. It resolves conflicts by using values in the following order of descending precedence:

Processing layers, ordered by descending precedence Source Examples Notes
8. User-specific data values, from or written to top-level config file AZURE_NODE_MACHINE_TYPE: Standard_D2s_v3

The main source of workload (and management) cluster parameters is the file passed to CLI --file option, which defaults to ~/.config/tanzu/tkg/cluster-config.yaml.

7. Factory default data values Shipped with TKG config_default.yaml These are the supported cluster template configuration "knobs", with documentation and their default settings where applicable.
6. BOM metadata data values bom-1.4.0+vmware.1.yaml One per Kubernetes version released by TKG
5 (tie). User-provided customizations Customizable ytt myhacks.yaml Topmost layer of ytt processing files before the Data Values layers; takes precedence over the layers below it
5 (tie). Additional processing YAMLs, not user-provided rm-bastion.yaml, rm-mhc.yaml, custom-resource-annotations.yaml
4. Add-on YAMLs and overlays calico.yaml, antrea.yaml A specific class of customization representing one of more resources to be applied to the cluster post-creation.
3. Plan-specific processing YAMLs prod.yaml, dev.yaml Plan-specific customizations.
2. Overlay YAML ytt/overlay.yaml Defines what in the basic template is overridable, using legacy, "KEY_NAME:value" style entries.
1. Base Cluster template YAML ytt/base-template.yaml Base CAPI template with actual default values and no ytt annotations.

ytt Overlays

Tanzu Kubernetes Grid supports customizing workload cluster configurations by adding or modifying configuration files directly, but using ytt overlays instead lets you customize configurations at different scopes and manage multiple, modular configuration files, without destructively editing upstream and inherited configuration values.

For more information, see Clusters and Cluster Plans in Customizing Clusters and Plans with ytt Overlays.

IMPORTANT: You can only use ytt overlays to modify workload clusters. Using ytt overlays to modify management clusters is not supported.

The following examples show how to use configuration overlay files to customize workload clusters and create a new cluster plan.

For an overlay that customizes cluster certificates, see Trust Custom CA Certificates on Cluster Nodes in the Manage Cluster Secrets topic.

Nameservers on vSphere

This example adds one or more custom nameservers to worker and control plane nodes in Tanzu Kubernetes Grid clusters on vSphere. It disables DNS resolution from DHCP so that the custom nameservers take precedence.

Two overlay files apply to control plane nodes, and the other two apply to worker nodes. You add all four files into your ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/ directory.

The last line of each overlay-dns file sets the nameserver addresses. The code below shows a single nameserver, but you can specify multiple nameservers as a list, for example nameservers: ["1.2.3.4","5.6.7.8"].

File vsphere-overlay-dns-control-plane.yaml:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-control-plane"}})
---
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.all, expects="1+"
        -
          #@overlay/match missing_ok=True
          nameservers: ["8.8.8.8"]

File vsphere-overlay-dhcp-control-plane.yaml:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    preKubeadmCommands:
    #! disable dns from being emitted by dhcp client
    #@overlay/append
    - echo '[DHCPv4]' >> /etc/systemd/network/10-id0.network
    #@overlay/append
    - echo 'UseDNS=no' >> /etc/systemd/network/10-id0.network
    #@overlay/append
    - '/usr/bin/systemctl restart systemd-networkd 2>/dev/null'

File vsphere-overlay-dns-workers.yaml:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-worker"}})
---
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.all, expects="1+"
        -
          #@overlay/match missing_ok=True
          nameservers: ["8.8.8.8"]

File vsphere-overlay-dhcp-workers.yaml:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"})
---
spec:
  template:
    spec:
      #@overlay/match missing_ok=True
      preKubeadmCommands:
      #! disable dns from being emitted by dhcp client
      #@overlay/append
      - echo '[DHCPv4]' >> /etc/systemd/network/10-id0.network
      #@overlay/append
      - echo 'UseDNS=no' >> /etc/systemd/network/10-id0.network
      #@overlay/append
      - '/usr/bin/systemctl restart systemd-networkd 2>/dev/null'

Resolve .local Domain

In modern Linux systems, attempts to resolve hostnames that have a domain suffix that ends in .local can fail. This issue occurs because systemd-networkd, the DNS resolver in most Linux distributions, attempts to resolve the .local domain via multi-cast DNS (mDNS), not via standard DNS servers.

To work around this known issue, add a searchDomains line with your local domain suffix at the end of the vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml files in the ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/ directory.

Example for the vsphere-overlay-dns-control-plane.yaml file:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-control-plane"}})
---
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.all, expects="1+"
        -
          #@overlay/match missing_ok=True
          nameservers: ["8.8.8.8"]
          searchDomains: ["corp.local"]

Example for the vsphere-overlay-dns-workers.yaml file:

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"VSphereMachineTemplate", "metadata": {"name": data.values.CLUSTER_NAME+"-worker"}})
---
spec:
  template:
    spec:
      network:
        devices:
        #@overlay/match by=overlay.all, expects="1+"
        -
          #@overlay/match missing_ok=True
          nameservers: ["8.8.8.8"]
          searchDomains: ["corp.local"]

Configuring NTP without DHCP Option 42 (vSphere)

TLS authentication within Tanzu Kubernetes Grid clusters requires precise time synchronization. In most DHCP-based environments, you can configure synchronization using DHCP Option 42.

If you are deploying clusters in a vSphere environment that lacks DHCP Option 42, use overlay code as follows to have Tanzu Kubernetes Grid create clusters with NTP servers that maintain synchronization:

  • In ~/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/ directory, create a new .yaml file or augment an existing overlay file with the following code, changing the example time.google.com to the desired NTP server:

    #@ load("@ytt:overlay", "overlay")
    #@ load("@ytt:data", "data")
    
    #@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
    ---
    spec:
      kubeadmConfigSpec:
        #@overlay/match missing_ok=True
        ntp:
          #@overlay/match missing_ok=True
          enabled: true
          #@overlay/match missing_ok=True
          servers:
            - time.google.com
    
    #@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"})
    ---
    spec:
      template:
        spec:
          #@overlay/match missing_ok=True
          ntp:
            #@overlay/match missing_ok=True
            enabled: true
            #@overlay/match missing_ok=True
            servers:
              - time.google.com
    

Disable Bastion Host on AWS

For an example overlay that disables the Bastion host for workload clusters on AWS, see Disable Bastion Server on AWS in the TKG Lab repository.

New Plan nginx

This example adds and configures a new workload cluster plan nginx that runs an nginx server. It uses the Cluster Resource Set (CRS) to deploy the nginx server to vSphere clusters created with the vSphere Cluster API provider version v0.7.6.

  1. In .tkg/providers/infrastructure-vsphere/v0.7.6/, add a new file cluster-template-definition-nginx.yaml with contents identical to the cluster-template-definition-dev.yaml and cluster-template-definition-prod.yaml files:

    apiVersion: run.tanzu.vmware.com/v1alpha1
    kind: TemplateDefinition
    spec:
      paths:
        - path: providers/infrastructure-vsphere/v0.7.6/ytt
        - path: providers/infrastructure-vsphere/ytt
        - path: providers/ytt
        - path: bom
          filemark: text-plain
        - path: providers/config_default.yaml
    

    The presence of this file creates a new plan, and the tanzu CLI parses its filename to create the option nginx to pass to tanzu cluster create --plan.

  2. In ~/.config/tanzu/tkg/providers/ytt/04_user_customizations/, create a new file deploy_service.yaml containing:

    #@ load("@ytt:overlay", "overlay")
    #@ load("@ytt:data", "data")
    #@ load("@ytt:yaml", "yaml")
    
    #@ def nginx_deployment():
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
    spec:
      selector:
        matchLabels:
          app: nginx
      replicas: 2
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.14.2
            ports:
            - containerPort: 80
    #@ end
    
    #@ if data.values.TKG_CLUSTER_ROLE == "workload" and data.values.CLUSTER_PLAN == "nginx":
    
    ---
    apiVersion: addons.cluster.x-k8s.io/v1alpha3
    kind: ClusterResourceSet
    metadata:
      name: #@ "{}-nginx-deployment".format(data.values.CLUSTER_NAME)
      labels:
        cluster.x-k8s.io/cluster-name: #@ data.values.CLUSTER_NAME
    spec:
      strategy: "ApplyOnce"
      clusterSelector:
        matchLabels:
          tkg.tanzu.vmware.com/cluster-name: #@ data.values.CLUSTER_NAME
      resources:
      - name: #@ "{}-nginx-deployment".format(data.values.CLUSTER_NAME)
        kind: ConfigMap
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: #@ "{}-nginx-deployment".format(data.values.CLUSTER_NAME)
    type: addons.cluster.x-k8s.io/resource-set
    stringData:
      value: #@ yaml.encode(nginx_deployment())
    
    #@ end
    

    In this file, the conditional #@ if data.values.TKG_CLUSTER_ROLE == "workload" and data.values.CLUSTER_PLAN == "nginx": applies the overlay that follows to workload clusters with the plan nginx.

    If the 04_user_customizations directory does not already exist under the top-level ytt directory, create it.

check-circle-line exclamation-circle-line close-line
Scroll to top icon