Customizing Clusters, Plans, and Packages with ytt Overlays

This topic explains how you can use ytt Overlays to customize Tanzu Kubernetes (workload) clusters, cluster plans, and Packages.

Tanzu Kubernetes Grid distributes configuration files for clusters and cluster plans in the ~/.config/tanzu/tkg/providers/ directory.

You can customize these configurations by adding or modifying configuration files directly, or by using ytt overlays.

Directly customizing configuration files is simpler, but if you are comfortable with ytt overlays, they let you customize configurations at different scopes and manage multiple, modular configuration files, without destructively editing upstream and inherited configuration values.

ytt Behavior and Conventions

Behaviors and conventions for ytt processing include:

Precedence: ytt traverses directories depth-first in filename alphabetical order, and overwrites duplicate settings as it proceeds. When there are duplicate definitions, the one that ytt processes last takes precedence.

Overlay Types: different ytt overlay types change or set different things:

• Data values files set or modify configuration values without modifying the structures of objects. These include Bill of Materials (BoM) files and, by convention, files with data in their filenames.

• Overlay files make changes to object structure definitions. By convention, these files have overlay in their filenames.

For more information about ytt, including overlay examples and an interactive validator tool, see:

• Carvel Tools > ytt > Interactive Playground
• The IT Hollow: Using YTT to Customize TKG Deployments

Clusters and Cluster Plans

The ~/.config/tanzu/tkg/providers/ directory includes ytt directories and overlay.yaml files at different levels, which lets you scope configuration settings at each level.

• Provider- and version-specific ytt directories. For example, ~/.config/tanzu/tkg/providers/infrastructure-aws/v0.6.4/ytt.
  • Specific configurations for provider API version.
  • The base-template.yaml file contains all-caps placeholders such as "${CLUSTER_NAME}" and should not be edited.
  • The overlay.yaml file is tailored to overlay values into base-template.yaml.
• Provider-wide ytt directories. For example, ~/.config/tanzu/tkg/providers/infrastructure-aws/ytt.
  • Provider-wide configurations that apply to all versions.
• Top-level ytt directory, ~/.config/tanzu/tkg/providers/ytt.
  • Cross-provider configurations.
  • Organized into numbered directories, and processed in number order.
  • You can create a /04_user_customizations subdirectory for configurations that take precedence over any in lower-numbered ytt subdirectories.

Important: You can only use ytt overlays to modify workload clusters. Using ytt overlays to modify management clusters is not supported.

Cluster and Plan Examples

Examples of ytt overlays for customizing workload clusters and cluster plans include:

• Nameservers on vSphere
• FQDN Control Plane Endpoint with NSX ALB
• Trust Custom CA Certificates on Cluster Nodes
• Resolve .local Domain
• Disable Bastion Server on AWS in the TKG Lab repository
• New nginx Workload Plan

Applying Overlays to Packages

You can apply or migrate overlays to the user managed packages.

Note: Tanzu package plugin does not natively support annotation. However, Carvel APIs expose an annotation that you can set on PackageInstall CR. Tanzu package plugin preserves this annotation to update package.​

To apply an overlay:

1. Update the package configuration values to meet your requirements.

2. Install the package.

   tanzu package install my-pkg –p contour.tanzu.vmware.com -v 1.15.1+vmware.1-tkg.1 --values-file my-values.yaml​

   Here is an example of values.yaml file:

   ```
   infrastructure_provider: vsphere
   namespace: tanzu-system-ingress
   contour:
     configFileContents: {}
     useProxyProtocol: false
     replicas: 2
     pspNames: "vmware-system-restricted"
     logLevel: info
   envoy:
     service:
       type: LoadBalancer
       annotations: {}
       nodePorts:
         http: null
         https: null
       externalTrafficPolicy: Cluster
       aws:
         LBType: classic
       disableWait: false
     hostPorts:
       enable: true
       http: 80
       https: 443
     hostNetwork: false
     terminationGracePeriodSeconds: 300
     logLevel: info
     pspNames: null
   certificates:
     duration: 8760h
     renewBefore: 360h
   ```
   

3. Create a secret for the overlay. For example:

   kubectl create secret generic my-overlay --from-file=my-overlay.yaml​

4. Annotate PackageInstall.

   kubectl annotate PackageInstall my-pkg ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=my-overlay