Configuration Files and Object Specs

This topic explains how to configure Tanzu Kubernetes Grid (TKG) workload clusters using a flat configuration file or a Kubernetes-style object spec. For IaaS-specific instructions, see:

Cluster Types

How you configure a workload cluster depends on the cluster type, as described in the sections below.

(Default type) Class-based clusters use a Kubernetes-style object spec for the Cluster object. To create a class-based cluster, you pass either this object spec or a cluster configuration file with a flat structure that sets uppercase-underscore variables like CLUSTER_NAME to tanzu cluster create.
- More advanced customizations may also require a specification for ClusterBootstrap and objects that it references.
(Legacy types) Plan-based and TKC clusters use a cluster configuration file with a flat structure that sets uppercase-underscore variables like CLUSTER_NAME. To create a legacy cluster, you pass this configuration file to tanzu cluster create.
- More advanced customizations may require ytt Overlays to customize workload clusters or cluster plans. For information about how to download and install ytt, see Install the Carvel Tools.

For more information about the above cluster types and the Cluster API providers that they use, see:

Workload Cluster Types
Cluster Types and Cluster API
Ways of Creating Workload Clusters, which includes the Tanzu CLI, Tanzu Mission Control, and kubectl.

For information about which configuration file to choose for your workload cluster, see the table below.

Configuration Files

You can use a ... with Tanzu CLI	to create workload clusters of type...	on infrastructure...
Flat cluster configuration file	Class-based clusters Plan-based clusters TKC clusters	Class-based clusters on vSphere 7 and 8 with a standalone management cluster; AWS; and Azure Plan-based clusters on vSphere 7 and 8 with a standalone management cluster; AWS; and Azure TKC clusters on vSphere 8 with Supervisor
Kubernetes-style object spec	Class-based clusters	vSphere 8 with Supervisor; vSphere 7 and 8 with a standalone management cluster; AWS; and Azure

Configuring a Workload Cluster with a Cluster Configuration File

You can use a cluster configuration file to configure the following types of clusters:

Class-based clusters: This includes deployments to:
- vSphere 7 and 8 with a standalone management cluster
- AWS
- Azure
If you want to deploy a class-based workload to vSphere 8 with Supervisor, you must deploy it from an object spec, as described in Configure a Supervisor-Deployed Class-Based Cluster (vSphere 8 only).
(Legacy) Plan-based and TKC clusters: This includes deployments to:
- For plan-based clusters: vSphere 7 and 8 with a standalone management cluster, AWS, and Azure
- For TKC clusters: vSphere 8 with Supervisor

About the Cluster Configuration File

Before deploying a workload cluster to vSphere, AWS, or Azure, you create a configuration file for the cluster. When you pass this file to the --file option of tanzu cluster create, the Tanzu CLI uses the configuration information defined in the file to connect to your target platform and create the resources that the cluster will use.

To create a cluster configuration file, you can copy an existing configuration file for a previous deployment and update it. Alternatively, you can create a file from scratch by using an empty template:

To set the configuration information relevant to your infrastructure, see:
For the full list of options that you can specify in the cluster configuration file, see Configuration File Variable Reference.

Creating Your First Cluster Configuration File

When you deploy a workload cluster, most of the configuration for the cluster is the same as the configuration of the standalone management cluster that you use to deploy it. Because of this, the easiest way to create a configuration file for a workload cluster is to start with a copy of the standalone management cluster configuration file.

Locate the YAML configuration file for the management cluster:
- If you deployed the management cluster from the installer interface and you did not specify the --file option when you ran tanzu mc create --ui, the configuration file is saved in ~/.config/tanzu/tkg/clusterconfigs/. The file has a randomly generated name, for example, bm8xk9bv1v.yaml.
- If you deployed the management cluster from the installer interface and you did specify the --file option, the management cluster configuration is taken from the file that you specified.
- If you deployed the management cluster from the Tanzu CLI without using the installer interface, the management cluster configuration is taken from either a file that you specified in the --file option, or from the default location, ~/.config/tanzu/tkg/cluster-config.yaml.
Make a copy of the management cluster configuration file and save it with a new name. For example, save the file as my-aws-cluster.yaml, my-azure-cluster.yaml, or my-vsphere-cluster.yaml.
Update the settings in the cluster configuration file. If you want to configure a workload cluster to use an OS other than the default Ubuntu 20.04, you must set the OS_NAME and OS_VERSION values. The installer interface does not include node VM OS values in the management cluster configuration files that it saves to ~/.config/tanzu/tkg/clusterconfigs.
Save the file.

To prepare a cluster configuration file for a TKC cluster on vSphere 8 with Supervisor, see Configure a Supervisor-Deployed TKC Cluster (Legacy).

Configuring a Workload Cluster With an Object Spec

You can use a Kubernetes-style object spec file to deploy a class-based workload cluster to:

vSphere 7 and 8 without Supervisor
vSphere 8 with Supervisor
AWS
Azure

About the Object Spec File

An object spec file serves the same purpose as a cluster configuration file. When you pass an object spec to the --file option of tanzu cluster create, the Tanzu CLI creates the cluster using the configuration information defined in the object spec.

Creating Your First Object Spec File

As with other Kubernetes objects, you can configure a class-based workload cluster by creating and editing object specs. For cluster objects, these are:

Cluster object: Configures most cluster options, such as cluster topology. You can change and re-apply this object spec in order to change the configuration of a running cluster.
(Optional; vSphere with Tanzu) ClusterBootstrap object: Configures non-default options that are only applied when the cluster is first created, such as the container network interface (CNI) and container storage interface (CSI), and which cannot be re-configured in a running cluster. For more information, see Configure One-Time Infrastructure Settings.

To create your first object spec file:

If you are deploying a class-based workload cluster to vSphere without Supervisor, AWS or Azure, you can use the Tanzu CLI to convert a cluster configuration file into an object spec file without deploying the cluster:
- (Global configuration) Use the auto-apply-generated-clusterclass-based-configuration feature of the Tanzu CLI. This feature is set to false by default. When auto-apply-generated-clusterclass-based-configuration is set to false and you run tanzu cluster create with the --file flag, the command converts your cluster configuration file into an object spec file and exits without creating the cluster. After reviewing the configuration, you re-run tanzu cluster create with the object spec file generated by the Tanzu CLI, as described in Create a Class-Based Cluster from the Object Spec. If you have updated the default configuration, to set it back to false, run:
```
tanzu config set features.cluster.auto-apply-generated-clusterclass-based-configuration false
```
- (Single cluster) Pass the --dry-run option to tanzu cluster create and save the output to a file. Use the same options and configuration --file that you would use if you were creating the cluster, for example:
```
tanzu cluster create my-cluster --file my-cluster-config.yaml --dry-run > my-cluster-spec.yaml
```
  You can then use this object spec to deploy a cluster as described in Create a Class-Based Cluster from the Object Spec.
If you are deploying a class-based workload cluster to vSphere 8 with Supervisor, create or adapt a Cluster object spec as described in Configuring a Supervisor-Deployed Class-Based Cluster (vSphere 8 only). The vSphere 8 documentation has example Cluster object specs to work from, for example, v1beta1 Example: Default Cluster.

Configuring One-Time Infrastructure Settings (vSphere with Tanzu)

You can configure most common cluster settings in the Cluster object spec, but some cluster components come from the Tanzu Kubernetes release (TKr) that the cluster’s nodes are based on. For example, the TKr specifies the CNI, CSI, and Pinniped versions that a cluster uses.

These infrastructure-level options are applied at cluster creation time by the ClusterBootstrap object and cannot be changed in a running cluster. The general procedure for configuring these options before creating a cluster is:

Create object specs for the ClusterBootstrap object and any custom objects that it references, for example, a CalicoConfig object.
Create an object spec for the Cluster object itself.
In the metadata for all of the object specs, include the name and namespace for the cluster that you are creating, for example:
```
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: ClusterBootstrap
metadata:
  name: MY-CLUSTER
  namespace: MY-NAMESPACE
```
The customization may require additional metadata, such as annotations.
Concatenate all of the object specs, including the Cluster definition, into a single file. Separate the specs with lines consisting of three dashes (---).

Pass the object spec to the --file option of kubectl apply, for example:

kubectl config use-context CONTEXT-NAME

kubectl apply -f my-custom-cluster-objects.yaml

Alternative method: As an alternative to the last two steps, you can run kubectl apply -f on all of the object specs except for the Cluster object and then run tanzu cluster create --file with the Cluster object spec.

For a specific example of this procedure, see Create a Cluster with a Non-Default CNI, which sets a cluster’s CNI to Calico.

Workload Clusters, `kubectl`, and `kubeconfig`

Tanzu Kubernetes Grid does not automatically set the kubectl context to a workload cluster when you create it. You must set the kubectl context to a workload cluster manually by using the kubectl config use-context command.

By default, unless you specify the KUBECONFIG option to save the kubeconfig for a cluster to a specific file, all workload clusters that you deploy are added to a shared .kube/config file. If you delete the shared .kube/config file and you still have the .kube-tkg/config file for the management cluster, you can recover the .kube/config of the workload clusters with the tanzu cluster kubeconfig get CLUSTER-NAME command.

Do not change context or edit the .kube-tkg/config or .kube/config files while Tanzu Kubernetes Grid operations are running.

What to Do Next

Proceed to the cluster configuration topic for your infrastructure: