Multiple Kubernetes Versions

This topic explains how to list available Kubernetes versions and deploy a workload cluster with a non-default Kubernetes version.

About Kubernetes Versions

Each version of Tanzu Kubernetes Grid with a standalone management cluster provides a default and two non-default versions of Kubernetes, which you can list by running tanzu kubernetes-release get.

Tanzu Kubernetes Grid manages Kubernetes versions with Tanzu Kubernetes release (TKr) objects. A TKr specifies the OS images, core Kubernetes components, and bootstrap packages that are compatible with the Kubernetes version that is defined in the TKr. When you run tanzu cluster create with a default or a non-default TKr, Tanzu Kubernetes Grid uses the components and bootstrap packages specified in the TKr to create the cluster. It also reads your cluster configuration file to determine which of the compatible OS images to use when creating the cluster.

For a full list of supported Kubernetes versions, see Supported Kubernetes Versions in Tanzu Kubernetes Grid v2.3 in VMware Tanzu Kubernetes Grid 2.3 Release Notes.

List Available Versions

To list all available Kubernetes releases with their current compatibility and upgrade status, run tanzu kubernetes-release get with an optional version match argument, for example:

To list all releases, run tanzu kubernetes-release get.
To list all releases matching v1.24.17, run tanzu kubernetes-release get v1.24.17.

tanzu kubernetes-release get
NAME                       VERSION                  COMPATIBLE  ACTIVE  UPDATES AVAILABLE
v1.24.17---vmware.1-tkg.1  v1.24.17+vmware.1-tkg.1  True        True
v1.25.13---vmware.1-tkg.1  v1.25.13+vmware.1-tkg.1  True        True
v1.26.8---vmware.2-tkg.1   v1.26.8+vmware.1-tkg.1   True        True

List Available Upgrades

To discover the TKr versions that are available for a specific workload cluster, run tanzu cluster available-upgrades get with the full name of the cluster, for example:

tanzu cluster available-upgrades get my-cluster

Activate or Deactivate a Tanzu Kubernetes Release

A TKr can be activated or deactivated. To activate a TKr:

tanzu kubernetes-release activate TKR-NAME

For example:

tanzu kubernetes-release activate v1.25.13---vmware.1-tkg.1

To deactivate a TKr:

tanzu kubernetes-release deactivate TKR-NAME

For example:

tanzu kubernetes-release deactivate v1.25.13---vmware.1-tkg.1

Deploy a Cluster with a Non-Default Kubernetes Version

Each release of Tanzu Kubernetes Grid provides a default version of Kubernetes. The default version for Tanzu Kubernetes Grid v2.3 is Kubernetes v1.26.8.

As upstream Kubernetes releases patches or new versions, VMware publishes them in a public registry and the Tanzu Kubernetes release controller imports them into the management cluster. This lets the tanzu CLI create clusters based on the new versions.

To list available Kubernetes versions, see Available Kubernetes Versions above.
To deploy clusters that run a non-default version of Kubernetes, follow the steps below.

Publish the Kubernetes Version to your Infrastructure

On vSphere and Azure, you need to take an additional step before you can deploy clusters that run non-default versions of Kubernetes:

vSphere: Import the appropriate base image template OVA file into vSphere and convert it to a VM template. For information about importing base OVA files into vSphere, see Import the Base Image Template into vSphere.
Azure: Run the Azure CLI command to accept the license for the base OS version. Once you have accepted a license, you can skip this step in the future:
1. Convert your target Kubernetes version listed in the output of the tanzu kubernetes-release get command into its Azure image SKU as follows:
  - Change leading v to k8s-.
  - Change . to dot in the version number.
  - Change trailing +vmware.* to -ubuntu-2004, to designate Ubuntu v20.04, the default OS version for all Tanzu Kubernetes Grid VMs on Azure.
  - Examples: k8s-1dot26dot8-ubuntu-2004, k8s-1dot25dot13-ubuntu-2004.
2. Run az vm image terms accept. For example:
```
az vm image terms accept --publisher vmware-inc --offer tkg-capi-2022-06-24 --plan k8s-1dot26dot8-ubuntu-2004
```
Amazon Web Services (AWS): No action required. The Amazon Linux 2 Amazon Machine Images (AMI) that include the supported Kubernetes versions are publicly available to all AWS users, in all supported AWS regions. Tanzu Kubernetes Grid automatically uses the appropriate AMI for the Kubernetes version that you specify.

Deploy the Kubernetes Cluster

To deploy the workload cluster running a non-default Kubernetes version:

If you are deploying a plan-based cluster, set an environment variable ALLOW_LEGACY_CLUSTER to true:
```
export ALLOW_LEGACY_CLUSTER=true
```
If the Kubernetes patch version that you are deploying the cluster with is older than the latest supported patch for its minor version as listed in Supported Kubernetes Versions create a ConfigMap object for its TKr in the tkg-system namespace:
1. Create an ConfigMap object definition in the tkg-system namespace that lists the older TKrs:
```
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: tkg-system
  name: TKR-CONFIG-NAME
  labels:
    run.tanzu.vmware.com/additional-compatible-tkrs: ""
data:
  tkrVersions: |
      - TKR-VERSON-1
      - TKR-VERSON-2
```
  Where TKR-CONFIG-NAME is any legal object name, such as tkg-v2.3.1, and the TKR-VERSON-* values are the older TKr versions as listed by tanzu kubernetes-release get, for example v1.24.14--vmware.1-tkg.
2. Create the ConfigMap object:
```
kubectl apply -f old-tkrs-config.yaml
```
To deploy a workload cluster with a version of Kubernetes that is not the default for your version of Tanzu Kubernetes Grid, specify the Tanzu Kubernetes release in the --tkr option. For example, to deploy a Kubernetes v1.25.13 cluster, run:
```
tanzu cluster create my-1-25-13-cluster --tkr v1.25.13---vmware.1-tkg
```

For more details on how to create a workload cluster, see Create Workload Clusters.

Deploy a Cluster with a Custom Machine Image

For common combinations of OS version, Kubernetes version, and target infrastructure, Tanzu Kubernetes Grid with a standalone management cluster provides default machine images. You can optionally specify custom machine images, including images that you build yourself.

The sections below explain how to deploy a cluster with an alternative or custom-built machine image:

Use an Alternative Machine Image

In a cluster configuration file you set OS_NAME, OS_VERSION, and OS_ARCH to specify the base machine image, for example ubuntu, 20.04 or photon, 3; and amd64. These default to OS images that VMware distributes, but you can also specify your own alternative OS image to use for cluster nodes:

Upload the image to vSphere as described in Import the Base Image Template into vSphere.
Do one of the following, depending on whether your image shares the same OS version as an existing machine image distributed by VMware:
- If you are specifying an alternative image with the same OS name, version, and architecture as an existing TKr and image that VMware distributes, edit its OSImage object, for example:
```
kubectl edit OSImage v1.26.4---vmware.1-tkg.1-6c92aa1cbb96b644085e6229f41ef14d
```
- Otherwise, create and apply a new OSImage object, using an existing OSImage object definition as a template. Set its metadata.name and spec.image.ref.version values to the OVA filename.
In the OSImage object’s spec.image.ref block, add an identifier field and value. For example, to enable an alternative to the Kubernetes v1.26.4 on Ubuntu 20.04 image that VMware distributes:
```
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: OSImage
metadata:
  creationTimestamp: "2023-06-02T06:08:35Z"
  generation: 2
  labels:
    image-type: ova
    os-arch: amd64
    os-name: ubuntu
    os-type: linux
    ...
spec:
  image:
    ref:
      MY-FIELD: MY-VALUE
      version: v1.26.4---vmware.1-tkg.1-6c92aa1cbb96b644085e6229f41ef14d
    type: ova
  kubernetesVersion: v1.26.4+vmware.1
  os:
    arch: amd64
    name: ubuntu
    type: linux
    version: "2004"
```
Where MY-FIELD and MY-VALUE are the identifier field and value that you set to include in Cluster object specs.
- MY-VALUE must be 0-63 characters long and can be empty, or else it must start and end with alphanumeric characters and contain alphanumerics, dashes (-), underscores (_), or dots (.).
- TKG copies these values into the OSImage object’s metadata.labels settings following the pattern ova-MY-FIELD: MY-VALUE.
To use the alternative machine image in a cluster, edit the Cluster object to add the setting ova-MY-FIELD=MY-VALUE to the metadata.annotations for the cluster nodes that will use the image.
- For control plane nodes, edit the metadata.annotations under topology.controlPlane.
- For worker nodes, edit the metadata.annotations under topology.workers.machineDeployments for the machine deployments that will use the image.
- For example:
```
metadata:
  annotations:
    run.tanzu.vmware.com/resolve-os-image: image-type=ova,ova-my-field=my-value
```

Build and Use a Custom Machine Image

You can build your own machine image to use for cluster nodes. Reasons to do this include:

To create clusters on a base OS that VMware supports but does not distribute, such as Red Hat Enterprise Linux (RHEL) v8.
To install additional packages into the base machine image, or otherwise customize it as described in Customization in the Image Builder documentation.

For instructions, see Build Machine Images.