A Tanzu Kubernetes release provides the Kubernetes software distribution signed and supported by VMware for use with TKG clusters on Supervisor. The TKr format is updated for TKG to support packages and multiple operating systems.

TKr Release Notes

Refer to the Tanzu Kubernetes releases Release Notes for the complete list of available TKrs, what's new for each release, known issues, and TKr compatability with Supervisor and TKG. See also vSphere with Tanzu Upgrade, Support and Kubernetes Versions.

TKr Distribution and Consumption

VMware distributes Tanzu Kubernetes releases through a content delivery network. You use a vSphere content library to associate TKrs with vSphere Namespaces. To automate the consumption of TKrs, use a subscribed content library. For internet-restricted environments, use a local content library.

Each Tanzu Kubernetes release is distributed as an OVA template. The TKr Controller on Supervisor uses the OVA template to construct the VMs for TKG cluster nodes. The VM disk size is set by the TKr OVA template. You specify CPU and RAM resources using VM classes.

The TKr content library is not namespace-scoped. All vSphere Namespaces use same TKr content library for TKG with Supervisor. Editing the TKr content library for one vSphere Namespace will update it for all others.

TKr Name String

Use the command kubectl get tkr to list the TKr images that are available for use in the vSphere Namespace. For example:
kubectl get tkr
NAME                                  VERSION                             READY   COMPATIBLE   CREATED
v1.16.14---vmware.1-tkg.1.ada4837     v1.16.14+vmware.1-tkg.1.ada4837     False   False        19d
v1.17.17---vmware.1-tkg.1.d44d45a     v1.17.17+vmware.1-tkg.1.d44d45a     False   False        19d
v1.18.19---vmware.1-tkg.1.17af790     v1.18.19+vmware.1-tkg.1.17af790     False   False        19d
v1.19.16---vmware.1-tkg.1.df910e2     v1.19.16+vmware.1-tkg.1.df910e2     False   False        19d
v1.20.12---vmware.1-tkg.1.b9a42f3     v1.20.12+vmware.1-tkg.1.b9a42f3     False   False        19d
v1.21.6---vmware.1-tkg.1.b3d708a      v1.21.6+vmware.1-tkg.1.b3d708a      True    True         19d
v1.22.9---vmware.1-tkg.1.cc71bc8      v1.22.9+vmware.1-tkg.1.cc71bc8      True    True         19d
v1.23.8---vmware.2-tkg.2-zshippable   v1.23.8+vmware.2-tkg.2-zshippable   True    True         19d
v1.23.8---vmware.3-tkg.1              v1.23.8+vmware.3-tkg.1              True    True         19d
You use the TKr NAME string to provision TKG clusters on Supervisor. If you are using the v1alpha3 API, you provide full TKr NAME string in the tkr.reference.name field. If you are using the v1beta1 API, you provide the full TKr NAME string in the topology.version field.
Note: Do not use the VERSION string when referencing the TKr in the cluster specification. The format must match the TKr NAME string exactly.

The name of the TKr in the content library needs to be the full TKr NAME string. If you are using a subscribed content library, the TKr Name string is created for you. If you are using a local content library, make sure the name you give to the TKr matches the TKr NAME string. See local content library for details.

TKr Compatibility with Supervisor

TKrs are released and updated independent from Supervisor.

To provision a TKG cluster, the TKr must be compatible with the TKG Controller component that runs on Supervisor. You cannot use a TKr that is not compatible with the Supervisor instance you are running. In addition, you must make sure that you a running a compatible TKr for the version of Supervisor you target for upgrade.

You can check TKr compatibility with Supervisor using the command kubectl get tkr. The COMPATIBLE column returns a boolean. True means the TKr is compatible with Supervisor, False means the TKr is not compatible with Supervisor.

TKr Compatibility with TKG 2.x

TKG 2.x is the latest release of the Tanzu Kubernetes Grid with new functionality, including provisioning APIs. TKG 2.0 was introduced with vSphere 8. See Provisioning TKG Clusters on Supervisor for more information.

The TKr format is different for TKG 2.x clusters than for legacy TKG clusters supported on vSphere 7. To provision a TKG 2.x cluster, you must use a non-legacy TKr. If the TKr is a legacy TKr, it can only be used to provision a vSphere 7 TKG cluster. TKr images that are not compatible with TKG 2.x include the legacy-tkr annotation label.

You may be able to provision a vSphere 7 TKG cluster on a compatible Supervisor, but you cannot take advantage of TKG 2.x features unless you use a non-legacy TKr. For example, while the command kubectl get tkr may indicate that the TKr is compatible with Supervisor, if it is a legacy TKr it can only be used to provision a vSphere 7 TKG cluster.

You can check TKr compatibility with TKG 2.x using the commands kubectl get tkr -o yaml and kubectl get tkr --show-labels. If the annotation label legacy-tkr is present, the TKr cannot be used to provision a TKG 2.x cluster.

For example, the following command shows that the specified image is a legacy-tkr. This means it cannot be used to provision a TKG 2.x cluster. This TKr can only be used to provision a legacy TKG cluster.
kubectl get tkr v1.23.8---vmware.3-tkg.1 -o yaml
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesRelease
metadata:
  creationTimestamp: "2023-03-15T20:33:17Z"
  finalizers:
  - tanzukubernetesrelease.run.tanzu.vmware.com
  generation: 1
  labels:
    os-arch: amd64
    os-name: photon
    os-type: linux
    os-version: "3.0"
    run.tanzu.vmware.com/legacy-tkr: ""
    v1: ""
    v1.23: ""
    v1.23.8: ""
    v1.23.8---vmware: ""
    v1.23.8---vmware.3: ""
    v1.23.8---vmware.3-tkg: ""
    v1.23.8---vmware.3-tkg.1: ""
  name: v1.23.8---vmware.3-tkg.1
The following example uses the --show-labels flag to check if the TKr is compatible for provisioning TKG 2.x clusters. The label legacy-tkr is present so the image can only be used to create a legacy TKG cluster.
kubectl get tkr v1.23.8---vmware.3-tkg.1 --show-labels
NAME                       VERSION                  READY   COMPATIBLE   CREATED   LABELS
v1.23.8---vmware.3-tkg.1   v1.23.8+vmware.3-tkg.1   True    True         19d       os-arch=amd64,os-name=photon,os-type=linux,os-version=3.0,run.tanzu.vmware.com/legacy-tkr=,
The following example shows a TKr that is compatible for TKG 2.x cluster creation because the label legacy-tkr is absent from the list of labels.
kubectl get tkr v1.23.8---vmware.2-tkg.2-zshippable --show-labels
NAME                                  VERSION                             READY   COMPATIBLE   CREATED   LABELS
v1.23.8---vmware.2-tkg.2-zshippable   v1.23.8+vmware.2-tkg.2-zshippable   True    True         19d       os-arch=amd64,os-name=ubuntu,os-type=linux,os-version=20.04,v1.23.8---vmware.2-tkg.2-zshippable=,
Running the equivalent command using -o yaml also shows that the legacy-tkr label is absent, indicating the TKr can be used to provision a TKG 2.x cluster.
kubectl get tkr v1.23.8---vmware.2-tkg.2-zshippable -o yaml
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesRelease
metadata:
  creationTimestamp: "2023-03-15T20:31:45Z"
  finalizers:
  - tanzukubernetesrelease.run.tanzu.vmware.com
  generation: 2
  labels:
    os-arch: amd64
    os-name: ubuntu
    os-type: linux
    os-version: "20.04"
    v1: ""
    v1.23: ""
    v1.23.8: ""
    v1.23.8---vmware: ""
    v1.23.8---vmware.2: ""
    v1.23.8---vmware.2-tkg: ""
    v1.23.8---vmware.2-tkg.2: ""
    v1.23.8---vmware.2-tkg.2-zshippable: ""
  name: v1.23.8---vmware.2-tkg.2-zshippable
Important: The -zshippable suffix is not a reliable method for determining if the TKr is compatible with TKG 2.x. This suffix is removed from the TKr NAME string starting with the vSphere 8 U1 release. You must check the TKr for the presence of the legacy-tkr label to determine if the TKr can be used with vSphere 8 to provision a TKG 2.x cluster. If you have already provisioned a TKG 2.x cluster using a TKr with the -zshippable suffix, you need to change the content library URL to a temporary location during the transition away from the use of this suffix. Refer to the Tanzu Kubernetes releases Release Notes for details.

TKr OSImage Format

The TKr OSImage format supports multiple operating system images for a single TKr. This means that there is a single Tanzu Kubernetes release for a specific Kubernetes version for all supported operating systems, which are currently PhotonOS and Ubuntu. The default OSImage is PhotonOS.

By default the PhotonOS edition of the named TKr is used for TKG cluster nodes. If the referenced TKr supports the OSImage format and has an Ubuntu OS edition available, use the run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu annotation to specify the Ubuntu OS edition of the TKr. For example, the following TKG cluster specification uses the Ubuntu edition of the named TKr.
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: tkg2-cluster-ubuntu
  namespace: tkg2-cluster-ns
  annotations:
    run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
spec:
   topology:
     controlPlane:
       ...
       tkr:
         reference:
           name: v1.23.8---vmware.2-tkg.2-zshippable
The OSImage format supports heterogeneous cluster deployments. For example, the following cluster manifest creates a Tanzu Kubernetes cluster with the default PhotonOS for the control plane and Ubuntu for the worker nodes. The TKr version is reference in the control plane and the annotation specifies Ubuntu for the named worker node pool.
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:    
  name: tkg2-cluster-multiOS
  namespace: tkg2-cluster-ubuntu
  annotations:
    //Worker's annotation
    run.tanzu.vmware.com/resolve-os-image.np-1: os-name=ubuntu
spec:    
  topology:      
    controlPlane:        
      tkr:          
        reference:            
          name: v1.23.8---vmware.2-tkg.2-zshippable
      replicas: 3
      vmClass: guaranteed-medium
      storageClass: tkg2-storage-profile 
    nodePools:     
    - replicas: 3
      name: np-1
      vmClass: guaranteed-medium
      storageClass: tkg2-storage-profile
When vSphere with Tanzu is upgraded to TKG 2 on vSphere 8 Supervisor, existing TKrs are automatically converted to the TKr OSImage format with reference to a single OSImage. This enables legacy TKrs to be compatible with TKG 2. If you are using the Ubuntu edition of the TKr and you only specify the short version string ( v1.21.6+vmware.1-tkg.1) rather than the full version string ( ubuntu-2004-v1.21.6---vmware.1-tkg.1), you must include the OS annotation in the cluster spec. This is true if you are using the v1beta1 API, v1alpha3 API, or the vlapha2 API. For example:
apiVersion: run.tanzu.vmware.com/v1alpha2
kind: TanzuKubernetesCluster
metadata:    
  name: tkg-cluster-ubuntu
  namespace: tkg-cluster-ubuntu
  annotations: 
    run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
spec:    
  topology:      
    controlPlane:        
      replicas: 3
      vmClass: guaranteed-medium
      storageClass: tkg-storage-policy
      tkr:          
        reference:            
          name: v1.21.6+vmware.1-tkg.1
    nodePools:     
    - name: worker-nodepool
      replicas: 3
      vmClass: guaranteed-medium
      storageClass: tkg2-storage-policy
      tkr:          
        reference:            
          name: v1.21.6+vmware.1-tkg.1
  settings:
    storage:
      defaultClass: tkg-storage-policy

TKr Packages

TKr images compatible with vSphere 8 are updated to a package-based framework for TKG 2.0 components, such as the container storage interface (CSI) and container network interface (CNI). If you are using the v1beta API, changing or updating these components is done using custom resource definitions.

To view the packages that comprising a TKr, run the following command:
kubectl get tkr TKR-NAME -o yaml
For example:
kubectl get tkr v1.23.8---vmware.2-tkg.2-zshippable -o yaml
The command returns all the packages in the TKr. For example:
spec:
bootstrapPackages:
- name: antrea.tanzu.vmware.com.1.5.3+tkg.2-zshippable
- name: vsphere-pv-csi.tanzu.vmware.com.2.6.0+vmware.1-tkg.1-zshippable
- name: vsphere-cpi.tanzu.vmware.com.1.23.1+vmware.1-tkg.2-zshippable
- name: kapp-controller.tanzu.vmware.com.0.38.4+vmware.1-tkg.2-zshippable
- name: guest-cluster-auth-service.tanzu.vmware.com.1.0.0+tkg.2-zshippable
- name: metrics-server.tanzu.vmware.com.0.6.1+vmware.1-tkg.2-zshippable
- name: secretgen-controller.tanzu.vmware.com.0.9.1+vmware.1-tkg.2-zshippable
- name: pinniped.tanzu.vmware.com.0.12.1+vmware.2-tkg.2-zshippable
- name: capabilities.tanzu.vmware.com.0.26.0-dev-68-g09198af6+vmware.1
- name: calico.tanzu.vmware.com.3.22.1+vmware.1-tkg.2-zshippable

See v1beta1 Example: Cluster with Calico CNI for an example use case.

Migrating TKr OS Types

In-place cluster updates between TKr operating systems are not supported. This means, for example, that you cannot upgrade a TKG cluster using TKr version 1.21 Photon to TKr version 1.22 Ubuntu.

If you want to change the TKr OS type that a TKG cluster is using, consider the following procedure. In this example, the origin cluster is using TKr Photon and the target is TKr Ubuntu.

TKr Hardening

Security Technical Implementation Guides (STIG) for vSphere with Tanzu 8 components, including Supervisor and TKrs, are available. See Tanzu STIG Hardening for details.

Build Your Own TKr

Starting with a specific release of TKr, you can build custom TKr machine images for TKG 2.x clusters running on vSphere 8 Supervisor. A custom machine image packages a supported operating system and version, a Kubernetes version based on a released TKr, and any customizations you make.

To build custom machine images for TKG 2.x cluster nodes, use the vSphere Tanzu Kubernetes Grid Image Builder. Refer to the documentation for details on building custom images, supported TKr versions, and supported customizations.