A Tanzu Kubernetes release (TKr) provides the Kubernetes software distribution signed and supported by VMware for use with TKG Service clusters. The TKr format is updated for vSphere 8 to support packages and multiple operating systems.

TKr Release Notes

Refer to the Tanzu Kubernetes releases Release Notes for the complete list of available TKrs, what's new for each release, known issues, and TKr compatibility.

TKr Distribution and Consumption

VMware distributes Tanzu Kubernetes releases through a content delivery network. You use a vSphere content library to associate TKrs with vSphere Namespaces. To automate the consumption of TKrs, use a subscribed content library. For internet-restricted environments, use a local content library.

Each Tanzu Kubernetes release is distributed as an OVA template. The TKr Controller on Supervisor uses the OVA template to construct the VMs for TKG cluster nodes. The VM disk size is set by the TKr OVA template. You specify CPU and RAM resources using VM classes.

The TKr content library is not namespace-scoped. All vSphere Namespaces use same TKr content library for TKG with Supervisor. Editing the TKr content library for one vSphere Namespace will update it for all others.

TKr Name String

Use the command kubectl get tkr to list the TKr images that are available for use in the vSphere Namespace. For example: 
kubectl get tkr
NAME                                  VERSION                             READY   COMPATIBLE   CREATED
v1.16.14---vmware.1-tkg.1.ada4837     v1.16.14+vmware.1-tkg.1.ada4837     False   False        19d
v1.17.17---vmware.1-tkg.1.d44d45a     v1.17.17+vmware.1-tkg.1.d44d45a     False   False        19d
v1.18.19---vmware.1-tkg.1.17af790     v1.18.19+vmware.1-tkg.1.17af790     False   False        19d
v1.19.16---vmware.1-tkg.1.df910e2     v1.19.16+vmware.1-tkg.1.df910e2     False   False        19d
v1.20.12---vmware.1-tkg.1.b9a42f3     v1.20.12+vmware.1-tkg.1.b9a42f3     False   False        19d
v1.21.6---vmware.1-tkg.1.b3d708a      v1.21.6+vmware.1-tkg.1.b3d708a      True    True         19d
v1.22.9---vmware.1-tkg.1.cc71bc8      v1.22.9+vmware.1-tkg.1.cc71bc8      True    True         19d
v1.23.8---vmware.2-tkg.2-zshippable   v1.23.8+vmware.2-tkg.2-zshippable   True    True         19d
v1.23.8---vmware.3-tkg.1              v1.23.8+vmware.3-tkg.1              True    True         19d
You use the TKr NAME string to provision TKG clusters on Supervisor. If you are using the v1alpha3 API, you provide full TKr NAME string in the tkr.reference.name field. If you are using the v1beta1 API, you provide the full TKr NAME string in the topology.version field.
Note: Do not use the VERSION string when referencing the TKr in the cluster specification. The format must match the TKr NAME string exactly.

The name of the TKr in the content library needs to be the full TKr NAME string. If you are using a subscribed content library, the TKr Name string is created for you. If you are using a local content library, make sure the name you give to the TKr matches the TKr NAME string. See local content library for details.

TKr Compatibility with TKG Service

TKrs are released and updated independent from the TKG Service and from Supervisor.

To provision a TKG cluster, the TKr must be compatible with the TKG Service. You cannot use a TKr that is not compatible with the TKG Service. In addition, you must make sure that you a running a compatible TKr for the version you target for upgrade.

You can check TKr compatibility using the command kubectl get tkr. The COMPATIBLE column returns a boolean. True means the TKr is compatible, False means the TKr is not compatible.

TKr Compatibility with vSphere

The TKr format is updated for vSphere 8. TKrs for vSphere 8 can only run on vSphere 8.x. TKrs for vSphere 7.x are legacy images that work with vSphere 7. Such images can run on vSphere 8 but for upgrade purposes only. Legacy TKr images are identified by the legacy-tkr annotation label.

You can check TKr compatibility using the commands kubectl get tkr -o yaml and kubectl get tkr --show-labels. If the annotation label legacy-tkr is present, the TKr does not support vSphere 8 features and should only be used to upgrade to vSphere 8 from vSphere 7.

For example, the following command shows that the specified image is a legacy-tkr. 
kubectl get tkr v1.23.8---vmware.3-tkg.1 -o yaml
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesRelease
metadata:
  creationTimestamp: "2023-03-15T20:33:17Z"
  finalizers:
  - tanzukubernetesrelease.run.tanzu.vmware.com
  generation: 1
  labels:
    os-arch: amd64
    os-name: photon
    os-type: linux
    os-version: "3.0"
    run.tanzu.vmware.com/legacy-tkr: ""
    v1: ""
    v1.23: ""
    v1.23.8: ""
    v1.23.8---vmware: ""
    v1.23.8---vmware.3: ""
    v1.23.8---vmware.3-tkg: ""
    v1.23.8---vmware.3-tkg.1: ""
  name: v1.23.8---vmware.3-tkg.1
The following example uses the --show-labels flag to check TKr compatibility. The label legacy-tkr is present so the image can only be used to create a legacy TKG cluster. 
kubectl get tkr v1.23.8---vmware.3-tkg.1 --show-labels
NAME                       VERSION                  READY   COMPATIBLE   CREATED   LABELS
v1.23.8---vmware.3-tkg.1   v1.23.8+vmware.3-tkg.1   True    True         19d       os-arch=amd64,os-name=photon,os-type=linux,os-version=3.0,run.tanzu.vmware.com/legacy-tkr=,
The following example shows that a TKr is purpose-built for vSphere 8.x because the label legacy-tkr is absent from the list of labels. 
kubectl get tkr v1.28.8---vmware.1-fips.1-tkg.2 --show-labels
NAME                              VERSION                         READY   COMPATIBLE   CREATED   LABELS
v1.28.8---vmware.1-fips.1-tkg.2   v1.28.8+vmware.1-fips.1-tkg.2   True    True         21d       os-arch=amd64,os-name=photon,os-type=linux,os-version=5.0,tkr.tanzu.vmware.com/standard=,v1.28.8---vmware.1-fips.1-tkg.2=,v1.28.8---vmware.1-fips.1-tkg=,v1.28.8---vmware.1-fips.1=,v1.28.8---vmware.1-fips=,v1.28.8---vmware.1=,v1.28.8---vmware=,v1.28.8=,v1.28=,v1=
Running the equivalent command using -o yaml also shows that the legacy-tkr label is absent, indicating the TKr is purpose-built for vSphere 8.x. 
 kubectl get tkr v1.28.8---vmware.1-fips.1-tkg.2 -o yaml
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesRelease
metadata:
  creationTimestamp: "2024-05-08T20:03:57Z"
  finalizers:
  - tanzukubernetesrelease.run.tanzu.vmware.com
  generation: 2
  labels:
    os-arch: amd64
    os-name: photon
    os-type: linux
    os-version: "5.0"
    tkr.tanzu.vmware.com/standard: ""
    v1: ""
    v1.28: ""
    v1.28.8: ""
    v1.28.8---vmware: ""
    v1.28.8---vmware.1: ""
    v1.28.8---vmware.1-fips: ""
    v1.28.8---vmware.1-fips.1: ""
    v1.28.8---vmware.1-fips.1-tkg: ""
    v1.28.8---vmware.1-fips.1-tkg.2: ""
  name: v1.28.8---vmware.1-fips.1-tkg.2
...

TKr Operating System Image Format

The TKr OS image format supports multiple operating system images for a single TKr. This means that there is a single Tanzu Kubernetes release for a specific Kubernetes version for all supported operating systems, which are currently PhotonOS and Ubuntu. The default OS image format is PhotonOS.

By default the PhotonOS edition of the named TKr is used for TKG cluster nodes. If the referenced TKr supports the OS image format and has an Ubuntu OS edition available, use the run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu annotation to specify the Ubuntu OS edition of the TKr. For example, the following TKG cluster specification uses the Ubuntu edition of the named TKr. 
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:
  name: tkgs-cluster-ubuntu
  namespace: tkgs-cluster-ns
  annotations:
    run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
spec:
   topology:
     controlPlane:
       ...
       tkr:
         reference:
           name: v1.28.8---vmware.1-fips.1-tkg.2
The OS image format supports heterogeneous cluster deployments. For example, the following cluster manifest creates a Tanzu Kubernetes cluster with the default PhotonOS for control plane nodes and Ubuntu for the worker nodes. The TKr version is referenced in the control plane section and the annotation specifies Ubuntu for the named worker node pool. 
apiVersion: run.tanzu.vmware.com/v1alpha3
kind: TanzuKubernetesCluster
metadata:    
  name: tkgs-cluster-multiOS
  namespace: tkgs-cluster-ubuntu
  annotations:
    //Worker nodes annotation
    run.tanzu.vmware.com/resolve-os-image.np-1: os-name=ubuntu
spec:    
  topology:      
    controlPlane:        
      tkr:          
        reference:            
          name: v1.28.8---vmware.1-fips.1-tkg.2
      replicas: 3
      vmClass: guaranteed-medium
      storageClass: tkgs-storage-profile 
    nodePools:     
    - replicas: 3
      name: np-1
      vmClass: guaranteed-medium
      storageClass: tkgs-storage-profile

When the system is upgraded to vSphere 8, existing TKrs are automatically converted to the TKr OS image format with reference to a single OS image. This enables legacy TKrs to be compatible for upgrade to a non-legacy TKr.

Legacy TKrs have two editions: Photon and Ubuntu. When the legacy TKr has an Ubuntu specific edition, you can use either the full version string (with "ubuntu") and omit the annotation label, or use the short version string (without "ubuntu") and include the version label.

TKr Packages

TKr images compatible with vSphere 8 are updated to a package-based framework for core components, such as the container storage interface (CSI) and container network interface (CNI). If you are using the v1beta API, changing or updating these components is done using custom resource definitions.

To view the packages that comprising a TKr, run the following command: 
kubectl get tkr TKR-NAME -o yaml
For example: 
kubectl get tkr v1.28.8---vmware.1-fips.1-tkg.2 -o yaml
The command returns all the packages in the TKr. For example: 
spec:
  bootstrapPackages:
  - name: antrea.tanzu.vmware.com.1.13.3+vmware.3-tkg.2-vmware
  - name: vsphere-pv-csi.tanzu.vmware.com.3.1.0+vmware.1-tkg.6-vmware
  - name: vsphere-cpi.tanzu.vmware.com.1.28.0+vmware.1-tkg.1-vmware
  - name: kapp-controller.tanzu.vmware.com.0.48.2+vmware.1-tkg.1-vmware
  - name: guest-cluster-auth-service.tanzu.vmware.com.1.3.3+tkg.1-vmware
  - name: metrics-server.tanzu.vmware.com.0.6.2+vmware.3-tkg.5-vmware
  - name: secretgen-controller.tanzu.vmware.com.0.15.0+vmware.1-tkg.1-vmware
  - name: pinniped.tanzu.vmware.com.0.25.0+vmware.2-tkg.1-vmware
  - name: capabilities.tanzu.vmware.com.0.32.1+vmware.1
  - name: gateway-api.tanzu.vmware.com.1.0.0+vmware.1-tkg.1-vmware
  - name: calico.tanzu.vmware.com.3.26.3+vmware.1-tkg.1-vmware

See v1beta1 Example: Cluster with Calico CNI for an example use case.

Migrating TKr OS Types

In-place cluster updates between TKr operating systems are not supported. This means, for example, that you cannot upgrade a TKG cluster using TKr v1.27.11 Photon to TKr v1.28.8 Ubuntu.

If you want to change the TKr OS type that a TKG cluster is using, consider the following procedure. In this example, the origin cluster is using TKr Photon and the target is TKr Ubuntu.

TKr Hardening

Security Technical Implementation Guides (STIG) for system components, including Supervisor and TKrs, are available. See Tanzu STIG Hardening for details.

Build Your Own TKr

Starting with TKr v1.25.7 for vSphere 8.x, you can build custom TKr machine images for TKG clusters on vSphere 8. A custom machine image packages a supported operating system and version, a Kubernetes version based on a released TKr, and any customizations you make.

To build custom machine images for TKG cluster nodes, use the vSphere Tanzu Kubernetes Grid Image Builder. Refer to the documentation for details on building custom images, supported TKr versions, and supported customizations.