Upgrade Workload Clusters

This topic explains how to upgrade Tanzu Kubernetes Grid (TKG) workload clusters. For TKG with a standalone management cluster, you first need to upgrade the management cluster that manages the workload clusters.

Important

To upgrade workload clusters that you deployed with Supervisor in vSphere 8, see Upgrading Clusters Deployed by Supervisor (vSphere 8 Only) in Creating and Managing TKG 2.5 Workload Clusters on vSphere with the Tanzu CLI.

You cannot upgrade clusters with Windows worker nodes as described in Deploy a Multi-OS Workload Cluster.

To upgrade edge workload clusters that use VM templates stored locally, see Upgrade an Edge Cluster with a Local VM Template.

To upgrade workload clusters that you created from a custom cluster manifest, see Upgrade Custom Clusters.

Standalone management clusters and workload clusters use client certificates to authenticate clients. These certificates are valid for one year. To renew them, upgrade your clusters at least once a year.

Prerequisites

You have completed the steps in Upgrading Tanzu Kubernetes Grid.
If you are running TKG with a standalone management cluster, you have completed the steps in Upgrade Standalone Management Clusters.
Before you can upgrade clusters to a non-default version of Kubernetes for your version of Tanzu Kubernetes Grid, the appropriate base image template OVAs must be available in vSphere as VM templates. For information about importing OVA files into vSphere, see Prepare to Upgrade Clusters.

Important

Tanzu Kubernetes Grid v2.5.x does not support upgrading existing TKG workload clusters on AWS and Azure. For more information, see End of Support for TKG Management and Workload Clusters on AWS and Azure in the VMware Tanzu Kubernetes Grid v2.5.x Release Notes.

Tanzu Kubernetes Grid does not support upgrading workload clusters to v2.5.1 on vSphere 6.7. For more information, see End of Support for TKG Management and Workload Clusters on vSphere 6.7.

Procedure

The upgrade process upgrades the version of Kubernetes in all of the control plane and worker nodes of your workload clusters.

This procedure assumes that you are upgrading to Tanzu Kubernetes Grid v2.5.1.

To see an interactive list of available management clusters and select the management cluster that manages the clusters that you want to upgrade, run the tanzu context use command:
```
tanzu context use
```

To list your workload clusters, run:

tanzu cluster list --include-management-cluster -A

The tanzu cluster list command with the --include-management-cluster -A options shows the version of Kubernetes that is running in the management cluster and all of the clusters that it manages. In this example, you can see that the management cluster has already been upgraded to v1.28.7, but the workload clusters are running older versions of Kubernetes.

NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN  TKR
k8s-1-26-8-cluster   default     running   1/1           1/1      v1.26.8+vmware.1   <none>      dev   v1.26.8---vmware.1-tkg.1
k8s-1-27-5-cluster   default     running   1/1           1/1      v1.27.5+vmware.1   <none>      dev   v1.27.5---vmware.1-tkg.1
mgmt-cluster         tkg-system  running   1/1           1/1      v1.28.7+vmware.1   management  dev   v1.28.7---vmware.1-tkg.1

To discover which versions of Kubernetes are made available by a management cluster, run the tanzu kubernetes-release get command:

tanzu kubernetes-release get

The output lists all of the versions of Kubernetes that you can use to deploy clusters, with the following notes:

COMPATIBLE: The current management cluster can deploy workload clusters with this Tanzu Kubernetes release (tkr).
UPDATES AVAILABLE: This tkr is not the most current in its Kubernetes version line. Any workload clusters running this tkr version can be upgraded to newer versions.

For example:

NAME                              VERSION                         COMPATIBLE  ACTIVE  UPDATES AVAILABLE
v1.26.14---vmware.2-tiny.2-tkg.4  v1.26.14+vmware.2-tiny.2-tkg.4  True        True
v1.26.14---vmware.1-tkg.1         v1.26.14+vmware.1-tkg.1         True        True
v1.27.11---vmware.1-tiny.2-tkg.3  v1.27.11+vmware.1-tiny.2-tkg.3  True        True
v1.27.11---vmware.1-tkg.1         v1.27.11+vmware.1-tkg.1         True        True
v1.28.7---vmware.1-tiny.2-tkg.1   v1.28.7+vmware.1-tiny.2-tkg.1   True        True
v1.28.7---vmware.1-tkg.1          v1.28.7+vmware.1-tkg.1          True        True

To discover the newer tkr versions to which you can upgrade a workload cluster running an older tkr version, run the tanzu kubernetes-release available-upgrades get command, specifying the current tkr version:
```
tanzu kubernetes-release available-upgrades get v1.26.8---vmware.2-tkg.1
```
This command lists all of the available Kubernetes versions to which you can upgrade clusters that are running the specified version.

You can also discover the tkr versions that are available for a specific workload cluster by specifying the cluster name in the tanzu cluster available-upgrades get command:
```
tanzu cluster available-upgrades get k8s-1-26-8-cluster
```
This command lists all of the Kubernetes versions that are compatible with the specified cluster.

You cannot skip minor versions when upgrading your tkr version. For example, you cannot upgrade a cluster directly from v1.26.x to v1.28.x. You must upgrade a v1.26.x cluster to v1.27.x before upgrading the cluster to v1.28.x.

Any cluster that uses Node IPAM requires at least one unassigned IP address in its IP pool before upgrading.

To check if a cluster uses Node IPAM, look for addressesFromPools in the cluster object’s spec.topology.variables network setting:

kubectl -n NAMESPACE get cluster CLUSTER-NAME -o json | jq '.spec.topology.variables[] | select(.name=="network")'

Where CLUSTER-NAME and NAMESPACE are the workload cluster’s name and management cluster namespace. For example:

kubectl -n default get cluster my-work-cluster -o json | jq '.spec.topology.variables[] | select(.name=="network")'
{
  "name": "network",
  "value": {
    "addressesFromPools": [
      {
        "apiGroup": "ipam.cluster.x-k8s.io",
        "kind": "InClusterIPPool",
        "name": "mgmt-cluster-nimbus"
      }
    ],
    "ipv6Primary": false
  }
}

To check that the cluster’s IP pool has unused addresses, compare the total number in the pool with the number currently allocated, as output by the following command:
```
kubectl -n NAMESPACE get ipaddress | grep POOL-NAME | wc -l
```

Run the tanzu cluster upgrade CLUSTER-NAME command and enter y to confirm. To skip the confirmation step, specify the --yes option.

Important
For additional command options and configuration steps that may be required, see Additional Steps for Certain OS, Kubernetes, and Cluster Type Combinations below.

To upgrade the cluster to the default version of Kubernetes for this release of Tanzu Kubernetes Grid, run the tanzu cluster upgrade command without any options. In this release, the default version is v1.28.7. For example:
```
tanzu cluster upgrade k8s-1-27-8-cluster
```
If the cluster is not running in the default namespace, specify the --namespace option:
```
tanzu cluster upgrade CLUSTER-NAME --namespace NAMESPACE-NAME
```
If an upgrade times out before it completes, run tanzu cluster upgrade again and specify the --timeout option with a value greater than the default of 30 minutes:
```
tanzu cluster upgrade CLUSTER-NAME --timeout 45m0s
```
If multiple base VM images in your IaaS account have the same version of Kubernetes that you are upgrading to, you can include --os-name and other options to specify the target OS as described in Select an OS to Upgrade To:
```
tanzu cluster upgrade CLUSTER-NAME --os-name ubuntu
```
On vSphere, you can use the --vsphere-vm-template-name option to specify a target OVA template for cluster nodes as described in Select an OVA Template to Upgrade To:
```
tanzu cluster upgrade CLUSTER-NAME --vsphere-vm-template-name "/dc0/vm/tanzu/ubuntu-2204-kube-v1.28.7-vmware.1"
```
To upgrade a cluster to a version of Kubernetes that is not the default, for example if you are upgrading a cluster stepwise from v1.26.8 to v1.27.11 before upgrading to v1.27.11, specify the --tkr option with the NAME of the target version as listed by tanzu kubernetes-release get above. For example:
```
tanzu cluster upgrade k8s-1-26-8-cluster --tkr v1.27.11---vmware.1-tkg.1
```
Additional Steps for Certain OS, Kubernetes, and Cluster Type Combinations: Because TKG supports different Ubuntu and Photon OS versions for Kubernetes v1.26, v1.27, and v1.28, upgrading a workload cluster may require additional steps depending on its current and target OS and Kubernetes versions, and whether the cluster is plan-based or class-based as described in Workload Cluster Types:
- Plan-based to Kubernetes v1.28: If you are upgrading a legacy plan-based workload cluster to Kubernetes v1.28, you must specify the --os-name, --os-version, and --os-arch options in the tanzu cluster upgrade command in order to update Ubuntu clusters to 22.04 and Photon clusters to Photon 5. For example:
```
tanzu cluster upgrade k8s-1-27-8-cluster --os-name photon --os-version 5 --os-arch amd64
```
```
tanzu cluster upgrade k8s-1-27-8-cluster --os-name ubuntu --os-version 22.04 --os-arch amd64
```
  These options are not required when upgrading class-based clusters.
- Plan-based Photon 3 to Kubernetes v1.27 on Photon 3: If you are upgrading a plan-based cluster on Photon 3 to Kubernetes v1.27, and want it to continue using Photon 3 instead of the default Photon 5, specify photon3 as the OS name:
```
tanzu cluster upgrade k8s-1-26-8-cluster --tkr v1.27.11---vmware.1-tkg.1 -t 2h --yes --os-name photon3 --os-version 3 --os-arch amd64
```
- Class-based Photon 3 to Kubernetes v1.27 on Photon 3: If you are upgrading a class-based cluster on Photon 3 to Kubernetes v1.27, and want it to continue using Photon 3 instead of the default Photon 5:
  1. From the management cluster context, edit the workload cluster’s Cluster object. For example:
```
kubectl edit cluster k8s-1-26-8-cluster --context=tkg-mgmt-vc-admin@tkg-mgmt-vc
```
    Make the changes below to the Cluster object’s spec.topology block.
  2. Update class to cluster class version tkg-vsphere-default-v1.2.0:
```
spec:
topology:
  class: tkg-vsphere-default-v1.2.0
```
  3. Update version to v1.27.11+vmware.1
```
spec:
topology:
  ...
  version: v1.27.11+vmware.1
  ...
```
  4. Update the controlPlane > run.tanzu.vmware.com/resolve-os-image annotation key to os-name=photon3:
```
spec:
topology:
  ...
  controlPlane:
    metadata:
      annotations:
        run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=photon3
    replicas: 1
  ...
```
  5. Update the workers > run.tanzu.vmware.com/resolve-os-image annotation key to os-name=photon3:
```
spec:
topology:
  ...
  workers:
    machineDeployments:
    - class: tkg-worker
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: image-type=ova,os-name=photon3
        name: md-0
        replicas: 1
        strategy:
          type: RollingUpdate
  ...
```
When the upgrade finishes, run the tanzu cluster list command to check that the workload cluster has been upgraded:
```
tanzu cluster list --include-management-cluster -A
```
Regenerate the admin kubeconfig:
```
tanzu cluster kubeconfig get CLUSTER-NAME --admin
```
Where CLUSTER-NAME is the name of the workload cluster.

Important
If you don’t renew the kubeconfig after upgrading, you won’t be able to access the cluster once it expires.
If you are using an LDAP or OIDC identity provider, confirm that you can authenticate to the cluster with kubectl. For example:
```
kubectl get pods -A --kubeconfig my-cluster-credentials
```
Upgrade any CLI-managed packages such as Contour, Fluent Bit or Prometheus that are running on your workload clusters. For more information on upgrading CLI-managed packages, see Update a Package.

Upgrade an Edge Cluster with a Local VM Template

You cannot use the tanzu cluster upgrade command to upgrade the Kubernetes version of an edge workload cluster with a local VM template as described in Specifying a Local VM Template above.

Instead, upgrade the Kubernetes version of the edge workload cluster as follows:

Upload the new VM template to the local vCenter and record its inventory path, for example /dc0/vm/ubuntu-2204-kube-v1.28.7+vmware.1-tkg.1.
Edit the manifest for the Cluster object to upgrade:
```
kubectl edit cluster CLUSTER-NAME
```
Where CLUSTER-NAME is the name of the cluster
Under spec.topology update the following:
- Set version to reflect the new Kubernetes version.
- Set all vcenter.template values to update, cluster-wide or under individual machine deployments, to the inventory path of the new local template.
Save and exit to apply the new Cluster object settings.

What to Do Next

You can now continue to use the Tanzu CLI to manage your clusters. For more information, see Creating and Managing TKG 2.5 Workload Clusters on vSphere with the Tanzu CLI.