Upgrade Tanzu Kubernetes Clusters

After you have upgraded a management cluster, you can upgrade the Tanzu Kubernetes clusters that the management cluster manages.

Important: Management clusters and Tanzu Kubernetes clusters use client certificates to authenticate clients. These certificates are valid for one year. To renew them, upgrade your clusters at least once a year.

Prerequisites

You performed the steps in Upgrade Tanzu Kubernetes Grid that occur before the step for upgrading Tanzu Kubernetes Clusters.
You performed the steps in Upgrade Management Clusters to upgrade the management cluster that manages the Tanzu Kubernetes clusters that you want to upgrade.
If you are upgrading clusters that run on vSphere, before you can upgrade clusters to a non-default version of Kubernetes for your version of Tanzu Kubernetes Grid, the appropriate base image template OVAs must be available in vSphere as VM templates. For information about importing OVA files into vSphere, see Prepare to Upgrade Clusters on vSphere.
If you are upgrading clusters that run on Amazon EC2, the Amazon Linux 2 Amazon Machine Images (AMI) that include the supported Kubernetes versions are publicly available to all Amazon EC2 users, in all supported AWS regions. Tanzu Kubernetes Grid automatically uses the appropriate AMI for the Kubernetes version that you specify during upgrade.
If you are upgrading clusters that run on Azure, ensure that you completed the steps in Prepare to Upgrade Clusters on Azure.

Procedure

The upgrade process upgrades the version of Kubernetes in all of the control plane and worker nodes of your Tanzu Kubernetes clusters.

Run the tanzu login command to see an interactive list of available management clusters.
```
tanzu login
```
Select a management cluster to switch the context of the Tanzu CLI. You should select the management cluster that manages the clusters you want to upgrade. See List Management Clusters and Change Context for more information.
Run the tanzu cluster list command with the --include-management-cluster option.
```
tanzu cluster list --include-management-cluster
```
The tanzu cluster list command shows the version of Kubernetes that is running in the management cluster and all of the clusters that it manages. In this example, you can see that the management cluster has already been upgraded to v1.22.9, but the Tanzu Kubernetes clusters are running older versions of Kubernetes.
```
 NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN
 k8s-1-20-14-cluster   default     running   1/1           1/1      v1.20-14+vmware.1   <none>      dev
 k8s-1-21-8-cluster   default     running   1/1           1/1      v1.21.8+vmware.1   <none>      dev
 mgmt-cluster         tkg-system  running   1/1           1/1
 v1.22.9+vmware.1   management  dev
```
Note: After you have installed the v1.5 CLI but before a management cluster has been deployed or upgraded, all context-specific CLI command groups (tanzu cluster, tanzu kubernetes-release) plus all of the management-cluster plugin commands except for tanzu mc upgrade and tanzu mc create are unavailable and not included in Tanzu CLI --help output.

To discover which versions of Kubernetes are made available by a management cluster, run the tanzu kubernetes-release get command.

tanzu kubernetes-release get

The output lists all of the versions of Kubernetes that you can use to deploy clusters, with the following notes: - COMPATIBLE: The current management cluster can deploy workload clusters with this Tanzu Kubernetes release (tkr). - UPGRADEAVAILABLE: This tkr is not the most current in its Kubernetes version line. Any workload clusters running this tkr version can be upgraded to newer versions.

For example:

 NAME                        VERSION                   COMPATIBLE  UPGRADEAVAILABLE
 v1.20.14---vmware.1-tkg.1    v1.20.14+vmware.1-tkg.1    True        True
 v1.20.15---vmware.1-tkg.1   v1.20.15+vmware.1-tkg.1   True        True
 v1.21.8---vmware.1-tkg.1    v1.21.8+vmware.1-tkg.1    True        True
 v1.21.11---vmware.1-tkg.1    v1.21.11+vmware.1-tkg.1    True        False
 v1.22.8---vmware.1-tkg.1    v1.22.8+vmware.1-tkg.1    True        False
 v1.22.9---vmware.1-tkg.1    v1.22.9+vmware.1-tkg.1    True        False

To discover the newer tkr versions to which you can upgrade a workload cluster running an older tkr version, run the tanzu kubernetes-release available-upgrades get command, specifying the current version of the cluster.

tanzu kubernetes-release available-upgrades get v1.20.14---vmware.1-tkg.1

This command lists all of the available Kubernetes versions to which you can upgrade clusters that are running the specified version.

NAME                       VERSION
v1.20.15---vmware.1-tkg.1  v1.20.15+vmware.1-tkg.1
v1.21.8---vmware.1-tkg.1   v1.21.8+vmware.1-tkg.1
v1.21.11---vmware.1-tkg.1   v1.21.11+vmware.1-tkg.1
v1.22.8---vmware.1-tkg.1    v1.22.8+vmware.1-tkg.1
v1.22.9---vmware.1-tkg.1    v1.22.9+vmware.1-tkg.1

You can also discover the tkr versions that are available for a specific workload cluster by specifying the cluster name in the tanzu cluster available-upgrades get command.

tanzu cluster available-upgrades get k8s-1-20-14-cluster

This command lists all of the Kubernetes versions that are compatible with the specified cluster.

NAME                         VERSION                            COMPATIBLE
v1.20.14---vmware.1-tkg.1     v1.20.14+vmware.1-tkg.1           True
v1.20.15---vmware.1-tkg.2    v1.20.15+vmware.1-tkg.2            True
v1.21.8---vmware.1-tkg.1     v1.21.8+vmware.1-tkg.1             True
v1.21.11---vmware.1-tkg.1     v1.21.11+vmware.1-tkg.1           True
v1.22.8---vmware.1-tkg.1     v1.22.8+vmware.1-tkg.1             True
v1.22.9---vmware.1-tkg.1     v1.22.9+vmware.1-tkg.1             True

You cannot skip minor versions when upgrading your tkr version. For example, you cannot upgrade a cluster directly from v1.20.x to v1.22.x. You must upgrade a v1.20.x cluster to v1.21.x before upgrading the cluster to v1.22.x.

If the cluster is running on Azure, set the AZURE_CLIENT_SECRET environment variable before upgrading the cluster:
```
export AZURE_CLIENT_SECRET=YOUR-AZURE-CLIENT-SECRET
```
Run the tanzu cluster upgrade CLUSTER-NAME command and enter y to confirm.

To upgrade the cluster to the default version of Kubernetes for this release of Tanzu Kubernetes Grid, run the tanzu cluster upgrade command without any options. For example, the following command upgrades the cluster k8s-1-21-8-cluster from v1.21.8 to v1.22.9.
```
tanzu cluster upgrade k8s-1-21-8-cluster
```
If the cluster is not running in the default namespace, specify the --namespace option.
```
tanzu cluster upgrade CLUSTER-NAME --namespace NAMESPACE-NAME
```
To skip the confirmation step when you upgrade a cluster, specify the --yes option.
```
tanzu cluster upgrade CLUSTER-NAME --yes
```
If an upgrade times out before it completes, run tanzu cluster upgrade again and specify the --timeout option with a value greater than the default of 30 minutes.
```
tanzu cluster upgrade CLUSTER-NAME --timeout 45m0s
```
If multiple base VM images in your IaaS account have the same version of Kubernetes that you are upgrading to, use the --os-name option to specify the OS you want. See Selecting an OS During Cluster Upgrade for more information.

For example, on vSphere if you have uploaded both Photon and Ubuntu OVA templates with Kubernetes v1.22.9, specify --os-name ubuntu to upgrade your workload cluster to run on an Ubuntu VM.
```
tanzu cluster upgrade CLUSTER-NAME --os-name ubuntu
```
Since you cannot skip minor versions of tkr, the upgrade command fails if you try to upgrade a cluster that is more than one minor version behind the default version. For example, you cannot upgrade directly from v1.20.x to v1.22.x. To upgrade a cluster to a version of Kubernetes that is not the default version for this release of Tanzu Kubernetes Grid, specify the --tkr option with the NAME of the chosen version, as listed by tanzu kubernetes-release get above. For example, to upgrade the cluster k8s-1-20-14-cluster from v1.20.14 to v1.21.11.
```
tanzu cluster upgrade k8s-1-20-14-cluster --tkr v1.21.11---vmware.1-tkg.1 --yes
```

When the upgrade finishes, run the tanzu cluster list command with the --include-management-cluster option again, to check that the Tanzu Kubernetes cluster has been upgraded.

tanzu cluster list --include-management-cluster

You see that the k8s-1-20-14-cluster and k8s-1-21-8-cluster workload clusters are now running Kubernetes v1.21.11 and v1.22.9 respectively.

 NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN
 k8s-1-20-14-cluster   default     running   1/1           1/1      v1.21.11+vmware.1  <none>      dev
 k8s-1-21-8-cluster   default     running   1/1           1/1      v1.22.9+vmware.1   <none>      dev
 mgmt-cluster         tkg-system  running   1/1           1/1
 v1.22.9+vmware.1   management  dev

Regenerate the admin kubeconfig:
```
tanzu cluster kubeconfig get CLUSTER-NAME --admin
```
Where CLUSTER-NAME is the name of the workload cluster.
If you are using an LDAP or OIDC identity provider, confirm that you can authenticate to the cluster with kubectl. For example:
```
kubectl get pods -A --kubeconfig my-cluster-credentials
```
If you receive an error message, see Pinniped Authentication Error on Workload Cluster After Management Cluster Upgrade in Troubleshooting Tanzu Kubernetes Grid Issues.
Upgrade any CLI-managed packages such as Contour, Fluent Bit or Prometheus that are running on your Tanzu Kubernetes clusters. For more information on upgrading CLI-managed packages, see Update a Package.

What to Do Next

You can now continue to use the Tanzu CLI to manage your clusters, and run your applications with the new version of Kubernetes.