This topic describes how to use the VMware Tanzu Kubernetes Grid Integrated Edition Command Line Interface (TKGI CLI) to upgrade TKGI-provisioned Kubernetes clusters.
For information about how to upgrade TKGI-provisioned clusters through the Tanzu Kubernetes Grid Integrated Edition tile, see Verify Errand Configuration in one of the following topics:
For conceptual information about Tanzu Kubernetes Grid Integrated Edition upgrades, see About Tanzu Kubernetes Grid Integrated Edition Upgrades.
Upgrading a TKGI-provisioned Kubernetes cluster updates the Tanzu Kubernetes Grid Integrated Edition version and the Kubernetes version of the cluster.
TKGI-provisioned Kubernetes clusters upgrade when:
tkgi upgrade-cluster
or tkgi upgrade-clusters
as described in Upgrade Clusters below.For example, running tkgi upgrade-cluster
upgrades the cluster you specify to your current version of Tanzu Kubernetes Grid Integrated Edition and to the version of Kubernetes that is included with your current version of Tanzu Kubernetes Grid Integrated Edition.
WARNING: Do not change the number of control plane/etcd nodes for any plan that was used to create currently-running clusters. Tanzu Kubernetes Grid Integrated Edition does not support changing the number of control plane/etcd nodes for plans with existing clusters.
Before upgrading TKGI-provisioned Kubernetes clusters:
tkgi login
. For more information, see Logging in to Tanzu Kubernetes Grid Integrated Edition.VMware recommends that you switch your Docker container runtime clusters to containerd before upgrading to TKGI v1.15.
The TKGI v1.15 upgrade will switch a cluster’s container runtime from Docker to containerd.
Warning: Cluster workloads will experience downtime during the TKGI v1.15 upgrade if the cluster they are on is switched from using the Docker container runtime to containerd during the upgrade.
For more information, see Switch Docker Container Runtime Clusters to the containerd Container Runtime in Upgrade Preparation Checklist for TKGI.
You can upgrade a cluster’s TKGI version to the TKGI version currently running on the TKGI control plane.
To upgrade a cluster’s TKGI version:
Use the TKGI CLI to upgrade the TKGI version on individual or multiple clusters:
To monitor or stop a cluster upgrade, follow the procedures in Manage Your Kubernetes Cluster Upgrade Job below.
The Tanzu Kubernetes Grid Integrated Edition CLI provides upgrade-cluster
for upgrading an individual Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes cluster.
To upgrade an individual Kubernetes cluster:
Run the following command:
tkgi upgrade-cluster CLUSTER-NAME
Where CLUSTER-NAME
is the name of the Kubernetes cluster you want to upgrade.
For more information about the tkgi upgrade-cluster
command, see tkgi upgrade-cluster in the TKGI CLI documentation.
Note: The nodes in an upgrading cluster are processed serially.
To upgrade multiple clusters, see Upgrade Multiple Kubernetes Clusters below.
The Tanzu Kubernetes Grid Integrated Edition CLI provides upgrade-clusters
for upgrading multiple Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes clusters. You can upgrade clusters serially, serially with some clusters designated as canary clusters, or entirely in parallel.
To upgrade multiple Kubernetes clusters:
Run the following command:
tkgi upgrade-clusters CLUSTER-NAMES
Where CLUSTER-NAMES
is a list of names of the Kubernetes clusters that you want to upgrade.
For more information about the tkgi upgrade-clusters
command, see tkgi upgrade-clusters in the TKGI CLI documentation.
Note: The nodes in an upgrading cluster are always processed serially.
To upgrade a single cluster, see Upgrade a Single Kubernetes Cluster above.
To upgrade multiple Kubernetes clusters:
Run the following command:
tkgi upgrade-clusters --clusters CLUSTER-NAMES --max-in-flight CLUSTER-COUNT --wait
Where:
CLUSTER-NAMES
is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade.
CLUSTER-COUNT
is the maximum number of clusters to upgrade in parallel within an AZ. The CLUSTER-COUNT
must be less than your TKGI tile > TKGI API Service > Worker VM Max in Flight value. For example, if your TKGI tile Worker VM Max in Flight value remains the default of 4
, run upgrade-clusters
with a –max-in-flight
argument value less than 4
.
Considerations when running tkgi upgrade-clusters
:
--clusters
list fails, the tkgi upgrade-clusters
job continues to a subsequent cluster in the list.--max-in-flight
is not set.--clusters
list is more than the --max-in-flight
value, the first set of clusters are upgraded in parallel and subsequent clusters are queued. As the initial cluster upgrades complete, the remaining clusters are pulled from the queue and upgraded in parallel.--wait
argument.Note: The nodes in an upgrading cluster are always processed serially.
For example:
$ tkgi upgrade-clusters --clusters k8-cluster-000,k8-cluster-001,k8-cluster-002 --max-in-flight 2 --wait
You are about to upgrade k8-cluster-000, k8-cluster-001 and k8-cluster-002.
Warning: This operation might be long running and might block further operations on the cluster(s) until complete
Continue? (y/n):y
Your taskID for the upgrade task is: d772aba0-2670-4fba-b26c-044b19d6ab60
Started upgrading cluster: k8-cluster-000
Started upgrading cluster: k8-cluster-001
Finished upgrading cluster: k8-cluster-000
Started upgrading cluster: k8-cluster-002
Finished upgrading cluster: k8-cluster-001
Finished upgrading cluster: k8-cluster-002
Upgrade task d772aba0-2670-4fba-b26c-044b19d6ab60 is done.
To upgrade multiple clusters and automatically stop upgrading clusters if a cluster upgrade fails, specify your cluster list as canary clusters. You can specify one or more clusters as canary clusters.
To upgrade multiple clusters with one or more canary clusters:
Run the following command:
tkgi upgrade-clusters --canaries CANARY-CLUSTER-NAMES --clusters CLUSTER-NAMES --wait
Where:
CANARY-CLUSTER-NAMES
is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade as canary clusters.
CLUSTER-NAMES
is a comma-delimited list of Kubernetes clusters to upgrade if all canary clusters successfully upgrade.
Note: The –clusters
argument is required.
Considerations when running tkgi upgrade-clusters
with a --canaries
list:
--canaries
list are upgraded prior to upgrading the clusters in your --clusters
list.tkgi upgrade-clusters
job stops.--clusters
list cluster upgrade fails, the tkgi upgrade-clusters
job continues to a subsequent cluster in the list.tkgi upgrade-clusters
to stop for any cluster upgrade failure, specify only one cluster in your –clusters
list and the remaining clusters in your –canaries
list.--clusters
list in parallel, see Upgrade Clusters in Parallel above.--wait
argument.Note: The nodes in an upgrading cluster are always processed serially.
For example:
$ tkgi upgrade-clusters --canaries k8-cluster-dev,k8-cluster-000,k8-cluster-001 --clusters k8-cluster-002 --wait
You are about to upgrade k8-cluster-dev k8-cluster-000, k8-cluster-001 and k8-cluster-002.
Warning: This operation might be long running and might block further operations on the cluster(s) until complete
Continue? (y/n):y
Your taskID for the upgrade task is: ce31a1bb-380a-453f-afa0-835ffa1ce6ac
Started upgrading cluster: k8-cluster-000
Upgrading cluster succeeded: k8-cluster-000
Started upgrading cluster: k8-cluster-001
Upgrading cluster succeeded: k8-cluster-001
Started upgrading cluster: k8-cluster-dev
Upgrading cluster failed: k8-cluster-dev
Upgrade task ce31a1bb-380a-453f-afa0-835ffa1ce6ac is done.
You can use the TKGI CLI to monitor and manage your Tanzu Kubernetes Grid Integrated Edition-provisioned Kubernetes cluster upgrade jobs:
To review the status of the actions being performed on your clusters, run the following command:
tkgi clusters
For example:
$ tkgi clusters
Upgrade is available to TKGI Version: 1.9.0-build.1
TKGI Version Name k8s Version Plan Name UUID Status Action
1.9.0-build.1 k8-cluster-000 1.18.8 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 succeeded UPGRADE
1.9.0-build.1 k8-cluster-001 1.18.8 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 failed UPGRADE
1.9.0-build.1 k8-cluster-002 1.18.8 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 in progress UPGRADE
1.9.0-build.1 k8-cluster-003 1.18.8 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 queued UPGRADE
To review the status of your upgrade-clusters
job, run the following command:
tkgi task TASKID
Where TASKID
is the ID of the task that was returned when you ran tkgi upgrade-clusters
.
For example:
$ tkgi task ce31a1bb-380a-453f-afa0-835ffa1ce6ac
Your upgrade task is: done
Name Status Start time End time isCanary
k8-cluster-000 succeeded Mon, 14 Oct 2019 12:00:00 PDT Mon, 14 Oct 2019 12:19:54 PDT true
k8-cluster-001 failed Mon, 14 Oct 2019 12:20:00 PDT --- true
To cancel a running upgrade-clusters
job, run the following TKGI CLI command:
tkgi cancel-task TASKID
Where TASKID
is the ID of the task that was returned when you ran tkgi upgrade-clusters
.
Warning: tkgi cancel-task
does not cancel cluster upgrades currently in progress. This command only cancels a job’s pending cluster upgrades.
Complete the following optional procedures after you have upgraded your cluster:
TKGI v1.14 uses Velero v1.8.1. You must upgrade Velero to v1.8.1 on all of your existing clusters.
To upgrade Velero:
Warning: Ensure you have updated the Velero custom resource definitions on your clusters as described in Instructions in Upgrading to Velero 1.8 in the Velero documentation.
If you scaled your cluster up for the upgrade and you prefer to restore your cluster to its original sizing, you can now scale the cluster back down to its previous configuration. VMware recommends that you not scale down your clusters and continue to run them with recommended configurations, reducing the chance of a future outage.