check-circle-line exclamation-circle-line close-line

<

This topic describes how to upgrade Kubernetes clusters provisioned by VMware Enterprise PKS.

Overview

When upgrading Enterprise PKS, you can choose one of the following approaches:

  • Upgrade the Enterprise PKS tile and all Enterprise PKS-provisioned Kubernetes clusters simultaneously. This is the default behavior of the Enterprise PKS tile.
  • Upgrade the Enterprise PKS tile only and then upgrade Enterprise PKS-provisioned Kubernetes clusters separately. If you want to upgrade your clusters separately, do the following:

    1. Review Limitations.
    2. Complete the steps listed in Prerequisites.
    3. Complete the steps listed in Upgrade Clusters.

    Note: When Enterprise PKS is set to upgrade the Enterprise PKS tile only and not the clusters, the Kubernetes cluster version falls behind the Enterprise PKS tile version. If the clusters fall more than one version behind the tile, Enterprise PKS cannot upgrade the clusters. The clusters must be upgraded to match the Enterprise PKS tile version before the next tile upgrade.

For more information about Enterprise PKS upgrades, see What Happens During Enterprise PKS Upgrades.

Prerequisites

Before upgrading Kubernetes clusters provisioned by Enterprise PKS, you must do the following:

  • Disable Upgrade all clusters errand in the Enterprise PKS tile. For more information, see Modify Errand Configuration in the Enterprise PKS Tile below.

    Warning: If you upgrade the Enterprise PKS tile or update your tile configuration, with Upgrade all clusters errand disabled, you must upgrade all your Kubernetes clusters before the next upgrade or configuration update.

  • Install the PKS command line interface (CLI) v1.6. For more information, see Installing the PKS CLI.
  • Log in to Enterprise PKS using pks login. For more information, see Logging in to Enterprise PKS.

Modify Errand Configuration in the Enterprise PKS Tile

To modify errand configuration when upgrading your Enterprise PKS version, perform the steps below. For more information about upgrading Enterprise PKS, see Upgrading Enterprise PKS.

  1. If you have not already, import the product file for the new Enterprise PKS version to the Ops Manager Installation Dashboard.
  2. In the Enterprise PKS tile, navigate to the Errands pane.
  3. Disable Upgrade all clusters errand and click Save. Errands pane
  4. Return to the Ops Manager Installation Dashboard and click Review Pending Changes.
  5. Select Enterprise PKS and any other products that you want to deploy and click Apply Changes.

Upgrade Clusters

You can upgrade your Enterprise PKS tile separately from upgrading your Enterprise PKS-provisioned Kubernetes clusters.

After upgrading your Enterprise PKS tile, upgrade individual or multiple clusters using the following procedures:

To monitor or stop a cluster upgrade, follow the procedures in Manage Your Kubernetes Cluster Upgrade Job, below.

Upgrade a Single Kubernetes Cluster

The Enterprise PKS CLI provides upgrade-cluster for upgrading an individual Enterprise PKS-provisioned Kubernetes cluster.

To upgrade an individual Kubernetes cluster, run the following command:

pks upgrade-cluster CLUSTER-NAME

Where CLUSTER-NAME is the name of the Kubernetes cluster you want to upgrade.

To upgrade multiple clusters, see Upgrade Multiple Kubernetes Clusters, below.

For more information about the pks upgrade-cluster command, see pks upgrade-cluster in the PKS CLI documentation.

Upgrade Multiple Kubernetes Clusters

The Enterprise PKS CLI provides upgrade-clusters for upgrading multiple Enterprise PKS-provisioned Kubernetes clusters. You can upgrade clusters serially, serially with some clusters designated as canary clusters, or entirely in parallel.

To upgrade a single cluster, see Upgrade a Single Kubernetes Cluster above.

For more information about the pks upgrade-clusters command, see pks upgrade-clusters in the PKS CLI documentation.

Upgrade Multiple Kubernetes Clusters in Parallel

To upgrade multiple Kubernetes clusters, run the following command:

pks upgrade-clusters --clusters CLUSTER-NAMES --max-in-flight CLUSTER-COUNT --wait

Where:

  • CLUSTER-NAMES is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade.

  • CLUSTER-COUNT is the maximum number of clusters to upgrade in parallel within an an AZ.

    • If the CLUSTER-NAMES list is longer than the CLUSTER-COUNT, the first set of clusters are upgraded in parallel and subsequent clusters are queued and then upgraded in parallel as the preceding cluster upgrades complete.
    • If an upgrade fails for a cluster in the CLUSTER-NAMES list, the job continues to a subsequent cluster in the list.
    • To run the cluster upgrade job as a background task, remove the --wait argument.

    Note: Run upgrade-clusters with a --max-in-flight argument less than your BOSH Director > Director Config > Director Workers value. For example, if your Director Workers value remains the default of 5, run upgrade-clusters with a --max-in-flight argument value less than 5.

    Note: max-in-flight is an optional argument. If max-in-flight is not set, Enterprise PKS uses the default max-in-flight value of 1 and the clusters are upgraded serially.

For example:

 $ pks upgrade-clusters --clusters k8-cluster-000,k8-cluster-001,k8-cluster-002 --max-in-flight 2  --wait 
You are about to upgrade k8-cluster-000, k8-cluster-001 and k8-cluster-002. Warning: This operation may be long running and may block further operations on the cluster(s) until complete
Continue? (y/n):y Your taskID for the upgrade task is: d772aba0-2670-4fba-b26c-044b19d6ab60 Started upgrading cluster: k8-cluster-000 Started upgrading cluster: k8-cluster-001 Finished upgrading cluster: k8-cluster-000 Started upgrading cluster: k8-cluster-002 Finished upgrading cluster: k8-cluster-001 Finished upgrading cluster: k8-cluster-002 Upgrade task d772aba0-2670-4fba-b26c-044b19d6ab60 is done.

Upgrade Multiple Kubernetes Clusters With Canaries

To upgrade multiple clusters and automatically stop upgrading clusters if a cluster upgrade fails, specify your cluster list as canary clusters. You can specify one or more clusters as canary clusters.

To upgrade multiple clusters with one or more canary clusters, run the following command:

pks upgrade-clusters --canaries CANARY-CLUSTER-NAMES --clusters CLUSTER-NAMES --wait

Where:

  • CANARY-CLUSTER-NAMES is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade as canary clusters.

  • CLUSTER-NAMES is a comma-delimited list of Kubernetes clusters to upgrade if all canary clusters successfully upgrade.

    • The specified canary clusters are upgraded prior to upgrading the clusters in your CLUSTER-NAMES list.
    • Canary clusters are always upgraded serially.
    • If an upgrade fails for a canary cluster, the entire upgrade-clusters job stops.
    • If an upgrade fails for a cluster in the CLUSTER-NAMES list, the upgrade-clusters job continues to a subsequent cluster in the list.
    • To run the cluster upgrade job as a background task, remove the --wait argument.

    Note: --clusters is a required argument. To configure upgrade-clusters to stop for any cluster upgrade failure, specify only one cluster in your CLUSTER-NAMES list and the remaining clusters in your CANARY-CLUSTER-NAMES list.

    Note: Canary clusters are always upgraded serially. Only the clusters specified in the --clusters list are upgraded in parallel when you run upgrade-clusters with both --canaries and --max-in-flight arguments.

For example:

 $ pks upgrade-clusters --canaries k8-cluster-dev,k8-cluster-000,k8-cluster-001  --clusters k8-cluster-002  --wait 
You are about to upgrade k8-cluster-dev k8-cluster-000, k8-cluster-001 and k8-cluster-002. Warning: This operation may be long running and may block further operations on the cluster(s) until complete

Continue? (y/n):y Your taskID for the upgrade task is: ce31a1bb-380a-453f-afa0-835ffa1ce6ac Started upgrading cluster: k8-cluster-000 Upgrading cluster succeeded: k8-cluster-000 Started upgrading cluster: k8-cluster-001 Upgrading cluster succeeded: k8-cluster-001 Started upgrading cluster: k8-cluster-dev Upgrading cluster failed: k8-cluster-dev Upgrade task ce31a1bb-380a-453f-afa0-835ffa1ce6ac is done.

Manage Your Kubernetes Cluster Upgrade Job

You can use the PKS CLI to monitor and manage your Enterprise PKS-provisioned Kubernetes cluster upgrade jobs.

Monitor the Status of Your Kubernetes Clusters

To review the status of the actions being performed on your clusters, run the following command:

pks clusters

For example:

 $ pks clusters 
Upgrade is available to PKS Version: 1.6.0-build.14
PKS Version Name k8s Version Plan Name UUID Status Action 1.6.0-build.14 k8-cluster-000 1.14.5 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 succeeded UPGRADE 1.6.0-build.14 k8-cluster-001 1.14.5 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 failed UPGRADE 1.6.0-build.14 k8-cluster-002 1.14.5 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 in progress UPGRADE 1.6.0-build.14 k8-cluster-003 1.14.5 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 queued UPGRADE

Monitor the Status of Your Kubernetes Cluster Upgrade Job

To review the status of your upgrade-clusters job, run the following command:

pks task TASKID

Where TASKID is the ID of the task that was returned when you ran pks upgrade-clusters.

For example:

 $ pks task ce31a1bb-380a-453f-afa0-835ffa1ce6ac 
Your upgrade task is: done
Name Status Start time End time isCanary k8-cluster-000 succeeded Mon, 14 Oct 2019 12:00:00 PDT Mon, 14 Oct 2019 12:19:54 PDT true k8-cluster-001 failed Mon, 14 Oct 2019 12:20:00 PDT --- true

Stop Your Kubernetes Cluster Upgrade Job

To cancel a running upgrade-clusters job, run the following PKS CLI command:

pks cancel-task TASKID

Where TASKID is the ID of the task that was returned when you ran pks upgrade-clusters.

Warning: pks cancel-task does not cancel cluster upgrades currently in progress. This command only cancels a job's pending cluster upgrades.

Limitations

If you upgrade the Enterprise PKS tile only and then attempt to resize or update your existing Kubernetes clusters, these operations fail until you upgrade the clusters.

If you want to upgrade the Enterprise PKS tile and your existing Kubernetes clusters simultaneously, do not disable Upgrade all clusters errand in the Enterprise PKS tile.