Upgrade an Internet-Restricted AWS Deployment

This topic describes how to upgrade management and workload clusters to Tanzu Kubernetes Grid v1.6 in an internet-restricted Amazon Web Services (AWS) environment.

This procedure is required for both minor v1.5.x to v1.6.x and patch v1.6.x to v1.6.y upgrades.

Important: All clusters use one-year client certificates and upgrading a cluster renews its certificate, so VMware recommends upgrading your clusters at least once a year.

Prerequisites

To upgrade an internet-restricted Tanzu Kubernetes Grid deployment on AWS to TKG v1.6.x, you must have:

  • Tanzu Kubernetes Grid v1.4 or later: You cannot upgrade to Tanzu Kubernetes Grid v1.6 from v1.4 or earlier versions.
    You must first upgrade to a v1.5 version as described in Upgrade Tanzu Kubernetes Grid in the Tanzu Kubernetes Grid v1.5 documentation.
  • An internet-connected Linux machine
  • An offline Linux machine
  • Tanzu CLI v0.25.0 with compatible plugins and kubectl installed on both Linux machines. See Install the CLI and Other Tools.
    • Run tanzu version to check that you are running a v1.6 version of tanzu. Tanzu Kubernetes Grid v1.6.0 uses Tanzu CLI v0.25.0, based on Tanzu Framework v0.25.0.
    • Run kubectl version to check that you are running the kubectl version listed in the Tanzu Kubernetes Grid downloads page for your v1.6.x version.
    • Run tanzu plugin list to confirm v0.25.0 versions of the plugins management-cluster, cluster, login, kubernetes-release, package, and pinniped-auth.
    • For the offline machine, download, transfer, and install the Tanzu CLI bundle and other software following your usual process for that machine.

Step 1: Copy the Images

From your internet-connected machine, copy the container images used by Tanzu Kubernetes Grid into your offline registry. The procedure depends on whether your offline environment is proxied or physically airgapped.

  • Proxied environment:

    1. Run the gen-publish-images.sh script by following Step 2: Generate the images-copy-list File in the Prepare an Internet-Restricted Environment topic.

      • The gen-publish-images.sh script generates an images-copy-list file from the latest Bill of Materials (BoM) files.
      • If you have an images-copy-list or publish-images.sh file from a previous deployment, you must still run gen-publish-images.sh to regenerate the images-copy-list so that it includes the latest component versions.
    2. Run the download-images.sh script on the images-copy-list file by following Step 3: Run the download-images Script in the Prepare an Internet-Restricted Environment topic.

      • Running download-images.sh pulls the correct component versions for your Tanzu Kubernetes Grid version and saves them to your local private Docker registry.
  • Physically airgapped environment:

Step 2: Upgrade Management Clusters

To upgrade Tanzu Kubernetes Grid, you must first upgrade all management clusters in your deployment. You cannot upgrade workload clusters until you have upgraded their management clusters. Follow the procedure below to upgrade the management clusters.

  1. Run the tanzu login command to see an interactive list of management clusters available for upgrade.

  2. Select the management cluster that you want to upgrade. See List Management Clusters and Change Context for more information.

    tanzu login --kubeconfig /root/.kube/config --context mgmt-cluster@mgmt-cluster --name mgmt-cluster
    
  3. Run the tanzu cluster list command with the --include-management-cluster option.

    tanzu cluster list --include-management-cluster
    

    This command shows the versions of Kubernetes running on the management cluster and all of the clusters that it manages:

    $ tanzu cluster list --include-management-cluster
     NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES          ROLES       PLAN    TKR
     k8s-1-21-8-cluster   default     running   1/1           1/1      v1.21.8+vmware.1    <none>      dev     v1.21.8---vmware.1-tkg.1
     k8s-1-22-9-cluster   default     running   1/1           1/1      v1.22.5+vmware.1    <none>      dev     v1.22.9---vmware.1-tkg.1
     mgmt-cluster         tkg-system  running   1/1           1/1      v1.22.9+vmware.1    management  dev     v1.22.9---vmware.1-tkg.1
    
  4. Set the following environment variables so the Tanzu CLI can access your private image registry and know which region and zone to deploy to. Find the access key settings in your ~/.aws/config file.

     export TKG_CUSTOM_IMAGE_REPOSITORY=
     export TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE=
     export AWS_REGION=
     export AWS_NODE_AZ=
     export AWS_ACCESS_KEY_ID=
     export AWS_SECRET_ACCESS_KEY=
     export AWS_SSH_KEY_NAME=
    
  5. Run the tanzu mc upgrade command and enter y to confirm.

    The following command upgrades the current management cluster.

    tanzu mc upgrade
    

    If multiple base VM images in your IaaS account have the same version of Kubernetes that you are upgrading to, use the --os-name option to specify the OS you want. See Selecting an OS During Cluster Upgrade for more information.

    For example, on vSphere if you have uploaded both Photon and Ubuntu OVA templates with Kubernetes v1.23.8, specify --os-name ubuntu to upgrade your management cluster to run on an Ubuntu VM.

    tanzu mc upgrade --os-name ubuntu
    

    To skip the confirmation step when you upgrade a cluster, specify the --yes option.

    tanzu mc upgrade --yes
    

    The upgrade process first upgrades the Cluster API providers for vSphere, AWS, or Azure that are running in the management cluster. Then, it upgrades the version of Kubernetes in all of the control plane and worker nodes of the management cluster.

    If the upgrade times out before it completes, run tanzu mc upgrade again and specify the --timeout option with a value greater than the default of 30 minutes.

    tanzu mc upgrade --timeout 45m0s
    

    Note: After you have installed the v1.6 CLI but before a management cluster has been upgraded, all context-specific CLI command groups (tanzu cluster, tanzu kubernetes-release) plus all of the management-cluster plugin commands except for tanzu mc upgrade and tanzu mc create are unavailable and not included in Tanzu CLI --help output.

  6. When the upgrade finishes, run the tanzu cluster list command with the --include-management-cluster option again to check that the management cluster has been upgraded.

    tanzu cluster list --include-management-cluster
    

    You see that the management cluster is now running the new version of Kubernetes, but that the workload clusters are still running previous versions of Kubernetes.

     NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN     TKR
     k8s-1-21-8-cluster   default     running   1/1           1/1      v1.21.8+vmware.1   <none>       dev     v1.21.8---vmware.1-tkg.1
     k8s-1-22-9-cluster   default     running   1/1           1/1      v1.22.9+vmware.1   <none>       dev     v1.22.9---vmware.1-tkg.1
     mgmt-cluster         tkg-system  running   1/1           1/1      v1.23.8+vmware.1   management   dev     v1.23.8---vmware.1-tkg.1
    
  7. Run tanzu config server list to confirm that all management cluster configurations have migrated into the ~/.config/tanzu. After confirmation, you can delete the ~/.tanzu folder.

Step 3: Upgrade Workload Clusters

After you upgrade the management clusters in your deployment, you can upgrade the workload clusters that are managed by those management clusters.

Follow the procedure below to upgrade the workload clusters that are running your workloads.

The upgrade process upgrades the version of Kubernetes in all of the control plane and worker nodes of your workload clusters.

  1. Run the tanzu login command to see an interactive list of available management clusters.

  2. Select a management cluster to switch the context of the Tanzu CLI to the management cluster that manages the clusters you want to upgrade. See List Management Clusters and Change Context for more information.

    tanzu login --kubeconfig /root/.kube/config --context mgmt-cluster@mgmt-cluster --name mgmt-cluster
    
  3. Run the tanzu cluster list command with the --include-management-cluster option.

    tanzu cluster list --include-management-cluster
    

    The tanzu cluster list command shows the version of Kubernetes that is running in the management cluster and all of the clusters that it manages. In this example, you can see that the management cluster has already been upgraded to v1.23.8, but the workload clusters are running older versions of Kubernetes.

     NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN     TKR
     k8s-1-21-8-cluster   default     running   1/1           1/1      v1.21.8+vmware.1   <none>      dev      v1.21.8---vmware.1-tkg.1
     k8s-1-22-9-cluster   default     running   1/1           1/1      v1.22.9+vmware.1   <none>      dev      v1.22.9---vmware.1-tkg.1
     mgmt-cluster         tkg-system  running   1/1           1/1      v1.23.8+vmware.1   management  dev      v1.23.8---vmware.1-tkg.1
    
  4. To discover which versions of Kubernetes are made available by a management cluster, run the tanzu kubernetes-release get command.

    tanzu kubernetes-release get
    

    The output lists all of the versions of Kubernetes that you can use to deploy clusters, with the following notes: - COMPATIBLE: The current management cluster can deploy workload clusters with this Tanzu Kubernetes release (tkr). - UPGRADEAVAILABLE: This tkr is not the most current in its Kubernetes version line. Any workload clusters running this tkr version can be upgraded to newer versions.

    For example:

     NAME                        VERSION                   COMPATIBLE  UPGRADEAVAILABLE
     v1.21.11---vmware.1-tkg.1   v1.21.11+vmware.1-tkg.1   True        True
     v1.21.14---vmware.1-tkg.1   v1.21.14+vmware.1-tkg.1   True        False
     v1.22.8---vmware.1-tkg.1    v1.22.8+vmware.1-tkg.1    True        True
     v1.22.9---vmware.1-tkg.1    v1.22.9+vmware.1-tkg.1    True        True
     v1.22.11---vmware.1-tkg.1   v1.22.11+vmware.1-tkg.1   True        False
     v1.23.8---vmware.1-tkg.1    v1.23.8+vmware.1-tkg.1    True        False
    
  5. To discover the newer tkr versions to which you can upgrade a workload cluster running an older tkr version, run the tanzu kubernetes-release available-upgrades get command, specifying the current version of the cluster.

    tanzu kubernetes-release available-upgrades get v1.22.8---vmware.1-tkg.1
    

    This command lists all of the available Kubernetes versions to which you can upgrade clusters that are running the specified version.

    NAME                       VERSION
    v1.22.9---vmware.1-tkg.1   v1.22.9+vmware.1-tkg.1
    v1.22.11---vmware.1-tkg.1  v1.22.11+vmware.1-tkg.1
    v1.23.8---vmware.1-tkg.1   v1.23.8+vmware.1-tkg.1
    

    You can also discover the tkr versions that are available for a specific workload cluster by specifying the cluster name in the tanzu cluster available-upgrades get command.

    tanzu cluster available-upgrades get k8s-1-22-8-cluster
    

    This command lists all of the Kubernetes versions that are compatible with the specified cluster.

    NAME                         VERSION                            COMPATIBLE
    v1.22.9---vmware.1-tkg.1     v1.22.9+vmware.1-tkg.1             True
    v1.22.11---vmware.1-tkg.1    v1.22.11+vmware.1-tkg.1            True
    v1.23.8---vmware.1-tkg.1     v1.23.8+vmware.1-tkg.1             True
    

    You cannot skip minor versions when upgrading your tkr version. For example, you cannot upgrade a cluster directly from v1.21.x to v1.23.x. You must upgrade a v1.21.x cluster to v1.22.x before upgrading the cluster to v1.23.x. Ensure that you have copied the required images to your private registry for the selected tkr version using download-images.sh script with the generated images-copy-list file before upgrading the tanzu cluster.

  6. Run the tanzu cluster upgrade CLUSTER-NAME command and enter y to confirm.

    To upgrade the cluster to the default version of Kubernetes for this release of Tanzu Kubernetes Grid, run the tanzu cluster upgrade command without any options. For example, the following command upgrades the cluster k8s-1-22-9-cluster from v1.22.9 to v1.23.8.

    tanzu cluster upgrade k8s-1-22-9-cluster
    

    If the cluster is not running in the default namespace, specify the --namespace option.

    tanzu cluster upgrade CLUSTER-NAME --namespace NAMESPACE-NAME
    

    To skip the confirmation step when you upgrade a cluster, specify the --yes option.

    tanzu cluster upgrade CLUSTER-NAME --yes
    

    If an upgrade times out before it completes, run tanzu cluster upgrade again and specify the --timeout option with a value greater than the default of 30 minutes.

    tanzu cluster upgrade CLUSTER-NAME --timeout 45m0s
    

    If multiple base VM images in your IaaS account have the same version of Kubernetes that you are upgrading to, use the --os-name option to specify the OS you want. See Selecting an OS During Cluster Upgrade for more information.

    For example, on vSphere if you have uploaded both Photon and Ubuntu OVA templates with Kubernetes v1.23.8, specify --os-name ubuntu to upgrade your workload cluster to run on an Ubuntu VM.

    tanzu cluster upgrade CLUSTER-NAME --os-name ubuntu
    

    Since you cannot skip minor versions of tkr, the upgrade command fails if you try to upgrade a cluster that is more than one minor version behind the default version. For example, you cannot upgrade directly from v1.21.x to v1.23.x. To upgrade a cluster to a version of Kubernetes that is not the default version for this release of Tanzu Kubernetes Grid, specify the --tkr option with the NAME of the chosen version, as listed by tanzu kubernetes-release get above. For example, to upgrade the cluster k8s-1-21-8-cluster from v1.21.8 to v1.22.11.

    tanzu cluster upgrade k8s-1-21-8-cluster --tkr v1.22.11---vmware.1-tkg.1 --yes
    
  7. When the upgrade finishes, run the tanzu cluster list command with the --include-management-cluster option again, to check that the workload cluster has been upgraded.

    tanzu cluster list --include-management-cluster
    

    You see that the k8s-1-21-8-cluster and k8s-1-22-9-cluster workload clusters are now running Kubernetes v1.21.8 and v1.23.8 respectively.

     NAME                 NAMESPACE   STATUS    CONTROLPLANE  WORKERS  KUBERNETES         ROLES       PLAN     TKR
     k8s-1-21-8-cluster   default     running   1/1           1/1      v1.22.11+vmware.1  <none>      dev      v1.22.11---vmware.1-tkg.1
     k8s-1-22-9-cluster   default     running   1/1           1/1      v1.23.8+vmware.1   <none>      dev      v1.23.8---vmware.1-tkg.1
     mgmt-cluster         tkg-system  running   1/1           1/1      v1.23.8+vmware.1   management  dev      v1.23.8---vmware.1-tkg.1
    

What to Do Next

You can now continue to use the Tanzu CLI to manage your clusters and run your applications with the new version of Kubernetes in your internet-restricted environment.

check-circle-line exclamation-circle-line close-line
Scroll to top icon