TKG Service clusters support a rolling update model. You can initiate a rolling update by changing the cluster specification. Some system operations may initiaite a rolling update. Before updating the environment you should familiarize yourself with the rolling update process.
Rolling Update Model for TKGS Clusters After TKG Service 3.0
As of TKG Service 3.0, the TKG Contoller is independent from vCenter Server and Supervisor. See Using the TKG Service. Uprgrading these components will not initaite a rolling update of TKGS clusters.
Upgrading the TKG Service version may trigger a rolling update of TKGS clusters.
Rolling Update Model for TKGS Clusters Prior to TKG Service 3.0
The TKG Controller runs on Supervisor. When you update Supervisor, the TKG Controller is automatically updated if an update is available. Each TKG Controller update can include updates for supporting services, such as CNI, CSI, CPI, as well as configuration updates for clusters. To support compatibility, the system performs pre-checks and enforces compliance.
vSphere IaaS control plane uses a rolling update model to update TKG clusters on Supervisor. The rolling update model ensures that there is minimal downtime during the cluster update process. Rolling updates include upgrading the Kubernetes software versions and the infrastructure and services supporting the clusters, such as virtual machine configurations and resources, services and namespaces, and custom resources. For the update to succeed, your configuration must meet several compatibility requirements, so the system enforces recheck conditions to ensure that clusters are ready for updates, and supports rollback if cluster upgrade is not successful.
You can initiate a rolling update of a TKG cluster by changing certain aspects of the cluster manifest. A rolling cluster update can also be initiated by the system. For example, when a vSphere Namespaces update is performed, the system immediately propagates updated configurations to all workload clusters. These updates can trigger a rolling update of cluster nodes. In addition, a change to any of the configuration elements can also initiate a rolling update. For example, renaming or replacing the VirtualMachineImage that corresponds to a distribution version initiates a rolling update as the system attempts to get all nodes running on the new image. Also, updating a Supervisor will likely trigger a rolling update of the workload clusters deployed there. For example, when the vmware-system-tkg-controller-manager is updated, the system introduces new values into the manifest generator and the controller initiates a rolling update to deploy those values.
The rolling update process for replacing the cluster nodes is similar to the rolling update of pods in a Kubernetes Deployment. There are two distinct controllers responsible for performing a rolling update of workload clusters: the Add-ons Controller and the Cluster controller. Within those two controllers there are three key stages to a rolling update: updating add-ons, updating the control plane, and updating the worker nodes. These stages occur in order, with pre-checks that prevent a step from beginning until the preceding step has sufficiently progressed. These steps might be skipped if they are determined to be unnecessary. For example, an update might only affect worker nodes and therefore not require any add-on or control plane updates.
During the update process, the system adds a new cluster node, and waits for the node to come online with the target Kubernetes version. The system then marks the old node for deletion, moves to the next node, and repeats the process. The old node is not deleted until all pods are removed. For example, if a pod is defined with PodDisruptionBudgets that prevent a node from being fully drained, the node is cordoned off but is not removed until those pods can be evicted. The system upgrades all control plane nodes first, then worker nodes. During an update, the cluster status changes to "updating". After the rolling update process completes, the cluster status changes to "running".
Pods running on a cluster that are not governed by a replication controller will be deleted during a Kubernetes version upgrade as part of the worker node drain during the cluster update. This is true if the cluster update is triggered manually or automatically by a vSphere Namespaces or Supervisor update. Pods not governed by a replication controller include pods that are not created as part of a Deployment or ReplicaSet spec. Refer to Pod Lifecycle: Pod lifetime in the Kubernetes documentation for more information.
User Initiated Rolling Updates
System Initiated Rolling Updates
- kubeadmcontrolplanetemplate/kubeadmcontrolplane
- kubeadmconfigtemplate/kubeadmconfig
- vspheremachinetemplate/vspheremachine (for vSphere 8.x)
- wcpmachinetemplate/wcpmachine (for vSphere 7.x)
In vSphere IaaS control plane, the TKG Controller that runs in Supervisor generates and keeps these objects in sync with the system code. This means when the controllers are updated to newer code, changes to any one of the above objects causes a rolling update of existing TKG clusters. In sum, system code changes affecting Supervisor result in rolling updates of TKG clusters.
| Upgrade Scenario | Description |
|---|---|
| Upgrading from any vCenter Server 7.x release to any vCenter Server release | May trigger a rolling update of all TKG clusters. A rolling update is triggered by the first upgrade of Supervisor following an upgrade of vCenter Server. A rolling update is usually not triggered by an upgrade of Supervisor on the same vCenter Server. Check the release notes for specific details. |
| Upgrading from any vCenter Server release to any vCenter Server 8.x release | Will trigger a rolling update of all TKG clusters because the following code changes need to be propagated:
|
| Upgrading from the vCenter Server 8.0 GA release (8.0.0) to the vCenter Server 8.0.0b or 8.0.0c releases | Will trigger a rolling update of specified TKG clusters if any of the following cases apply:
|
| Upgrading from the vSphere 8.0.0b release to the vSphere 8.0.0c release | No Automatic rollouts of workload clusters |
| Upgrading from the vSphere 8.0.0c release to the vSphere 8.0 Update 1 release (8.0.1) | No Automatic rollouts of workload clusters |
| Upgrading from any vSphere 8.x version to the 8.0 U2 release (8.0.2) | This will cause a rolling upgrade to all TKCs since the following changes need to happen:
|
| Upgrading from any vSphere 8.x versions less than 8.0 U2 (8.0.2) to the 8.0 U2c release | This will cause a rolling upgrade to all TKCs since the following changes need to happen:
|
For example, consider a scenario where you are using a Subscribed Content Library which automatically uses the system-defined OVA names. Then you switch to a Local Content Library and populate it with the same OVAs but give them different names. This will trigger a rolling update of all TKG clusters because the replacement Content Library has the same OVAs but with user-defined different names.
Rolling Update Considerations for Clusters with Multiple Nodepools
- Worker Nodepools
-
Worker nodepools were introduced with the TKGS v1alpha2 API which was released with vSphere 7 U3. Cluster API MachineDeployments is the underlying Kubernetes primitive of worker nodepools.
ClusterClass was introduced with the vSphere 8 release of TKGS. Both the v1alpha3 and the v1beta1 APIs are based on ClusterClass. (v1alpha3 is an abstraction layer on top of ClusterClass.)
- How Multiple Nodepools Are Updated During Rolling Update
-
When you update a TKGS workload cluster provisioned with multiple nodepools, the rolling update model differs depending on which version of vSphere that is being used.
vSphere TKGS API Upgrade Behavior vSphere 7 TKGS v1alpha2 API Multiple nodepools within the same cluster are updated at the same time (simultaneously) vSphere 8 TKGS v1alpha3 API and v1beta1 API Multiple nodepools within the same cluster are updated following a logical order (sequentially)
- Best Practice Considerations
-
Provisioning a vSphere 8 TKGS cluster with multiple nodepools that are identical serves no purpose in terms of sizing. Nodepools should be used for different sizes, VM classes, TKr versions, etc. Avoid using multiple identical nodepools to game the system and upgrade clusters quicker because it will not work.
Pod Disruption Budgets are the proper way to make sure upgrades do not interfere with running applications. The best way to handle this is to set PodDisruptionBudgets on the workloads (see https://kubernetes.io/docs/tasks/run-application/configure-pdb/ ). Cluster API respects these, and will not terminate a machine if the thresholds would be exceeded.
- Rolling Update Details for vSphere 8 TKGS Clusters
-
During a vSphere 8 TKGS cluster version update:
- Control Plane nodes are updated first, then worker nodes are rolled one worker node at a time starting with Zone-A nodepool. If two nodepools are used, there will be only 1 worker rolled out at a time.
