When you deploy vSphere Container Storage Plug-in in a vSphere environment that includes multiple data centers or host clusters, you can use zoning. Zoning enables orchestration systems, such as Kubernetes, to integrate with vSphere storage resources that are not equally available to all nodes. With zoning, the orchestration system can make intelligent decisions when dynamically provisioning volumes. The intelligent volume provisioning helps you avoid situations such as those where a pod cannot start because the storage resource it needs is not accessible.
Guidelines and Best Practices for Deployment with Topology
- If you already use vSphere Container Storage Plug-in to run applications, but haven't used the topology feature, you must re-create the entire cluster and delete any existing PVCs in the system to be able to use the topology feature.
- Depending on your vSphere storage environment, you can use different deployment scenarios for availability zones. For example, you can define availability zones per host, host cluster, data center or have a combination of these.
- Kubernetes assumes that even though the topology labels applied on a node are mutable, a node will not move between zones without being destroyed and re-created. See https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone. As a result, if you define the availability zones on the host level, make sure to pin the node VMs to their respective hosts to prevent migration of these VMs to other availability zones. You can do this by either creating the node VM on the hosts' local datastore or by setting the DRS VM-Host affinity rules. For information, see VM-Host Affinity Rules.
An exception to this guideline is an active-passive setup that has storage replicated between two topology domains as specified in the diagram in Deploy Workloads on a Preferential Datastore in a Topology-Aware Environment. In this case, you can migrate node VMs temporarily when either of the topology domains is down.
- Distribute your control plane VMs across the availability zones to ensure high availability.
- Have at least one worker node VM in each availability zone. Following this guideline is helpful when you use a StorageClass with no topology requirements explicitly set, and Kubernetes randomly selects a topology domain to schedule a pod in it. In such cases, if the topology domain does not have a worker node, the pod remains in pending state.
- Do not apply topology related vSphere tags to node VMs. vSphere Container Storage Plug-in cannot read topology labels applied on the nodes.
- Each node VM in a topology-aware Kubernetes cluster must belong to a tag under each category mentioned in the
topology-categoriesparameter in Step 2.a. Node registration fails if a node does not belong to every category under thetopology-categoriesparameter. - Use the
topology-categoriesparameter in the Labels section of the vSphere configuration file. This parameter adds topology custom labels to the nodes:topology.csi.vmware.com/category-name.vSphere Container Storage Plug-in of releases earlier than 2.4 does not support the
topology-categoriesparameter.vSphere Secret Configuration Sample Labels on a Node [Labels] topology-categories = "k8s-region,k8s-zone"
Name: k8s-node-0179 Roles: <none> Labels: topology.csi.vmware.com/k8s-region=region-1 topology.csi.vmware.com/k8s-zone=zone-a Annotations: ....
Sample vCenter Server Topology Configuration
In the following example, the vCenter Server environment consists of four clusters across two regions. Availability zones are defined per data center as well as per host cluster.
| Data Center | Cluster | Node VM |
|---|---|---|
Datacenter1
|
Cluster1
|
ControlPlaneVM1 WorkerNodeVM1 |
Cluster2
|
ControlPlaneVM2 WorkerNodeVM2 WorkerNodeVM3 |
|
Cluster3
|
ControlPlaneVM3 WorkerNodeVM4 |
|
Datacenter2
|
Cluster4
|
WorkerNodeVM5 WorkerNodeVM6 |
Deploy vSphere Container Storage Plug-in with Topology
Use this task for a greenfield deployment of vSphere Container Storage Plug-in with topology.
Prerequisites
- Have appropriate tagging privileges that control your ability to work with tags.
- Ancestors of node VMs, such as a host, cluster, folder, and data center, must have the ReadOnly role set for the vSphere user configured to use vSphere Container Storage Plug-in. This is required to allow reading tags and categories to discover the nodes' topology. For more information, see vSphere Tagging Privileges in the vSphere Security documentation.
