When you deploy vSphere Container Storage Plug-in in a vSphere environment that includes multiple data centers or host clusters, you can use zoning. Zoning enables orchestration systems, such as Kubernetes, to integrate with vSphere storage resources that are not equally available to all nodes. With zoning, the orchestration system can make intelligent decisions when dynamically provisioning volumes. The intelligent volume provisioning helps you avoid situations such as those where a pod cannot start because the storage resource it needs is not accessible.
Guidelines and Best Practices for Deployment with Topology
- If you already use vSphere Container Storage Plug-in to run applications, but haven't used the topology feature, you must re-create the entire cluster and delete any existing PVCs in the system to be able to use the topology feature.
- Depending on your vSphere storage environment, you can use different deployment scenarios for availability zones. For example, you can define availability zones per host, host cluster, data center or have a combination of these.
- Kubernetes assumes that even though the topology labels applied on a node are mutable, a node will not move between zones without being destroyed and re-created. See https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone. As a result, if you define the availability zones on the host level, make sure to pin the node VMs to their respective hosts to prevent migration of these VMs to other availability zones. You can do this by either creating the node VM on the hosts' local datastore or by setting the DRS VM-Host affinity rules. For information, see VM-Host Affinity Rules.
An exception to this guideline is an active-passive setup that has storage replicated between two topology domains as specified in the diagram in Deploy Workloads on a Preferential Datastore in a Topology-Aware Environment. In this case, you can migrate node VMs temporarily when either of the topology domains is down.
- Distribute your control plane VMs across the availability zones to ensure high availability.
- Have at least one worker node VM in each availability zone. Following this guideline is helpful when you use a StorageClass with no topology requirements explicitly set, and Kubernetes randomly selects a topology domain to schedule a pod in it. In such cases, if the topology domain does not have a worker node, the pod remains in pending state.
- Do not apply topology related vSphere tags to node VMs. vSphere Container Storage Plug-in cannot read topology labels applied on the nodes.
- Use the
topology-categoriesparameter in the Labels section of the vSphere configuration file. This parameter adds topology custom labels to the nodes:topology.csi.vmware.com/category-name.vSphere Container Storage Plug-in of releases earlier than 2.4 does not support the
topology-categoriesparameter.vSphere Secret Configuration Sample Labels on a Node [Labels] topology-categories = "k8s-region,k8s-zone"
Name: k8s-node-0179 Roles: <none> Labels: topology.csi.vmware.com/k8s-region=region-1 topology.csi.vmware.com/k8s-zone=zone-a Annotations: .... - Each node VM in a topology-aware Kubernetes cluster must belong to a tag under each category mentioned in the
topology-categoriesparameter in Step 2.a. Node registration fails if a node does not belong to every category under thetopology-categoriesparameter. - With vSphere Container Storage Plug-in version 3.1.0 and later, volume provisioning requests in topology-aware environments attempt to create volumes in datastores accessible to all hosts under a given topology segment. This includes hosts that do not have Kubernetes node VMs running on them. For example, if the vSphere Container Storage Plug-in driver gets a request to provision a volume in
zone-a, applied on the Datacenterdc-1, all hosts underdc-1must have access to the datastore selected for volume provisioning. The hosts include those that are directly underdc-1and those that are a part of clusters insidedc-1.
Sample vCenter Server Topology Configuration
In the following example, the vCenter Server environment consists of four clusters across two regions. Availability zones are defined per data center as well as per host cluster.
| Data Center | Cluster | Node VM |
|---|---|---|
Datacenter1
|
Cluster1
|
ControlPlaneVM1 WorkerNodeVM1 |
Cluster2
|
ControlPlaneVM2 WorkerNodeVM2 WorkerNodeVM3 |
|
Cluster3
|
ControlPlaneVM3 WorkerNodeVM4 |
|
Datacenter2
|
Cluster4
|
WorkerNodeVM5 WorkerNodeVM6 |
Deploy vSphere Container Storage Plug-in with Topology
Use this task for a greenfield deployment of vSphere Container Storage Plug-in with topology.
Prerequisites
- With vSphere Container Storage Plug-in version 3.0.2, you can transition from a non-topology configuration to a topology configuration if there are no Persistent Volumes (PVs) in the cluster. However, you cannot migrate non-topology setups with PVs to topology setups
- Using vSphere Container Storage Plug-in version 3.0.2, you can migrate non-topology setups to topology setups without managing internal topology CRs. This simplifies the transition process if you have already deployed a non-topology setup and want to enable topology before deploying any workload on the cluster.
- You cannot transition back from a topology setup to a non-topology setup.
- Have appropriate tagging privileges that control your ability to work with tags.
- Ancestors of node VMs, such as a host, cluster, folder, and data center, must have the ReadOnly role set for the vSphere user configured to use vSphere Container Storage Plug-in. This is required to allow reading tags and categories to discover the nodes' topology. For more information, see vSphere Tagging Privileges in the vSphere Security documentation.
