Deploying vSphere Container Storage Plug-in with Multiple vCenter Server Instances

To provide higher availability, performance, and scalability, vSphere Container Storage Plug-in supports deployments with multiple vCenter Server instances.

In a Kubernetes cluster based on a single vCenter Server system, vSphere Container Storage Plug-in is as highly available as vCenter Server. If vCenter Server fails, vSphere Container Storage Plug-in stops volume operations. In addition, the performance and throughput of volume life cycle operations and the scale of volumes are limited to what a single vCenter Server instance supports.

With the multi-instance vCenter Server functionality, you can stretch the Kubernetes cluster across multiple vCenter Server instances. This allows you to achieve higher availability, performance, and scale of persistent volumes. Also, in a multi-zone infrastructure topology, you can deploy one instance of vCenter Server per availability zone, or fault domain. You can then stretch the Kubernetes cluster across the availability zones for higher availability, performance, and scale of persistent volumes.

Advantages of Deployment with Multiple vCenter Server Instances

Deployments with multiple vCenter Server instances offer the following advantages:

Improved availability. In a multi-zone deployment topology, if an availability zone fails, the failure affects volume life cycle operations in only that particular availability zone. The life cycle operations in other availability zones continue.
Improved performance. In a Kubernetes deployment stretched across multiple vCenter Server instances, vSphere Container Storage Plug-in has more vCenter Server systems available for performing volume operations. As a result, the volume operation throughput increases.
Improved scale. A single vCenter Server instance supports a maximum of 10k CNS block volumes. In a Kubernetes deployment stretched across multiple vCenter Server instances, vSphere Container Storage Plug-in is able to support 10k CNS block volumes per vCenter Server.

Guidelines for Deployment with Multiple vCenter Server Instances

When you deploy vSphere Container Storage Plug-in in an environment with multiple vCenter Server instances, follow these guidelines and best practices.

The deployment supports only native Kubernetes clusters. The deployment does not support Tanzu Kubernetes Grid clusters.
The deployment supports only block volume provisioning. File volume provisioning is not supported.
Your deployment must be topology-aware. If you haven't used the topology functionality, you must recreate the entire cluster to be topology-aware. During volume provisioning, the topology mechanism helps identify and specify nodes for selecting vSphere storage resources spread across multiple vCenter Server systems.
Follow all guidelines for deploying vSphere Container Storage Plug-in with topology. For information, see Deploying vSphere Container Storage Plug-in with Topology.
Every node VM on each vCenter Server must have a tag. The value of the tag yields a unique combination of values across all categories mentioned in the topology-categories parameter. The topology-categories parameter is specified in vSphere configuration secret on associated vCenter Server.
Node registration fails if the node does not belong to every category under the topology-categories parameter. Or if the tag values do not generate unique combination across the different tag categories in associated vCenter Server.
vSphere Container Storage Plug-in version 3.0 does not support datastores shared across multiple vCenter Server instances.
To provision storage based on any specific storage policy, configure the storage policy for each individual vCenter Server, so that vCenter Server can follow the policy-based provisioning requirements. Make sure to use the same policy name and the same policy parameters on all participatingvCenter Server systems.
Specify configuration details of all configured vCenter Server instances under a separate VirtualCenter section in the vSphere configuration file.

Example of Topology with Multiple vCenter Server Instances

In the following example, three vCenter Server instances belong to three zones.


vCenter Server	Cluster	Node VM
vCenter Server - 1 Availability Zone Category: k8s-zone Tag: zone-1	vSphere Cluster	ControlPlaneVM1 WorkerNodeVM1 WorkerNodeVM2
vCenter Server - 2 Availability Zone Category: k8s-zone Tag: zone-2	vSphere Cluster	ControlPlaneVM2 WorkerNodeVM3 WorkerNodeVM4
vCenter Server - 3 Availability Zone Category: k8s-zone Tag: zone-3	vSphere Cluster	ControlPlaneVM3 WorkerNodeVM5 WorkerNodeVM6

Install vSphere Cloud Provider Interface in an Environment with Multiple vCenter Server Instances

Follow these steps to install the vSphere Cloud Provider Interface in an environment with multiple vCenter Server instances.

Procedure

Download the vsphere-cloud-controller-manager.yaml file.
$ wget https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/release-VERSION/releases/vVERSION/vsphere-cloud-controller-manager.yaml

Replace VERSION with an appropriate major version of Kubernetes. For example, if the version is 1.22.x, run the following command:

$ wget https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/release-1.22/releases/v1.22/vsphere-cloud-controller-manager.yaml

Modify the downloaded vsphere-cloud-controller-manager.yaml file to add information about all vCenter Server instances in the vcenter section.

Use the following as an example.

apiVersion: v1
kind: Secret
metadata:
  name: vsphere-cloud-secret
  labels:
    vsphere-cpi-infra: secret
    component: cloud-controller-manager
  namespace: kube-system
  # NOTE: this is just an example configuration, update with real values based on your environment
stringData:
  VC-1-IP.username: "username"
  VC-1-IP.password: "password"
  VC-2-IP.username: "username"
  VC-2-IP.password: "password"
  VC-3-IP.username: "username"
  VC-3-IP.password: "password"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vsphere-cloud-config
  labels:
    vsphere-cpi-infra: config
    component: cloud-controller-manager
  namespace: kube-system
data:
  # NOTE: this is just an example configuration, update with real values based on your environment
  vsphere.conf: |
    # Global properties in this section will be used for all specified vCenters unless overriden in VirtualCenter section.
    global:
      port: 443
      # set insecureFlag to true if the vCenter uses a self-signed cert
      insecureFlag: true
      # settings for using k8s secret
      secretName: vsphere-cloud-secret
      secretNamespace: kube-system

    # vcenter section
    vcenter:
      vc-1:
        server: <VC-1-IP or VC-1-FQDN>
        user: <username>
        password: <password>
        datacenters:
          - datacenter-path
      vc-2:
        server: <VC-2-IP or VC-2-FQDN>
        user: <username>
        password: <password>
        datacenters:
          - datacenter-path
      vc-3:
        server: <VC-3-IP or VC-3-FQDN>
        user: <username>
        password: <password>
        datacenters:
          - datacenter-path

Install the vSphere Cloud Provider Interface.
Follow steps in Install vSphere Cloud Provider Interface.

Deploy vSphere Container Storage Plug-in with Multiple vCenter Server Instances

To deploy vSphere Container Storage Plug-in in an environment with multiple vCenter Server instances, follow this procedure.

Prerequisites

Make sure that vSphere Container Storage Plug-in is of version 3.0 or later.

Procedure

In the vSphere configuration file, add configuration details for all instances of vCenter Server under the VirtualCenter section.

For information about the configuration file, see Create a Kubernetes Secret for vSphere Container Storage Plug-in.

Use the following configuration file as an example for provisioning block volumes in a vSphere environment with two instances of vCenter Server .

$ cat /etc/kubernetes/csi-vsphere.conf
[Global]
cluster-id = "<cluster-id>"
cluster-distribution = "<cluster-distribution>"

[VirtualCenter "<VC-1-IP or VC-1-FQDN>"]
insecure-flag = "<true or false>"
user = "<username>"
password = "<password>"
port = "<port>"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
ca-file = <ca file path> # optional, use with insecure-flag set to false
thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file

[VirtualCenter "<VC-2-IP or VC-2-FQDN>"]
insecure-flag = "<true or false>"
user = "<username>"
password = "<password>"
port = "<port>"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
ca-file = <ca file path> # optional, use with insecure-flag set to false
thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file

[VirtualCenter "<VC-3-IP or VC-3-FQDN>"]
insecure-flag = "<true or false>"
user = "<username>"
password = "<password>"
port = "<port>"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
ca-file = <ca file path> # optional, use with insecure-flag set to false
thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file


[Labels]
topology-categories = "k8s-zone"

Deploy topology-aware vSphere Container Storage Plug-in.
For information about enabling topology when deploying vSphere Container Storage Plug-in, see Deploy vSphere Container Storage Plug-in with Topology.
After the installation, verify the topology-aware setup with multiple vCenter Server instances.
Check that the driver pods of vSphere Container Storage Plug-in are up and running.