When preparing your native Kubernetes environment for installation of the vSphere Container Storage Plug-in, create a Kubernetes secret that contains configuration details to connect to vSphere.

Before installing the vSphere Container Storage Plug-in on a native Kubernetes cluster, create a configuration file that contains details to connect to vSphere. The default file for the configuration details is the csi-vsphere.conf file. If you prefer to use a file with another name, change the environment variable VSPHERE_CSI_CONFIG in the deployment YAMLs. For more information, see Install the vSphere Container Storage Plug-in.

For information about topology-aware deployments, see Deploy vSphere Container Storage Plug-in with Topology.

For information about deployments with multiple vCenter Server instances, see Deploying vSphere Container Storage Plug-in with Multiple vCenter Server Instances.

Procedure

  1. Create a vSphere configuration file for block volumes or file volumes.
    • Block volumes.

      vSphere configuration file for block volumes includes the following sample entries.

      $ cat /etc/kubernetes/csi-vsphere.conf
[Global]
cluster-id = "<cluster-id>"
cluster-distribution = "<cluster-distribution>"
ca-file = <ca file path> # optional, use with insecure-flag set to false
thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file

[VirtualCenter "<IP or FQDN>"]
insecure-flag = "<true or false>"
user = "<username>"
password = "<password>"
port = "<port>"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
      The entries have the following meanings.
      Block Volume Parameter Description
      cluster-id
      • The unique cluster identifier. Each Kubernetes cluster must contain a unique cluster-id set in the configuration file. The cluster ID cannot not exceed 64 characters. Use only alphanumeric characters, period (.), or hyphen (-).
      • This parameter is optional from vSphere Container Storage Plug-in version 3.0 or later. If you do not enter a cluster ID, vSphere Container Storage Plug-in internally generates a unique cluster ID. For more information, see Automatic Generation of Cluster IDs in vSphere Container Storage Plug-in.
      cluster-distribution The distribution of the Kubernetes cluster. This parameter is optional. Examples are Openshift, Anthos, and TKGI. When you enter values for this parameter, keep in mind the following:
      • vSphere Container Storage Plug-in controller goes into CrashLoopBackOff state when you enter values with special character \r.
      • When you enter values exceeding 128 characters, the PVC creation might be struck in Pending state.
        Note: This field will be marked as mandatory in vSphere Container Storage Plug-in version 3.2.0.
      VirtualCenter The section defines such parameters as the vCenter Server IP address and FQDN.
      insecure-flag Takes the following values:
      user The vCenter Server username. You must specify the username along with the domain name. For example, user = "userName@domainName" or user = "domainName\\username". If you don't specify the domain name for active directory users, the vSphere Container Storage Plug-in will not function properly.
      password Password for a vCenter Server user.
      Note: If the password value contains Golang escape characters, such as a backslash or double quote, make sure to represent them correctly.

      For example, if your vCenter Server password is pwd\123, the configuration file's password key-value should be password = "pwd\\123".

      Other examples:

      vCenter Server Password Configuration File Value
      pwd\nWithNewLine password = "pwd\\nWithNewLine"
      Admin!23\'" password = "Admin!23\\\'\""
      port vCenter Server port. The default is 443.
      ca-file The path to a CA certificate in PEM format. This is an optional parameter.
      Thumbprint The certificate thumbprint. It is an optional parameter. It is ignored when you are using an unsecured setup or when you provide ca-file.
      datacenters List of all comma separated datacenter paths where Kubernetes node VMs are present. Provide the name of the datacenter when it is located at the root. When it is placed in the folder, you need to specify the path as folder/datacenter-name. The datacenter name cannot contain a comma since it is used as a delimiter.
      migration-datastore-url If you use vSphere Container Storage Plug-in version 3, add this parameter when you migrate in-tree vSphere volumes to vSphere Container Storage Plug-in. The parameter allows to honor the default datastore feature of the in-tree vSphere plug-in.
      Note: To deploy the vSphere Container Storage Plug-in for block volumes in VMware Cloud environment, you must enter the cloud administrator username and password in the vSphere configuration file.
    • File volumes.

      For file volumes, it is optional to add parameters that specify network permissions and placement of volumes. Otherwise, default values will be used. Use the following configuration file as an example.

      $ cat /etc/kubernetes/csi-vsphere.conf
[Global]
cluster-id = "<cluster-id>"
cluster-distribution = "<cluster-distribution>"
ca-file = <ca file path> # optional, use with insecure-flag set to false

[NetPermissions "A"]
ips = "*"
permissions = "READ_WRITE"
rootsquash = false

[NetPermissions "B"]
ips = "10.20.20.0/24"
permissions = "READ_ONLY"
rootsquash = true

[NetPermissions "C"]
ips = "10.30.30.0/24"
permissions = "NO_ACCESS"

[NetPermissions "D"]
ips = "10.30.10.0/24"
rootsquash = true

[NetPermissions "E"]
ips = "10.30.1.0/24"

[VirtualCenter "<IP or FQDN>"]
insecure-flag = "<true or false>"
user = "<username>"
password = "<password>"
port = "<port>"
datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
      The entries have the following meanings.
      File Volume Parameter Description
      NetPermissions This parameter is exclusive to file volumes and is optional. In this sample vSphere configuration file, the set of parameters restricts the network capabilities of all file share volumes that are created. If you do not specify the complete set of NetPermissions for a given IP range or completely omit the section, the system uses default values. You can define as many NetPermissions sections as you want. Each section can include the following strings:

      • Ips: Defines the IP range or IP subnet to which these restrictions apply. The default value for Ips is *, which means all IPs.

      • Permissions: Defines the permissions level, such as READ_WRITE, READ_ONLY or NO_ACCESS. The default value for Permissions is READ_WRITE for the specified IP range.

      • RootSquash: Defines the security access level for the file share volume. The default for RootSquash is false. It allows root access to all file share volumes that are created within the specified IP range.

      Note: Do not use "NO_ACCESS" permissions for IPs "*" or the subnets of the node IPs in the Kubernetes cluster. Otherwise, the volume created with this network permissions cannot be used with the pod. Volume mount will fail with the error Internal desc = error publish volume to target path: mount failed: exit status 32...mounting .. failed, reason given by server: No such file or directory.
  2. Create a Kubernetes secret for vSphere credentials.
    1. Create the secret by running the following command. 
      kubectl create secret generic vsphere-config-secret --from-file=csi-vsphere.conf --namespace=vmware-system-csi
    2. Verify that the credential secret is successfully created in the vmware-system-csi namespace. 
      $ kubectl get secret vsphere-config-secret --namespace=vmware-system-csi
NAME                    TYPE     DATA   AGE
vsphere-config-secret   Opaque   1      43s
    3. Delete the configuration file for security purposes. 
      rm csi-vsphere.conf

Use a Secure Connection for vSphere Container Storage Plug-in

Follow this procedure if you want to use a secure connection instead of using a self-signed certificate for login.

Prerequisites

Make sure to enter false as a value for the insecure-flag parameter in the vSphere configuration file. The value indicates that you plan to use a secure connection.

If your environment includes multiple vCenter Server instances, see Use a Secure Connection in the Environment with Multiple vCenter Server Instances.

Procedure

  1. Download trusted root CA certificates from vCenter Server at https://vCenter-IP-Address/certs/download.zip, extract the download.zip file containing certificates, and create config-map using the certificate in the certs/lin directory. 
    $ curl -LO https://vCenter-IP-Address/certs/download.zip
$ unzip download.zip
$ tree certs/
certs/
├── lin
│   ├── 6355e8d1.0
│   └── 6355e8d1.r1
├── mac
│   ├── 6355e8d1.0
│   └── 6355e8d1.r1
└── win
    ├── 6355e8d1.0.crt
    └── 6355e8d1.r1.crl
 
3 directories, 6 files
  2. Create config-map for root-ca certificate. 
    $ cd certs/lin
$ kubectl create configmap vc-root-ca-cert --from-file=6355e8d1.0 --namespace=vmware-system-csi
configmap/vc-root-ca-cert created
  3. Set the following values for vsphere-config-secret in the vmware-system-csi namespace.
    1. Set insecure-flag to false. 
      [Global]
.
.
insecure-flag = "false"
ca-file = "/etc/ssl/certs/6355e8d1.0"
.
    2. Update the vCenter Server details to FQDN as shown in the example below. 
      [Global]
cluster-id = "cluster1"
cluster-distribution = "CSI-Vanilla"

[VirtualCenter "vCenter-FQDN"]
insecure-flag = "false"
ca-file = /certs/lin/1abc830c.0
user = "[email protected]"
password = "Admin!444"
port = "555"
datacenters = "VSAN-DC"

[Snapshot]
global-max-snapshots-per-block-volume = 3
  
[Labels]
topology-categories = "k8s-zone"
  4. Mount vc-root-ca-cert configmap as a volume to the CA root location of containers vsphere-syncer and vsphere-csi-controller in vsphere-csi-controller pod.
    Refer to the following change for the vsphere-csi-controller deployment for vsphere-csi-controller and vsphere-syncer containers. 
    .
.
containers:
  - name: vsphere-csi-controller
    volumeMounts:
      - mountPath: /etc/ssl/certs/6355e8d1.0
        subPath: 6355e8d1.0
        name: vc-root-ca-cert
  - name: vsphere-syncer
    volumeMounts:
      - mountPath: /etc/ssl/certs/6355e8d1.0
        subPath: 6355e8d1.0
        name: vc-root-ca-cert
.
.    
volumes:
- name: vc-root-ca-cert
  configMap:
    name: vc-root-ca-cert
.
.
  5. Apply the above change for vsphere-csi-controller deployment and wait for the vSphere Container Storage Plug-in controller pods to restart.

Use a Secure Connection in the Environment with Multiple vCenter Server Instances

Follow this procedure if you want to use a secure connection in the environment with multiple vCenter Server instances.

Prerequisites

Procedure

  1. For each vCenter Server that needs secure connection, download trusted root CA certificates from https://vCenter-IP-Address/certs/download.zip, extract the download.zip file containing certificates, and create config-map using the certificate in the certs/lin directory.
    In the following example, a Kubernetes cluster is spread across two vCenter Server instances. vSphere Container Storage Plug-in needs to establish a secure connection with both instances. 
    $ curl -LO https://<vCenter-1-IP-Address>/certs/download.zip
$ unzip download.zip
$ tree certs-vc1/
certs-vc1/
── lin
│ ├── 6355e8d1.0
│ └── 6355e8d1.r1
── mac
│ ├── 6355e8d1.0
│ └── 6355e8d1.r1
── win
├── 6355e8d1.0.crt
└── 6355e8d1.r1.crl 3 directories, 6 files

$ curl -LO https://<vCenter-2-IP-Address>/certs/download.zip
$ unzip download.zip
$ tree certs-vc2/
certs-vc2/
── lin
│ ├── 4135e8d1.0
│ └── 4135e8d1.r1
── mac
│ ├── 4135e8d1.0
│ └── 4135e8d1.r1
── win
├── 4135e8d1.0.crt
└── 4135e8d1.r1.crl 3 directories, 6 files
  2. For each vCenter Server instance, create config-map for a root-ca certificate. 
    $ cd certs-vc1/lin
$ kubectl create configmap vc-1-root-ca-cert --from-file=6355e8d1.0 --namespace=vmware-system-csi
configmap/vc-1-root-ca-cert created

$ cd certs-vc2/lin
$ kubectl create configmap vc-2-root-ca-cert --from-file=4135e8d1.0 --namespace=vmware-system-csi
configmap/vc-2-root-ca-cert created
  3. For each instance of secure vCenter Server, set insecure-flag to false in the vsphere-config-secret in the vmware-system-csi namespace. 
    [VirtualCenter "<VC-1-IP or VC-1-FQDN>"]
insecure-flag = "false"
ca-file = "/etc/ssl/certs/6355e8d1.0"
...

[VirtualCenter "<VC-2-IP or VC-2-FQDN>"]
insecure-flag = "false"
ca-file = "/etc/ssl/certs/4135e8d1.0"
...
  4. Mount each config-map created in Step 2 as a volume to the CA root location of the vsphere-syncer and vsphere-csi-controller containers in the vsphere-csi-controller pod.
    Use the following as an example of the changes for the vsphere-csi-controller deployment of the vsphere-syncer and vsphere-csi-controller containers. 
    . .
containers:
- name: vsphere-csi-controller
   volumeMounts:
  - mountPath: /etc/ssl/certs/6355e8d1.0
    subPath: 6355e8d1.0
    name: vc-1-root-ca-cert
  - mountPath: /etc/ssl/certs/4135e8d1.0
     subPath: 4135e8d1.0
     name: vc-2-root-ca-cert
- name: vsphere-syncer
   volumeMounts:
   - mountPath: /etc/ssl/certs/6355e8d1.0
      subPath: 6355e8d1.0
      name: vc-1-root-ca-cert
   - mountPath: /etc/ssl/certs/4135e8d1.0
     subPath: 4135e8d1.0
     name: vc-2-root-ca-cert
. .
volumes:
- name: vc-1-root-ca-cert
  configMap: name: vc-1-root-ca-cert
- name: vc-2-root-ca-cert
  configMap: name: vc-2-root-ca-cert
. .
  5. Apply the changes you made in Step 4 for vsphere-csi-controller deployment and wait for the vSphere Container Storage Plug-in controller pods to restart.

Automatic Generation of Cluster IDs in vSphere Container Storage Plug-in

Every Kubernetes cluster in vSphere Container Storage Plug-in contains a unique cluster ID set in the configuration file. This section contains information about automatic generation of cluster IDs.

If you do not provide the cluster ID field or keep it empty while creating a configuration secret for vSphere Container Storage Plug-in, it automatically generates a unique cluster ID across all clusters. vsphere-csi-cluster-id configuration map is created in the namespace where you have installed vSphere Container Storage Plug-in to store this cluster ID.

Upgrade vSphere Container Storage Plug-in

When you upgrade vSphere Container Storage Plug-in, do not remove the cluster ID attribute if it is already part of vSphere configuration secret. Otherwise, some of the old volumes will use the old cluster ID, and new volumes begin to use the new cluster ID. Due to this, PVs cannot be used which can cause volume operation failures.

Downgrade vSphere Container Storage Plug-in

If you want to downgrade vSphere Container Storage Plug-in version 3.x that uses an automatically created cluster ID to an older version 2.x, follow these steps.

Procedure

  1. Fetch the cluster ID from vsphere-csi-cluster-id configuration map in the namespace where you have installed vSphere Container Storage Plug-in.
  2. Uninstall vSphere Container Storage Plug-in with the new version.
  3. When you install vSphere Container Storage Plug-in version 2.x, specify the cluster ID retrieved in step 1 when you create a vSphere configuration secret.