You can migrate in-tree vSphere volumes to vSphere Container Storage Plug-in. After you migrate the in-tree vSphere volumes, vSphere Container Storage Plug-in performs all subsequent operations on migrated volumes.

vSphere Container Storage Plug-in and CNS provide functionality that is not available with the in-tree vSphere volume plug-in. For information, see Supported Kubernetes Functionality and Limitations and vSphere Functionality Supported by vSphere Container Storage Plug-in.
Note:
  • Kubernetes will deprecate the in-tree vSphere volume plug-in, and it will be removed in the future Kubernetes releases. Volumes provisioned using the vSphere in-tree plug-in will not get additional new features supported by the vSphere Container Storage Plug-in.
  • Kubernetes provides a seamless procedure to help migrate in-tree vSphere volumes to a vSphere Container Storage Plug-in. After you migrate the in-tree vSphere volumes vSphere Container Storage Plug-in, all subsequent operations on migrated volumes are performed by the vSphere Container Storage Plug-in. The migrated vSphere volume will not get additional capabilities supported by vSphere Container Storage Plug-in.

Considerations for Migration of In-Tree vSphere Volumes

When you prepare to use the vSphere Container Storage Plug-in migration, consider the following items.

  • vSphere Container Storage Plug-in migration is released as a Beta feature in Kubernetes 1.19. For more information, see Release note announcement.
  • Kubernetes 1.19 release deprecates vSAN raw policy parameters for the in-tree vSphere volume plug-in. These parameters will be removed in a future release. For more information, see Deprecation Announcement.
  • The following vSphere in-tree StorageClass parameters are not supported after you enable migration:
    • hostfailurestotolerate
    • forceprovisioning
    • cachereservation
    • diskstripes
    • objectspacereservation
    • iopslimit
    • diskformat
  • You cannot rename or delete the storage policy consumed by an in-tree vSphere volume. Volume migration requires the original storage policy used for provisioning the volume to be present on vCenter Server for registration of volume as a container volume in vSphere.
  • Do not rename the datastore consumed by in-tree vSphere volume. Volume migration relies on the original datastore name present on the volume source for registration of volume as container volume in vSphere.
  • Ensure to add the following annotations before enabling migration for statically created vSphere in-tree Persistent Volume Claims, and Persistent Volumes. Statically provisioned in-tree vSphere volumes cannot be migrated to the vSphere Container Storage Plug-in without adding these annotations. This also applies to new static in-tree PVs and PVCs created after you enable migration.
    Annotation on PV:
    annotations:
      pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
    Annotation on PVC:
    annotations:
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
  • vSphere Container Storage Plug-in does not support provisioning the eagerzeroedthick and zeroedthick volume. After you enable migration, vSphere Container Storage Plug-in fails the volume creation activity when you request a new volume using the in-tree provisioner and diskformat parameter set to eagerzeroedthick or zeroedthick.
  • After migration, the only supported value for diskformat parameter is thin. Volumes created before the migration with the disk format eagerzeroedthick or zeroedthick are migrated to CSI.
  • vSphere Container Storage Plug-in does not support raw vSAN policy parameters. After you enable the migration, vSphere Container Storage Plug-in fails the volume creation activity when you request a new volume using in-tree provisioner and vSAN raw policy parameters.
  • The vSphere Container Storage Plug-in migration requires vSphere 7.0 Update 1 and later. If you plan to migrate the in-tree vSphere volumes, you must upgrade to vSphere to 7.0 Update 1 or later.
  • vSphere Container Storage Plug-in does not support formatting volumes with the Windows file system. In-tree vSphere volumes migrated using the Windows file system cannot be used with the vSphere Container Storage Plug-in.
  • In-tree vSphere volume plug-in relies on the name of the datastore set on the PVs source. After you enable migration, do not enable Storage DRS or vMotion. If Storage DRS moves a disk from one datastore to another, further volume operations might break.
  • If you use zone and region aware in-tree deployments, upgrade to vSphere Container Storage Plug-in version 2.4.1 and later.

    Before installing vSphere Container Storage Plug-in, add the following section to the vSphere secret configuration.

    Kubernetes Version In-Tree vSphere Secret Configuration vSphere Container Storage Plug-in Secret Configuration Sample Labels on vSphere Container Storage Plug-in Nodes after Installation
    1.21.x and below

    Earliest version is 1.19

    [Labels]
    region = k8s-region
    zone = k8s-zone
    [Labels]
    region = k8s-region
    zone = k8s-zone
    Name:               k8s-node-0179
    Roles:              <none>
    Labels:             failure-domain.beta.kubernetes.io/region=region-1
                        failure-domain.beta.kubernetes.io/zone=zone-a 
    Annotations: ....

    1.22.x and 1.23.x

    If all of your existing PVs have the GA label, use this approach.

    In all other cases, do not migrate to vSphere Container Storage Plug-in.

    [Labels]
    region = k8s-region
    zone = k8s-zone
    [Labels]
    region = k8s-region
    zone = k8s-zone
    [TopologyCategory "k8s-region"]
    Label = "topology.kubernetes.io/region"
    [TopologyCategory "k8s-zone"]
    Label = "topology.kubernetes.io/zone"
    Name:               k8s-node-0179
    Roles:              <none>
    Labels:             topology.kubernetes.io/region=region-1
                        topology.kubernetes.io/zone=zone-a 
    Annotations: ....

    For information about deployments with topology, see Deploy the vSphere Container Storage Plug-in with Topology.

Enable Migration of In-Tree vSphere Volumes to vSphere Container Storage Plug-in

Migrate the in-tree vSphere volumes to vSphere Container Storage Plug-in.

Prerequisites

  • Upgrade your vSphere environment to version 7.0 Update 1 or later.
  • Upgrade Kubernetes to 1.19 release.
  • Make sure that you use kubectl version 1.19 or later.

Procedure

  1. Install vSphere Cloud Provider Interface (CPI).
    For more information, see Install vSphere Cloud Provider Interface.
  2. Install vSphere Container Storage Plug-in 2.4.
    See https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/release-2.4/manifests/vanilla/vsphere-csi-driver.yaml.

    In the deployment YAML file, enable the csi-migration feature.

     apiVersion: v1
      data:
        "csi-migration": "true"
      kind: ConfigMap
      metadata:
        name: internal-feature-states.csi.vsphere.vmware.com
        namespace: vmware-system-csi
  3. Install the admission webhook.
    vSphere Container Storage Plug-in does not support provisioning of a volume by specifying migration specific parameters in the StorageClass. These parameters are added by the vSphere Container Storage Plug-in translation library, and should not be used in the storage class directly.
    The Validating admission controller prevents you from creating or updating StorageClass using csi.vsphere.vmware.com as provisioner with these parameters:
    • csimigration
    • datastore-migrationparam
    • diskformat-migrationparam
    • hostfailurestotolerate-migrationparam
    • forceprovisioning-migrationparam
    • cachereservation-migrationparam
    • diskstripes-migrationparam
    • objectspacereservation-migrationparam
    • iopslimit-migrationparam

    In addition, the Validating admission controller prevents you from creating or updating StorageClass using kubernetes.io/vsphere-volume as provisioner with AllowVolumeExpansion set to true.

    As a prerequisite, you must install kubectl, openssl, and base64 commands on the system, from which you invoke admission webhook installation scripts.

    To deploy the admission webhook, download and execute the following file:

    https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/release-2.4/manifests/vanilla/deploy-vsphere-csi-validation-webhook.sh

    ./deploy-vsphere-csi-validation-webhook.sh
    creating certs in tmpdir /var/folders/vy/_6dvxx7j5db9sq9n38qjymwr002gzv/T/tmp.ogj5ioIk 
    Generating a 2048 bit RSA private key
    ..........................................................................................................................................................+++
    ...........................................+++
    writing new private key to '/var/folders/vy/_6dvxx7j5db9sq9n38qjymwr002gzv/T/tmp.ogj5ioIk/ca.key'
    -----
    Generating RSA private key, 2048 bit long modulus
    ..............................................................+++
    ...........+++
    e is 65537 (0x10001)
    Signature ok
    subject=/CN=vsphere-webhook-svc.vmware-system-csi.svc
    Getting CA Private Key
    secret "vsphere-webhook-certs" deleted
    secret/vsphere-webhook-certs created
    service/vsphere-webhook-svc created
    validatingwebhookconfiguration.admissionregistration.k8s.io/validation.csi.vsphere.vmware.com created
    serviceaccount/vsphere-csi-webhook created
    role.rbac.authorization.k8s.io/vsphere-csi-webhook-role created
    rolebinding.rbac.authorization.k8s.io/vsphere-csi-webhook-role-binding created
    deployment.apps/vsphere-csi-webhook created
  4. On all control plane nodes, enable CSIMigration and CSIMigrationvSphere parameters on kube-controller and kubelet.
    CSIMigrationvSphere flag enables shims and translation logic to route volume operations from the vSphere in-tree volume plug-in to vSphere Container Storage Plug-in. It also supports falling back to in-tree vSphere plug-in if a node does not have vSphere Container Storage Plug-in installed and configured.

    CSIMigrationvSphere requires CSIMigration feature flag to be enabled. This flag enables the vSphere Container Storage Plug-in migration on the Kubernetes cluster.

    1. Update kube-controller-manager manifest file and add the following arguments.
      The file is available at /etc/kubernetes/manifests/kube-controller-manager.yaml.
      `- --feature-gates=CSIMigration=true,CSIMigrationvSphere=true`
    2. Update kubelet configuration file and add the following flags.
      The file is available at /var/lib/kubelet/config.yaml.
      featureGates:
           CSIMigration: true
           CSIMigrationvSphere: true
    3. Restart the kubelet on the control plane nodes using the following command.
      systemctl restart kubelet
    4. Verify that the kubelet is functioning correctly using the following command
      systemctl status kubelet
    5. For any issues with the kubelet, check the logs on the control plane node using the following command.
      journalctl -xe
  5. Enable CSIMigration and CSIMigrationvSphere feature flags on kubelet at all workload nodes.
    1. Before you change the configuration on the kubelet on each workload node, drain the nodes by removing running application workloads.
      The following is a node drain example.
      $ kubectl drain k8s-node1 --force --ignore-daemonsets
       node/k8s-node1 cordoned
       WARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: default/vcppod; ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-gs7fr, kube-system/kube-proxy-rbjx4, kube-system/vsphere-csi-node-fh9f6
       evicting pod default/vcppod
         pod/vcppod evicted
         node/k8s-node1 evicted
    2. After you enable the migration, ensure the csinodes instance for the node is updated with the storage.alpha.kubernetes.io/migrated-plugins annotation.
      $ kubectl describe csinodes k8s-node1
        Name:               k8s-node1
        Labels:             <none>
        Annotations:        storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/vsphere-volume
        CreationTimestamp:  Wed, 29 Apr 2020 17:51:35 -0700
        Spec:
          Drivers:
            csi.vsphere.vmware.com:
              Node ID:  k8s-node1
        Events:         <none>
    3. Restart the kubelet on the workload nodes using the following command.
      systemctl restart kubelet
    4. Verify that the kubelet is functioning correctly using the following command.
      systemctl status kubelet
    5. For any issues with the kubelet, check the logs on the workload node using the following command.
      journalctl -xe
    6. After you restart the kubelet, uncordon the node so that it can be used for scheduling workloads.
      kubectl uncordon k8s-node1
      
    7. Repeat the above steps for all workload nodes in the Kubernetes cluster.
  6. (Optional) You can enable the CSIMigrationvSphereComplete flag if you enabled the vSphere Container Storage Plug-in migration on all nodes.
    CSIMigrationvSphereComplete helps you to stop registering the vSphere in-tree plug-in in kubelet and volume controllers. It also enables shims and translation logic to route volume operations from the vSphere in-tree plug-in to vSphere Container Storage Plug-in. The CSIMigrationvSphereComplete flag requires you to enable the CSIMigration and CSIMigrationvSphere feature flags. Also, you must install and configure vSphere Container Storage Plug-in on all nodes in the cluster.
  7. Verify that the vSphere in-tree PVCs and PVs are migrated to vSphere Container Storage Plug-in and the pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com annotations are present on PVCs and PVs
    Annotations on PVCs:
    Annotations:   pv.kubernetes.io/bind-completed: yes
                    pv.kubernetes.io/bound-by-controller: yes
                    pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com
                    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
    Annotations on PVs:
    Annotations:     kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
                      pv.kubernetes.io/bound-by-controller: yes
                      pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com
                      pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume

    After you enable the migration, vSphere Container Storage Plug-in creates a new in-tree vSphere volume. It can be identified by using the following annotations. The PV specification continues to hold the vSphere volume path. If you want to disable the migration, the provisioned volume can be used by the in-tree vSphere plug-in.

    Annotations on PVCs:
    Annotations:   pv.kubernetes.io/bind-completed: yes
                    pv.kubernetes.io/bound-by-controller: yes
                    volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
    Annotations on PVs:
    Annotations:     pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com