VMware vSphere Container Storage Plug-in 2.7 | 21 OCT 2022

Check for additions and updates to these release notes.

About 2.7 Release Notes

These Release Notes cover 2.7.0 versions of VMware vSphere Container Storage Plug-in, previously called vSphere CSI Driver.

For known issues observed in Kubernetes, see vSphere CSI Driver - Known Issues.

What's New

Version

What's New

Version 2.7.0

  • Support for Kubernetes version 1.25.

  • Enabled feature gate for in-tree vSphere volume to CSI migration.

  • Ability to suspend a specific datastore for volume provisioning using Cloud Native Storage Manager

Kubernetes Release

  • Minimum: 1.23

  • Maximum: 1.25

Supported Sidecar Containers Versions

  • csi-provisioner: v3.2.1

  • csi-attacher: v3.5.0

  • csi-resizer: v1.5.0

  • livenessprobe: v2.7.0

  • csi-node-driver-registrar: v2.5.1

  • csi-snapshotter: v6.0.1

Resolved Issues

Version

Resolved Issues

Version 2.7.0

  • When a Kubernetes node VM is removed from vCenter Server, PVC delete operations fail.

  • Persistent volume fails to be detached from a node.

Known Issues

  • Migration of a pod using in-tree vSphere volumes occasionally gets stuck in the ContainerCreating state with the error message failed to set keepAfterDeleteVm control flag for VolumeID

    This problem is rare and might occur when the pod is backed by deployments or statefulsets using in-tree vSphere volumes.

    Workaround:

    1. Uncordon the drained node from which you attempted to evict the pod.

    2. Cordon all other nodes.

    3. Delete the problematic pod.

      The pod will be up and running on the node to which the volumes are attached.

    4. Uncordon other nodes.

    5. Run the following command on the node to verify that the volumes are getting detached.

      kubectl drain <node-name> --ignore-daemonsets

    If, for some reason, the in-tree vSphere volume fails to detach from the node VM, you can manually detach the volume.

    1. In the vSphere Client, right-click the node virtual machine, and click Edit Settings.

    2. Under Virtual Hardware, select the hard disk to remove.

    3. Remove the disk by clicking the X icon that appears on the right.

    4. Click OK.

  • After a vSphere upgrade, vSphere Container Storage Plug-in might not pick up new vSphere features

    After a vSphere upgrade is performed in the background, the vSphere Container Storage Plug-in controller deployment needs to be restarted. This action is required to make vSphere Container Storage Plug-in pick up the new features compatible with the vSphere version.

    Workaround:

    Run the following command: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

  • Under certain conditions, you might be able to provision more than three snapshots per volume despite default limitations

    By default, vSphere Container Storage Plug-in allows a maximum of three snapshots per volume. This limitation is applicable only when snapshot requests are at different time intervals. If you send multiple and parallel requests to create a snapshot for the volume at the same time, the system allows you to provision more than three snapshots per volume. Although no volume or snapshot operations are impacted, exceeding the maximum number of snapshots per volume is not recommended.

    Workaround: Avoid creating more than three snapshots on a single volume.

  • Mounting a PVC on a Windows node fails with Size Not Supported error

    If the PVC was expanded before a filesystem was created on it, the pod using this PVC fails while mounting the volume and remains in Pending state. You can observe the error of the following type:

    MountVolume.MountDevice failed while expanding volume for volume "pvc-021b5d9c-ba3a-4c9e-ae22-df2cbfce9e67" : Expander.NodeExpand failed to expand the volume : rpc error: code = Internal desc = error when resizing filesystem on volume "6d5caf4c-36ef-49f1-a804-0269c785b40d" on node: error when resizing filesystem on devicePath \\?\Volume{3df70ab1-1cfe-4047-bd7f-a5d6e44b754e}\ and volumePath \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pvc-021b5d9c-ba3a-4c9e-ae22-df2cbfce9e67\globalmount, err: rpc error: code = Unknown desc = error resizing volume. cmd: Get-Volume -UniqueId "\\?\Volume{3df70ab1-1cfe-4047-bd7f-a5d6e44b754e}\" | Get-Partition | Resize-Partition -Size 3204431360, output: Resize-Partition : Size Not Supported

    Workaround: Delete and re-create the PVC with the expanded size to use it in applications.

  • When a site failure occurs, pods that were running on the worker nodes in that site remain in Terminating state

    When a site failure causes all Kubernetes nodes and ESXi hosts in the cluster on that site to fail, the pods that were running on the worker nodes in that site will be stuck in Terminating state.

    Workaround: Start some of the ESXi hosts in the site as soon as possible, so that vSphere HA can restart the failed Kubernetes nodes. This action ensures that the replacement pods begin to come up.

  • After a recovery from network partition or host failure, some nodes in the cluster do not have INTERNAL-IP or EXTERNAL-IP

    After a recovery from a network partition or host failure, CPI is unable to assign INTERNAL-IP or EXTERNAL-IP to the node when it is added back to the cluster.

    Workaround:

    1. De-register the affected node.

      # kubectl delete node node-name

    2. Re-register the affected node by restarting kubelet service within the affected node.

      # systemctl restart kubelet

    3. Wait for node to register with the cluster.

    4. Taint the affected nodes.

      # kubectl taint node node-name node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

    5. Wait for CPI to initialize the node. Make sure ProviderID is set and IP address is present for the node.

  • After recovering from a network partition or host failure, the control plane node becomes a worker node

    During network partition or host failure, the CPI might delete the node from the Kubernetes cluster if a VM is not found. After the recovery, the control plane node might become the worker node. Because of this, the pods tend to get scheduled on it unexpectedly.

    You can fix this issue in two ways:

    Workaround 1

    1. Taint and add labels to the affected nodes.

      # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule

    2. Delete the node from cluster.

      # kubeclt delete node <node name>

    3. Restart kublet service within the affected node.

      # systemctl restart kubelet

    4. Wait for the node to register with the cluster and add labels to the affected node.

      # kubectl label node <node name> node-role.kubernetes.io/control-plane=

    5. Delete the application pods which are already scheduled on the control plane node to get scheduled on new worker nodes.

      # kubectl delete pod <pod name>

    Workaround 2

    1. Add the environment variable with SKIP_NODE_DELETION=true.

      # kubectl set env daemonset vsphere-cloud-controller-manager -n kube-system SKIP_NODE_DELETION=true

    2. Verify whether the environment variable has been applied correctly.

      # kubectl describe daemonset vsphere-cloud-controller-manager -n kube-system

    3. Terminate the running pods. The next pod that you create will pull the environment variable.

      # kubectl delete pod <pod name>

    4. Wait for the pod to start.

    5. View logs with `kubectl logs [POD_NAME] -n kube-system`, and confirm if everything is healthy.

    Note: If you use the second method, it might result in leftover nodes and might introduce unexpected behaviors.

  • When a Kubernetes worker node shuts down non-gracefully, pods on that node remain in Terminating state

    Pods will not be rescheduled to other healthy worker nodes. As a result, the application might face a downtime or run in degraded mode. This depends on the number of replicas of the application present on the worker node that experiences non-graceful shut down.

    Workaround: Forcefully delete the pods that remain in terminating state.

  • After recovery from a network partition or host failure, pods might remain in containerCreating state

    During a network partition or host failure, CPI might delete the node from the Kubernetes cluster if a VM is not found. After recovery, the nodes might not be automatically added back to the cluster. This results in pods remaining in containerCreating state with the error message "Volume not attached according to node status for volume" or "".

    Workaround:

    If the issue occurs on a control plane node, perform the following steps.

    1. Restart kubelet service within the affected node.

      # systemctl restart kubelet

    2. Wait for the node to register with the cluster. Add labels and taints to the affected node.

      # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule

      # kubectl label node <node name> node-role.kubernetes.io/control-plane=

    If the issue affects a worker node, perform the following steps.

    1. Restart kubelet service within the affected node.

      # systemctl restart kubelet

    2. Taint the affected node(s).

      # kubectl taint node <node-name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

  • Changes to Datacenter and Port entries in the vsphere-config-secret are not applied until you restart the vSphere Container Storage Plug-in pod

    After you make changes to the Datacenter and Port entries in the vsphere-config-secret, volume life cycle operations fail.

    Workaround: Restart the vsphere-csi-controller deployment pod.

  • Volume life cycle operations might be delayed during vSAN network partitioning

    You can observe some delays in Pod creation and Pod deletion during network partitioning on a vSAN cluster. After vSphere Container Storage Plug-in retries all failed operations, the operations succeed.

    This issue might occur because vCenter Server cannot reach the correct host during network partitioning. The volume fails to be created if the request reaches a host that cannot create the volume. However, during a Kubernetes retry, the volume can be created if it reaches the right host.

    Workaround: None.

  • CreateVolume request fails with an error after you change the hostname or IP of vCenter Server in the vsphere-config-secret

    After you change the hostname or IP of vCenter Server in the vsphere-config-secret and then try to create a persistent volume claim, the action fails with the following error:

    failed to get Nodes from nodeManager with err virtual center wasn't found in registry

    You can observe this issue in the following releases of vSphere Container Storage Plug-in: v2.0.2, v2.1.2 , v2.2.2 , v2.3.0, and v2.4.0.

    Workaround: Restart the vsphere-csi-controller pod by running the following command:

    kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

  • When you perform various operations on a volume or a node VM, you might observe error messages that appear in vCenter Server

    vCenter Server might display the following error messages:

    • When attaching a volume: com.vmware.vc.InvalidController : "The device '0' is referring to a nonexisting controller '1,001'."

    • When detaching a volume: com.vmware.vc.NotFound : "The object or item referred to could not be found."

    • When resizing a volume: com.vmware.vc.InvalidArgument : "A specified parameter was not correct: spec.deviceChange.device"

    • When updating: com.vmware.vc.Timedout : "Operation timed out."

    • When reconfiguring a VM: com.vmware.vc.InsufficientMemoryResourcesFault : "The available Memory resources in the parent resource pool are insufficient for the operation."

    In addition, you can observe a few less frequent errors for the CSI migration feature specifically in 70u2.

    For update:

    • Cannot find the device '2,0xx', which is referenced in the edit or remove device operation.

    • A general system error occurred: Failed to lock the file: api = DiskLib_Open, _diskPath->CValue() = /vmfs/volumes/vsan:52c77e7d8115ccfa-3ec2df6cffce6713/782c2560-d5e7-0e1d-858a-ecf4bbdbf874/kubernetes-dynamic-pvc-f077b8cd-dbfb-4ba6-a9e8-d7d8c9f4c578.vmdk

    • The operation is not allowed in the current state.

    For reconfigure: Invalid configuration for device '0'.

    Workaround: Most of these errors are resolved after a retry from CSI.

  • A statefulset set replica pod remains in terminating state after you delete the statefulset

    Typically, the problem occurs after you perfrom the following steps:

    1. Create volumes in the Kubernetes cluster using vSphere Cloud Provider (VCP).

    2. Enable the CSIMigration feature flags on kube-controller-manager, kubelet, and install vSphere Container Storage Plug-in.

    3. Enable the csi-migration feature state to migrate the volumes that you previously created using VCP.

    4. Create a statefulset using the migrated volumes and continue to use them in the replica set pods.

    5. When you no longer need the application pods to run in the Kubernetes cluster, perform the delete operation on the statefulset.

    This action might occationally result in replica set pods to remain in terminating state.

    Workaround: Force delete the replica set pods in terminating state:

    kubectl delete pod replica-pod-name --force --grace-period=0

check-circle-line exclamation-circle-line close-line
Scroll to top icon