RSS
 

VMware vSphere Container Storage Plug-in 3.x | 17 MAR 2023

Check for additions and updates to these release notes.

About 3.x Release Notes

These Release Notes cover 3.x versions of VMware vSphere Container Storage Plug-in, previously called vSphere CSI Driver.

For known issues observed in Kubernetes, see vSphere CSI Driver - Known Issues.

vSphere Container Storage Plug-in 3.1.x

What's New

Version

What's New

Version 3.1.0

This minor release fixes issues observed in the 3.0.2 release and includes these changes:

  • Support for Kubernetes version 1.28.

  • HCI mesh support for CNS block volumes.

  • Added resize capability for migrated in-tree vSphere volumes.

  • Enhanced vCenter Server session management to ensure minimum number of sessions are kept open from vSphere Container Storage Plug-in to vCenter Server.

  • Reduced number of vpxd connection created by vSphere Container Storage Plug-in on vCenter Server enhancing vCenter Server task monitoring workflow.

Deployment Files

Important:

To ensure proper functionality, do not update the internal-feature-states.csi.vsphere.vmware.com configmap available in the deployment YAML file. VMware does not recommend to activate or deactivate features in this configmap.

Version

File

Version 3.1.0

https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v3.1.0/manifests/vanilla

Kubernetes Releases

  • Minimum: 1.26

  • Maximum: 1.28

Supported Sidecar Containers Versions

  • csi-provisioner: v3.5.0

  • csi-attacher: v4.3.0

  • csi-resizer: v1.8.0

  • livenessprobe: v2.10.0

  • csi-node-driver-registrar: v2.8.0

  • csi-snapshotter: v6.2.2

Resolved Issues

Version

Resolved Issues

Version 3.1.0

  • Fixed the logic of selecting a datastore in a topology aware setup to ensure local host datastores are selected for volume provisioning when topology is defined at a cluster or a higher level entity.

  • Prevent deletion of critical migration CRs when volumes are temporarily lost from vCenter Server.

  • Extended the validity of the webhook container certificate to 180 days which reduces the need for frequent certificate renewals.

Known Issues

  • When vCenter Server goes down, CSI snapshot creation tasks fail with an error message

    When vCenter Server goes down, any snapshot creation tasks that are in progress are interrupted. vSphere Container Storage Plug-in cannot get an update on the status of these tasks. As a result, it will continue to retry the operations at Kubernetes level. This will cause the snapshots to remain in a not ready state.

    You can see the following error message: failed to create snapshot on volume <volume-ID>: Failed to get taskInfo for CreateSnapshots task from vCenter <VC-IP> with err:

    Once vCenter Server becomes accessible, delete and recreate the affected snapshots that are still in not ready state. Make sure to set ReadyToUse field to false.

  • Volume attachment fails after recovering from Kubernetes infrastructure failures

    vSphere HA attempts to migrate worker node virtual machines from one host to another when it recovers from infrastructure failures like host reboots and downtimes. Once the virtual machine is successfully moved, if you attempt to power it on, Unable to write VMX File error message appears.

    This error message occurs because the virtual machine's configuration cannot locate the volume associated with the specified backing disk ID.

    As a result, vSphere Container Storage Plug-in receives a VM NotFound error. When the pod using this volume is rescheduled to another node, CSI reports a successful detach volume response. However, the volume detachment process is yet to be completed at the back end, and the disk remains attached to the previous node's virtual machine. Because of this, the rescheduled pod encounters a VolumeAttachment failure with the The resource volume is in use error.

    Workaround: Manually detach the volume from the vSphere Client using the following steps:

    1. Obtain the PersistentVolume name from the Kubernetes cluster which contains the VolumeAttachment failure due to  ResourceInUse and Unable to write VMX file error.

      kubectl get pods -n <namespace> | grep -iv Running
Output: <pod-name>kubectl describe pod <pod-name> -n <namespace> | grep ResourceInUse
Output: <ResourceInUseError>kubectl describe pod <pod-name> -n <namespace> | grep ClaimName
Output: <claim-name>

kubectl get pv | grep <claim-name>
Output: <pv-name>

kubectl describe pv <pv-name> | grep VolumeHandle
Output: <volume-handle>

kubectl logs <vsphere-csi-controller-pod-name> -c vsphere-csi-controller -n vmware-system-csi | grep <volume-handle> | grep "Unable to write VMX file"
Output: {"level":"error","time":"<time-stamp>","caller":"volume/manager.go:<num>","msg":"failed to detach cns volume: \"<volume-handle>\" from node vm: VirtualMachine:vm-<moref>...Unable to write VMX file....

    2. Obtain the VM name and the VolumePath from the ContainerVolumes section of vSphere Client.

      1. In the vSphere Client Inventory section right click the virtual machine obtained from the step 1, and click Edit Settings.

      2. Under Virtual Hardware, select the hard disk to remove. The Disk File should match the VolumePath obtained from the previous step.

      3. To remove a disk, click the ellipsis icon that appears on the right.

      For more information, see Monitor Container Volumes Across Kubernetes Clusters.

vSphere Container Storage Plug-in 3.0.x

What's New

Version

What's New

Version 3.0.2

This patch release fixes issues observed in the 3.0.0 and 3.0.1 releases and includes these changes:

  • Allow disabling of the useCSINodeID feature. A pod fails to come up when the useCSINodeID feature is disabled. See 2373 for details.

  • Fixed a segmentation problem observed when enabling the alpha feature pv-to-backingdiskobjectid-mapping. See 2370 for details.

  • Allow NTFS fsType to be uppercase. See 2305 for details.

  • Allow enabling topology if it has not been configured initially during the deployment of the vSphere Container Storage plug-in. See 2412 for details.

  • Fixed the List-volume feature to honor a vCenter Server without any nodes in the vSphere Config Secret. See 2393 for details.

  • Fixed authorization service to permit adding new vCenter Server without any datastores or nodes while working on extending topology setup to another vCenter Server. See 2443 for details.

  • Fixed pushing volume metadata to vCenter Server for migrated in-tree vSphere volume. See 2454 for details.

Version 3.0.1

  • Fixed a bug in the CSI full sync that caused the Delete PVC operation to fail due to the deletion of the CNSVolumeInfo CR. This issue has been resolved. For more information, see 2327 in GitHub.

  • Resolved the panic observed in the syncer container during the full sync process. This behavior occurred when stale CNS volumes were deleted while the corresponding PV was not present in the Kubernetes cluster. For more details, see 2347 in GitHub.

Version 3.0.0

Deployment Files

Important:

To ensure proper functionality, do not update the internal-feature-states.csi.vsphere.vmware.com configmap available in the deployment YAML file. VMware does not recommend to activate or deactivate features in this configmap.

Version

File

Version 3.0.2

https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v3.0.2/manifests/vanilla

Version 3.0.1

https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v3.0.1/manifests/vanilla

Version 3.0.0

https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v3.0.0/manifests/vanilla

Kubernetes Releases

  • Minimum: 1.24

  • Maximum: 1.27

Supported Sidecar Containers Versions

  • csi-provisioner: v3.4.0

  • csi-attacher: v4.2.0

  • csi-resizer: v1.7.0

  • livenessprobe: v2.9.0

  • csi-node-driver-registrar: v2.7.0

  • csi-snapshotter: v6.2.1

Resolved Issues

Version

Resolved Issues

Version 3.0.0

  • Volume provisioning using the datastoreURL parameter in StorageClass does not work correctly when this datastoreURL points to a shared datastore mounted across datacenters. See #2187 in GitHub.

  • Mounting a PVC on a Windows node fails with Size Not Supported error. See #2080 in GitHub.

  • CreateVolume request fails with an error after you change the hostname or IP of vCenter Server in the vsphere-config-secret. See #2221 in GitHub.

Known Issues

  • update known issue in the release note

  • Attempts to run a PV and a PV created out of a snapshot of the original PV on the same Windows node might fail

    This problem occurs when a pod with a persistent volume (PV) is running on a node and you try to schedule a new pod. If the new pod uses a PV created from a snapshot of the original PV and is scheduled to run on the same node, the pod might remain in a pending state.

    The problem does not occur on Linux nodes.

    Workaround: If you must run two pods, one using a PV and another a PV created from a snapshot of the original PV, schedule the pods on different Windows nodes. You can use the node selector in the pod specification.

  • When Cloud Native Storage Manager is used with vSphere Container Storage Plug-in, automatic generation of cluster IDs is not possible

    Automatic generation of cluster IDs is not compatible with an environment that uses Cloud Native Storage Manager with vSphere Container Storage Plug-in 3.0.0.

    Workaround: Manually specify the cluster ID in the vSphere configuration secret during deployment.

  • Attempts to create a pod with XFS file system might fail

    When you try to create a pod with XFS file system using vSphere Container Storage Plug-in on CentOS 7 and Red Hat Enterprise 7 nodes, the pod remains in the pending state. The following error message appears.

    output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3303985d-0d2e-4c3d-87ab-7a25a29ad0ff/globalmount: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error.

    Workaround: None.

  • Migration of a pod using in-tree vSphere volumes occasionally gets stuck in theContainerCreatingstate with the error messagefailed to set keepAfterDeleteVm control flag for VolumeID

    This problem is rare and might occur when the pod is backed by deployments or statefulsets using in-tree vSphere volumes.

    Workaround:

    1. Uncordon the drained node from which you attempted to evict the pod.

    2. Cordon all other nodes.

    3. Delete the problematic pod.

      The pod will be up and running on the node to which the volumes are attached.

    4. Uncordon other nodes.

    5. Run the following command on the node to verify that the volumes are getting detached.

      kubectl drain <node-name> --ignore-daemonsets

    If, for some reason, the in-tree vSphere volume fails to detach from the node VM, you can manually detach the volume.

    1. In the vSphere Client, right-click the node virtual machine, and click Edit Settings.

    2. Under Virtual Hardware, select the hard disk to remove.

    3. Remove the disk by clicking the X icon that appears on the right.

    4. Click OK.

    vSphere 7.0 p07 and vSphere 8.0 Update 1 provides a fix that enables you to set the control flag on the volume even when it is attached to the node VM. Alternatively, it is recommended to upgrade to vSphere 7.0 p07 or vSphere 8.0 Update 1 to resolve this issue.

  • After a vSphere upgrade, vSphere Container Storage Plug-in might not pick up new vSphere features

    After a vSphere upgrade is performed in the background, the vSphere Container Storage Plug-in controller deployment needs to be restarted. This action is required to make vSphere Container Storage Plug-in pick up the new features compatible with the vSphere version.

    Workaround:

    Run the following command: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

  • Under certain conditions, you might be able to provision more than three snapshots per volume despite default limitations

    By default, vSphere Container Storage Plug-in allows a maximum of three snapshots per volume. This limitation is applicable only when snapshot requests are at different time intervals. If you send multiple and parallel requests to create a snapshot for the volume at the same time, the system allows you to provision more than three snapshots per volume. Although no volume or snapshot operations are impacted, exceeding the maximum number of snapshots per volume is not recommended.

    Workaround: Avoid creating more than three snapshots on a single volume.

  • When a site failure occurs, pods that were running on the worker nodes in that site remain in Terminating state

    When a site failure causes all Kubernetes nodes and ESXi hosts in the cluster on that site to fail, the pods that were running on the worker nodes in that site will be stuck in Terminating state.

    Workaround: Start some of the ESXi hosts in the site as soon as possible, so that vSphere HA can restart the failed Kubernetes nodes. This action ensures that the replacement pods begin to come up.

  • After a recovery from network partition or host failure, some nodes in the cluster do not have INTERNAL-IP or EXTERNAL-IP

    After a recovery from a network partition or host failure, CPI is unable to assign INTERNAL-IP or EXTERNAL-IP to the node when it is added back to the cluster.

    Workaround:

    1. De-register the affected node.

      # kubectl delete node node-name

    2. Re-register the affected node by restarting kubelet service within the affected node.

      # systemctl restart kubelet

    3. Wait for node to register with the cluster.

    4. Taint the affected nodes.

      # kubectl taint node node-name node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

    5. Wait for CPI to initialize the node. Make sure ProviderID is set and IP address is present for the node.

  • After recovering from a network partition or host failure, the control plane node becomes a worker node

    During network partition or host failure, the CPI might delete the node from the Kubernetes cluster if a VM is not found. After the recovery, the control plane node might become the worker node. Because of this, the pods tend to get scheduled on it unexpectedly.

    You can fix this issue in two ways:

    Workaround 1

    1. Taint and add labels to the affected nodes.

      # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule

    2. Delete the node from cluster.

      # kubeclt delete node <node name>

    3. Restart kublet service within the affected node.

      # systemctl restart kubelet

    4. Wait for the node to register with the cluster and add labels to the affected node.

      # kubectl label node <node name> node-role.kubernetes.io/control-plane=

    5. Delete the application pods which are already scheduled on the control plane node to get scheduled on new worker nodes.

      # kubectl delete pod <pod name>

    Workaround 2

    1. Add the environment variable with SKIP_NODE_DELETION=true.

      # kubectl set env daemonset vsphere-cloud-controller-manager -n kube-system SKIP_NODE_DELETION=true

    2. Verify whether the environment variable has been applied correctly.

      # kubectl describe daemonset vsphere-cloud-controller-manager -n kube-system

    3. Terminate the running pods. The next pod that you create will pull the environment variable.

      # kubectl delete pod <pod name>

    4. Wait for the pod to start.

    5. View logs with `kubectl logs [POD_NAME] -n kube-system`, and confirm if everything is healthy.

    Note: If you use the second method, it might result in leftover nodes and might introduce unexpected behaviors.

  • When a Kubernetes worker node shuts down non-gracefully, pods on that node remain in Terminating state

    Pods will not be rescheduled to other healthy worker nodes. As a result, the application might face a downtime or run in degraded mode. This depends on the number of replicas of the application present on the worker node that experiences non-graceful shut down.

    Workaround: Forcefully delete the pods that remain in terminating state.

  • After recovery from a network partition or host failure, pods might remain in containerCreating state

    During a network partition or host failure, CPI might delete the node from the Kubernetes cluster if a VM is not found. After recovery, the nodes might not be automatically added back to the cluster. This results in pods remaining in containerCreating state with the error message "Volume not attached according to node status for volume" or "".

    Workaround:

    If the issue occurs on a control plane node, perform the following steps.

    1. Restart kubelet service within the affected node.

      # systemctl restart kubelet

    2. Wait for the node to register with the cluster. Add labels and taints to the affected node.

      # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule

      # kubectl label node <node name> node-role.kubernetes.io/control-plane=

    If the issue affects a worker node, perform the following steps.

    1. Restart kubelet service within the affected node.

      # systemctl restart kubelet

    2. Taint the affected node(s).

      # kubectl taint node <node-name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

  • Changes to Datacenter and Port entries in the vsphere-config-secret are not applied until you restart the vSphere Container Storage Plug-in pod

    After you make changes to the Datacenter and Port entries in the vsphere-config-secret, volume life cycle operations fail.

    Workaround: Restart the vsphere-csi-controller deployment pod.

  • Volume life cycle operations might be delayed during vSAN network partitioning

    You can observe some delays in Pod creation and Pod deletion during network partitioning on a vSAN cluster. After vSphere Container Storage Plug-in retries all failed operations, the operations succeed.

    This issue might occur because vCenter Server cannot reach the correct host during network partitioning. The volume fails to be created if the request reaches a host that cannot create the volume. However, during a Kubernetes retry, the volume can be created if it reaches the right host.

    Workaround: None.

  • When you perform various operations on a volume or a node VM, you might observe error messages that appear in vCenter Server

    vCenter Server might display the following error messages:

    • When attaching a volume: com.vmware.vc.InvalidController : "The device '0' is referring to a nonexisting controller '1,001'."

    • When detaching a volume: com.vmware.vc.NotFound : "The object or item referred to could not be found."

    • When resizing a volume: com.vmware.vc.InvalidArgument : "A specified parameter was not correct: spec.deviceChange.device"

    • When updating: com.vmware.vc.Timedout : "Operation timed out."

    • When reconfiguring a VM: com.vmware.vc.InsufficientMemoryResourcesFault : "The available Memory resources in the parent resource pool are insufficient for the operation."

    In addition, you can observe a few less frequent errors for the CSI migration feature specifically in 70u2.

    For update:

    • Cannot find the device '2,0xx', which is referenced in the edit or remove device operation.

    • A general system error occurred: Failed to lock the file: api = DiskLib_Open, _diskPath->CValue() = /vmfs/volumes/vsan:52c77e7d8115ccfa-3ec2df6cffce6713/782c2560-d5e7-0e1d-858a-ecf4bbdbf874/kubernetes-dynamic-pvc-f077b8cd-dbfb-4ba6-a9e8-d7d8c9f4c578.vmdk

    • The operation is not allowed in the current state.

    For reconfigure: Invalid configuration for device '0'.

    Workaround: Most of these errors are resolved after a retry from CSI.

  • A statefulset set replica pod remains in terminating state after you delete the statefulset

    Typically, the problem occurs after you perfrom the following steps:

    1. Create volumes in the Kubernetes cluster using vSphere Cloud Provider (VCP).

    2. Enable the CSIMigration feature flags on kube-controller-manager, kubelet, and install vSphere Container Storage Plug-in.

    3. Enable the csi-migration feature state to migrate the volumes that you previously created using VCP.

    4. Create a statefulset using the migrated volumes and continue to use them in the replica set pods.

    5. When you no longer need the application pods to run in the Kubernetes cluster, perform the delete operation on the statefulset.

    This action might occationally result in replica set pods to remain in terminating state.

    Workaround: Force delete the replica set pods in terminating state:

    kubectl delete pod replica-pod-name --force --grace-period=0

check-circle-line exclamation-circle-line close-line
Scroll to top icon