VMware vSphere Container Storage Plug-in 2.4 Release Notes

VMware vSphere Container Storage Plug-in 2.4 \| 15 SEP 2022 Check for additions and updates to these release notes.

VMware vSphere Container Storage Plug-in 2.4 | 15 SEP 2022

Check for additions and updates to these release notes.

About 2.4 Release Notes

These Release Notes cover 2.4.x versions of VMware vSphere Container Storage Plug-in, previously called vSphere CSI Driver.

What's New

Version	What's New
Version 2.4.3	Improvements to the idempotency feature that allow the vSphere Container Storage Plug-in to run in a namespace other than vmware-system-csi. For more details, see 1946 in GitHub. Changes that prevent re-discovering of the node topology after a syncer container restart. For more details, see 1906 in GitHub.
Version 2.4.2	Added protection for in-tree vSphere volumes that are migrated to vSphere Container Storage Plug-in when deleting node VMs. The migrated volume will not be deleted even after deleting node VMs. Fixed unmounting of in-tree in-line vSphere volumes when vSphere Container Storage Plug-in migration is enabled.
Version 2.4.1	This patch release enables offline resize in vSphere 6.7 P06 and any future patches on vSphere 6.7 release. See https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1437. This patch enables you to use volume topology feature with vSphere 6.7 P06 and any future patches on vSphere 6.7 release. If you use the Beta topology feature, this version enables you to upgrade to the latest vSphere Container Storage Plug-in release. See Upgrade vSphere Container Storage Plug-in.
Version 2.4.0	Performance and resiliency improvements. Volume topology support for block volumes. Warning: If you use the Beta topology feature in your current version of vSphere Container Storage Plug-in, do not upgrade to version 2.4.0. Upgrade to the patch version 2.4.1. If you use the Beta topology feature with the in-tree vSphere Cloud Provider, do not migrate to vSphere Container Storage Plug-in version 2.4.0. Instead, you can directly upgrade to version 2.4.1 and above, as long as you are using Kubernetes version less than 1.22. If you are not using the Beta topology feature, you can safely upgrade to vSphere Container Storage Plug-in version 2.4.0. Volume topology GA support for block volumes is available only in greenfield deployments. Support for Kubernetes version 1.22. Enhanced the idempotency handling logic of vSphere Container Storage Plug-in in the following scenarios: Restarts of vSphere Container Storage Plug-in during in-flight CreateVolume operations will not result in the creation of an orphan volume on the datastore. In case of multi control plane setups with multiple vSphere Container Storage Plug-in controller replicas, a change of leader during an in-flight CreateVolume operation will not result in the creation of an orphan volume on the datastore.

Deployment Files

Important:

To ensure proper functionality, do not update the internal-feature-states.csi.vsphere.vmware.com configmap available in the deployment YAML file. VMware does not recommend to activate or deactivate features in this configmap.

Version	File
Version 2.4.3	https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.4.3/manifests/vanilla
Version 2.4.2	https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.4.2/manifests/vanilla
Version 2.4.1	https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.4.1/manifests/vanilla
Version 2.4.0	https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.4.0/manifests/vanilla

Kubernetes Release

Minimum: 1.20
Maximum: 1.22

Supported Sidecar Containers Versions

csi-provisioner: v3.0.0
csi-attacher: v3.3.0
csi-resizer: v1.3.0
livenessprobe: v2.4.0
csi-node-driver-registrar: v2.3.0

Resolved Issues

Version	Resolved Issues
Version 2.4.2	When a Node VM is deleted, vSphere Container Storage Plug-in 2.4.2 prevents deletion of migrated volumes attached to the Node VM. For more information, see https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1762. Fixed unmounting of migrated in-tree in-line vSphere volumes. For more information, see https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/1466.
Version 2.4.1	Enable customers who use beta topology feature to upgrade to GA volume topology support. See https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1443. Fix issues related to volume path parsing during migration of in-tree (VCP) volumes to vSphere Container Storage Plug-in. See https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/1447.

Known Issues

Pod is stuck in the Error state when there is a site failure

During the mount operation, when there is a site failure, the Pod is stuck in the Error state due to volume not being found with the error "disk: <disk-uuid> not attached to node".

Workaround:

Delete and recreate the pod when the nodes from the failed site are restarted on another site or a node is recovered on the existing site.
Volume provisioning using the datastoreURL parameter in StorageClass does not work correctly when this datastoreURL points to a shared datastore mounted across datacenters

In a non-topology setup, if nodes span across multiple datacenters, the vSphere Container Storage plug-in fails to find shared datastores accessible to all nodes. This issue occurs because datastores shared across datacenters have different datastore MoRefs, but use the same datastore URL. Current logic in the vSphere Container Storage plug-in does not consider this fact.

In a topology setup, you can have multiple availability zones across datacenters with datastores shared across availability zones. If the volume is provisioned on that shared datastore, the vSphere Container Storage plug-in might end up publishing node affinity rules for the availability zone that are different from what you have requested in the StorgeClass.
Workaround:
- In a non-topology setup, do not spread node VMs across multiple datacenters.
- In a topology setup spanning across multiple datacenters, do not keep datastores shared across multiple datacenters.
Persistent volume fails to be detached from a node

This problem might occur when the Kubernetes node object or node VM on vCenter Server have been deleted without draining the Kubernetes node.
Workaround:
1. Detach the volume manually from the node VM associated with the deleted node object.
2. Delete the VolumeAttachment object associated with the deleted node object.
```
kubectl patch volumeattachments.storage.k8s.io csi-<uuid> -p '{"metadata":{"finalizers":[]}}' --type=merge 
kubectl delete volumeattachments.storage.k8s.io csi-<uuid>
```
After you delete a Pod with an in-tree inline vSphere volume, the Pod permanently remains in the terminating state

Workaround: Forcefully delete the pod and manually unmount the associated volumes from the node VM.

This issue is fixed in vSphere Container Storage Plug-in version 2.5.0. For more information, see https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/1466.
When a site failure occurs, pods that were running on the worker nodes in that site remain in Terminating state

When a site failure causes all Kubernetes nodes and ESXi hosts in the cluster on that site to fail, the pods that were running on the worker nodes in that site will be stuck in Terminating state.

Workaround: Start some of the ESXi hosts in the site as soon as possible, so that vSphere HA can restart the failed Kubernetes nodes. This action ensures that the replacement pods begin to come up.
After a recovery from network partition or host failure, some nodes in the cluster do not have INTERNAL-IP or EXTERNAL-IP

After a recovery from a network partition or host failure, CPI is unable to assign INTERNAL-IP or EXTERNAL-IP to the node when it is added back to the cluster.
Workaround:
1. De-register the affected node.
  
  # kubectl delete node node-name
2. Re-register the affected node by restarting kubelet service within the affected node.
  
  # systemctl restart kubelet
3. Wait for node to register with the cluster.
4. Taint the affected nodes.
  
  # kubectl taint node node-name node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
5. Wait for CPI to initialize the node. Make sure ProviderID is set and IP address is present for the node.
After recovering from a network partition or host failure, the control plane node becomes a worker node

During network partition or host failure, the CPI might delete the node from the Kubernetes cluster if a VM is not found. After the recovery, the control plane node might become the worker node. Because of this, the pods tend to get scheduled on it unexpectedly.
You can fix this issue in two ways:

Workaround 1
1. Taint and add labels to the affected nodes.
  
  # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule
2. Delete the node from cluster.
  
  # kubeclt delete node <node name>
3. Restart kublet service within the affected node.
  
  # systemctl restart kubelet
4. Wait for the node to register with the cluster and add labels to the affected node.
  
  # kubectl label node <node name> node-role.kubernetes.io/control-plane=
  
  # kubectl label node <node name> node-role.kubernetes.io/control-plane=
5. Delete the application pods which are already scheduled on the control plane node to get scheduled on new worker nodes.
  
  # kubectl delete pod <pod name>
Workaround 2
1. Add the environment variable with SKIP_NODE_DELETION=true.
  
  # kubectl set env daemonset vsphere-cloud-controller-manager -n kube-system SKIP_NODE_DELETION=true
2. Verify whether the environment variable has been applied correctly.
  
  # kubectl describe daemonset vsphere-cloud-controller-manager -n kube-system
3. Terminate the running pods. The next pod that you create will pull the environment variable.
  
  # kubectl delete pod <pod name>
4. Wait for the pod to start.
5. View logs with `kubectl logs [POD_NAME] -n kube-system`, and confirm if everything is healthy.
Note: If you use the second method, it might result in leftover nodes and might introduce unexpected behaviors.
When a Kubernetes worker node shuts down non-gracefully, pods on that node remain in Terminating state

Pods will not be rescheduled to other healthy worker nodes. As a result, the application might face a downtime or run in degraded mode. This depends on the number of replicas of the application present on the worker node that experiences non-graceful shut down.

Workaround: Forcefully delete the pods that remain in terminating state.
After recovery from a network partition or host failure, pods might remain in containerCreating state

During a network partition or host failure, CPI might delete the node from the Kubernetes cluster if a VM is not found. After recovery, the nodes might not be automatically added back to the cluster. This results in pods remaining in containerCreating state with the error message "Volume not attached according to node status for volume" or "".
Workaround:

If the issue occurs on a control plane node, perform the following steps.
1. Restart kubelet service within the affected node.
  
  # systemctl restart kubelet
2. Wait for the node to register with the cluster. Add labels and taints to the affected node.
  
  # kubectl taint node <node name> node-role.kubernetes.io/control-plane:NoSchedule
  
  # kubectl label node <node name> node-role.kubernetes.io/control-plane=
  
  # kubectl label node <node name> node-role.kubernetes.io/control-plane=
If the issue affects a worker node, perform the following steps.
1. Restart kubelet service within the affected node.
  
  # systemctl restart kubelet
2. Taint the affected node(s).
  
  # kubectl taint node <node-name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
A PV provisioned before vSphere Container Storage Plug-in migration was enabled fails to be attached after the migration is enabled

This happens only when the datastore name contains characters that trim the trailing characters from the vmdk path. When the vmdk path is trimmed, the volume cannot register with the CNS. As a result, the volume cannot be attached to the node using vSphere Container Storage Plug-in.

For more information, see https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1451.

Workaround: This issue has been resolved in version 2.4.1. Upgrade vSphere Container Storage Plug-in to version 2.4.1.
Changes to Datacenter and Port entries in the vsphere-config-secret are not applied until you restart the vSphere Container Storage Plug-in pod

After you make changes to the Datacenter and Port entries in the vsphere-config-secret, volume life cycle operations fail.

Workaround: Restart the vsphere-csi-controller deployment pod.
Volume life cycle operations might be delayed during vSAN network partitioning

You can observe some delays in Pod creation and Pod deletion during network partitioning on a vSAN cluster. After vSphere Container Storage Plug-in retries all failed operations, the operations succeed.

This issue might occur because vCenter Server cannot reach the correct host during network partitioning. The volume fails to be created if the request reaches a host that cannot create the volume. However, during a Kubernetes retry, the volume can be created if it reaches the right host.

Workaround: None.
CreateVolume request fails with an error after you change the hostname or IP of vCenter Server in the vsphere-config-secret

After you change the hostname or IP of vCenter Server in the vsphere-config-secret and then try to create a persistent volume claim, the action fails with the following error:

failed to get Nodes from nodeManager with err virtual center wasn't found in registry

You can observe this issue in the following releases of vSphere Container Storage Plug-in: v2.0.2, v2.1.2 , v2.2.2 , v2.3.0, and v2.4.0.

Workaround: Restart the vsphere-csi-controller pod by running the following command:

kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
When you perform various operations on a volume or a node VM, you might observe error messages that appear in vCenter Server
vCenter Server might display the following error messages:
- When attaching a volume: com.vmware.vc.InvalidController : "The device '0' is referring to a nonexisting controller '1,001'."
- When detaching a volume: com.vmware.vc.NotFound : "The object or item referred to could not be found."
- When resizing a volume: com.vmware.vc.InvalidArgument : "A specified parameter was not correct: spec.deviceChange.device"
- When updating: com.vmware.vc.Timedout : "Operation timed out."
- When reconfiguring a VM: com.vmware.vc.InsufficientMemoryResourcesFault : "The available Memory resources in the parent resource pool are insufficient for the operation."
In addition, you can observe a few less frequent errors for the CSI migration feature specifically in 70u2.

For update:
- Cannot find the device '2,0xx', which is referenced in the edit or remove device operation.
- A general system error occurred: Failed to lock the file: api = DiskLib_Open, _diskPath->CValue() = /vmfs/volumes/vsan:52c77e7d8115ccfa-3ec2df6cffce6713/782c2560-d5e7-0e1d-858a-ecf4bbdbf874/kubernetes-dynamic-pvc-f077b8cd-dbfb-4ba6-a9e8-d7d8c9f4c578.vmdk
- The operation is not allowed in the current state.
For reconfigure: Invalid configuration for device '0'.
Workaround: Most of these errors are resolved after a retry from CSI.
A statefulset set replica pod remains in terminating state after you delete the statefulset
Typically, the problem occurs after you perfrom the following steps:
1. Create volumes in the Kubernetes cluster using vSphere Cloud Provider (VCP).
2. Enable the CSIMigration feature flags on kube-controller-manager, kubelet, and install vSphere Container Storage Plug-in.
3. Enable the csi-migration feature state to migrate the volumes that you previously created using VCP.
4. Create a statefulset using the migrated volumes and continue to use them in the replica set pods.
5. When you no longer need the application pods to run in the Kubernetes cluster, perform the delete operation on the statefulset.
This action might occationally result in replica set pods to remain in terminating state.
Workaround: Force delete the replica set pods in terminating state:

kubectl delete pod replica-pod-name --force --grace-period=0
Pod provisioning fails even if you do not exceed the pod limit on the worker nodes
When the number of nodes in the cluster is limited and you try to create multiple pods, the provisioning might be imbalanced and cause errors. You can observe this behavior if, for example, the cluster has three worker nodes and you try to create 200 pods. During pod creation, the following error appears:
```
attachdetach-controller AttachVolume.Attach failed for volume "pvc-2de09beb-71bb-4704-9731-5ca7648d7a74" : rpc error: >code = Internal desc = failed to attach disk: "27f09420-e0aa-4f76-a409-9fcf8a660ac7" with node: "k8s-node-801->1632443973" err failed to attach cns volume:
```
Workaround: Set the value of the MAX_VOLUMES_PER_NODE environment variable in the deployment YAML of the vSphere Container Storage Plug-in to 59 and retry creating 150 pods. This workaround ensures that the pods get evenly distributed among nodes untill they reach the count of 59.