vSphere Container Storage Plug-in supports the volume snapshot and restore capabilities. You can use a snapshot to provision a new volume, pre-populated with the snapshot data. You can also restore the existing volume to a previous state represented by the snapshot.

Volume Snapshot and Restore Requirements

Your environment must meet general requirements that apply to vSphere Container Storage Plug-in. For more information, see Preparing for Installation of vSphere Container Storage Plug-in.

In addition, follow these requirements to use the volume snapshot and restore feature with vSphere Container Storage Plug-in:

  • CSI upstream external-snapshotter/snapshot-controller version 5.0.1 or higher.

  • vSphere version 7.0 Update 3 or higher.

    The minimum version applies to both vCenter Server and ESXi.

  • Volume Snapshot CRD v1 is supported.

  • Volume Snapshot CRD v1beta1 and v1alpha1 are not supported.

Note: Volume Snapshots are supported only for block volumes.

Enable Volume Snapshot and Restore

Enable the volume snapshot and restore capabilities for vSphere Container Storage Plug-in.

Procedure

  1. Install the version of vSphere Container Storage Plug-in that supports volume snapshots.
  2. Deploy the required components using the following script available at:

    https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/manifests/vanilla/deploy-csi-snapshot-components.sh

    To obtain a detailed workflow of the script, check out by running bash deploy-csi-snapshot-components.sh -h command.

    % ./deploy-csi-snapshot-components.sh
    No existing snapshot-controller Deployment found, deploying it now..
    Start snapshot-controller deployment...
    customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
    Created CRD volumesnapshotclasses.snapshot.storage.k8s.io
    customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
    Created CRD volumesnapshotcontents.snapshot.storage.k8s.io
    customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
    Created CRD volumesnapshots.snapshot.storage.k8s.io
    ✅ Deployed VolumeSnapshot CRDs
    serviceaccount/snapshot-controller unchanged
    clusterrole.rbac.authorization.k8s.io/snapshot-controller-runner unchanged
    clusterrolebinding.rbac.authorization.k8s.io/snapshot-controller-role unchanged
    role.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
    rolebinding.rbac.authorization.k8s.io/snapshot-controller-leaderelection unchanged
    ✅ Created  RBACs for snapshot-controller
    deployment.apps/snapshot-controller created
    deployment.apps/snapshot-controller image updated
    deployment.apps/snapshot-controller patched
    deployment.apps/snapshot-controller patched
    Waiting for deployment spec update to be observed...
    Waiting for deployment "snapshot-controller" rollout to finish: 0 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 out of 2 new replicas have been updated...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
    Waiting for deployment "snapshot-controller" rollout to finish: 1 of 2 updated replicas are available...
    deployment "snapshot-controller" successfully rolled out
    
    ✅ Successfully deployed snapshot-controller
    
    No existing snapshot-validation-deployment Deployment found, deploying it now..
    creating certs in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f 
    Generating a 2048 bit RSA private key
    .............................................................................................+++
    .........................+++
    writing new private key to '/var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.HmdOwrGg7f/ca.key'
    -----
    Generating RSA private key, 2048 bit long modulus
    ...............................................................+++
    .............................................................................+++
    e is 65537 (0x10001)
    Signature ok
    subject=/CN=snapshot-validation-service.kube-system.svc
    Getting CA Private Key
    secret "snapshot-webhook-certs" deleted
    secret/snapshot-webhook-certs created
    service "snapshot-validation-service" deleted
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  2238  100  2238    0     0   9060      0 --:--:-- --:--:-- --:--:--  9024
    service/snapshot-validation-service created
    validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook.snapshot.storage.k8s.io configured
    deployment.apps/snapshot-validation-deployment created
    deployment.apps/snapshot-validation-deployment image updated
    deployment.apps/snapshot-validation-deployment patched
    Waiting for deployment spec update to be observed...
    Waiting for deployment spec update to be observed...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 0 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
    Waiting for deployment "snapshot-validation-deployment" rollout to finish: 1 old replicas are pending termination...
    deployment "snapshot-validation-deployment" successfully rolled out
    
    ✅ Successfully deployed snapshot-validation-deployment
    
    csi-snapshotter side-car not found in vSphere CSI Driver Deployment, patching..
    creating patch file in tmpdir /var/folders/31/y77ywvzd6lqc0g60r4xnfyd80000gp/T/tmp.GiMDimfZdq
    Scale down the vSphere CSI driver
    deployment.apps/vsphere-csi-controller scaled
    Patching vSphere CSI driver..
    deployment.apps/vsphere-csi-controller patched
    Scaling the vSphere CSI driver back to original state..
    deployment.apps/vsphere-csi-controller scaled
    Waiting for deployment spec update to be observed...
    Waiting for deployment spec update to be observed...
    Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 out of 3 new replicas have been updated...
    Waiting for deployment "vsphere-csi-controller" rollout to finish: 0 of 3 updated replicas are available...
    Waiting for deployment "vsphere-csi-controller" rollout to finish: 1 of 3 updated replicas are available...
    Waiting for deployment "vsphere-csi-controller" rollout to finish: 2 of 3 updated replicas are available...
    deployment "vsphere-csi-controller" successfully rolled out
    
    ✅ Successfully deployed all components for CSI Snapshot feature!
    
    Note:
    • snapshot-validation-deployment 5.0.1 validation webhook is also deployed as a part of the deployment script.
    • Perform a version check only if snapshot-controller, snapshot-validation-deployment, csi-snapshotter, and CRDs already exist.
    • If the component version number is incorrect, the deployment fails with an error message.
    • If there is a mismatch in the existing component version number, manually upgrade the component, or delete it. After deletion, the script will deploy the appropriate version of the component.

Configure Maximum Number of Snapshots per Volume

Configure the maximum number of snapshots per volume for the vSphere Container Storage Plug-in.

The configuration parameters are listed below. You must configure the parameters only when the default constraint does not work for your use cases. Otherwise, you can skip the configuration steps.
Parameter Description
global-max-snapshots-per-block-volume Global configuration parameter that applies to volumes on all kinds of datastores. By default, it is set to three.
granular-max-snapshots-per-block-volume-vsan Granular configuration parameter on vSAN datastore only. It overrides the global constraint if set, while it falls back to the global constraint if unset. A maximum value of 32 can be set under vSAN ESA setup.
granular-max-snapshots-per-block-volume-vvol Granular configuration parameter on Virtual Volumes datastore only. It overrides the global constraint if set. It falls back to the global constraint if unset.

Prerequisites

Note:
  • The best practice guideline applies only to virtual disks on VMFS and NFS datastores while not to those on Virtual Volumes and vSAN datastores.
  • Granular configuration parameters are introduced apart from the global configuration parameter.

Procedure

  1. Delete the secret that stores the vSphere configuration.
    Kubernetes does not allow you to update secret resources in place.
    kubectl delete secret vsphere-config-secret --namespace=vmware-system-csi
  2. Update the config file of vSphere Container Storage Plug-in and add configuration parameters for the snapshot feature under the [Snapshot] section.
    $ cat /etc/kubernetes/csi-vsphere.conf
    [Global]
    ...
    
    [Snapshot]
    global-max-snapshots-per-block-volume = 5 # optional, set to 3 if unset
    granular-max-snapshots-per-block-volume-vsan = 7 # optional, fall back to the global constraint if unset
    granular-max-snapshots-per-block-volume-vvol = 8 # optional, fall back to the global constraint if unset
    ...
  3. Create a new secret with the updated config file.
    kubectl create secret generic vsphere-config-secret --from-file=csi-vsphere.conf --namespace=vmware-system-csi

Using Volume Snapshot and Restore

After you enable the volume snapshot and restore capabilities for vSphere Container Storage Plug-in, you can create a snapshot dynamically or statically. You can also create a PVC from a volume snapshot.

The following is an example to create a storageclass volume snapshot. Optional parameters are commented.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-vanilla-rwo-filesystem-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"  # Optional
provisioner: csi.vsphere.vmware.com
allowVolumeExpansion: true  # Optional: only applicable to vSphere 7.0U1 and above
parameters:
   csi.storage.k8s.io/fstype: "ext4"  
#  datastoreurl: "ds:///vmfs/volumes/vsan:52cdfa80721ff516-ea1e993113acfc77/"  # Optional Parameter
#  storagepolicyname: "vSAN Default Storage Policy"  # Optional Parameter 

Volume Snapshot and Restore Limitations

Before you start using the volume snapshot and restore feature, consider the following limitations:

  • When you create a PVC from a VolumeSnapshot, it should reside on the same datastore as the original VolumeSnapshot. Otherwise, the provisioning of that PVC will fail with the following error:
    failed to provision volume with StorageClass "vmfs12": rpc error: code = Internal desc = failed to create volume. Error: failed to get the compatible datastore for create volume from snapshot 0a3ce642-2c19-4d50-9534-7889b2a6db52+fc01aaa4-29d8-4a68-90ba-b1d53bb0657d with error: failed to find datastore with URL "ds:///vmfs/volumes/62fd07ba-4b18326e-137a-1c34da62fa18/" from the input datastore list, [Datastore:datastore- 33 Datastore:datastore-34]
    Note: The datastore for the target PVC that you create from the VolumeSnapshot is determined by the StorageClass in the PVC definition. Make sure that the StorageClass of the target PVC and the StorageClass of the original source PVC point to the same datastore, which is the datastore of the source PVC. This rule also applies to the topology requirements in the StorageClass definitions. The requirements must also point to the same common datastore. Any conflicting topology requirements result in the same error as shown above.
  • You cannot delete or expand a volume that contains associated snapshots. Delete all snapshots to expand or delete a volume.
  • When you create a volume from a snapshot, ensure that the size of the volume matches the size of the snapshot.

Create Dynamically Provisioned Snapshots

You can dynamically provision a snapshot for vSphere Container Storage Plug-in.

Procedure

  1. Create a StorageClass.
    $ kubectl apply -f example-sc.yaml
    $ kubectl get sc
    NAME                                          PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
    example-vanilla-rwo-filesystem-sc (default)   csi.vsphere.vmware.com   Delete          Immediate           true                   2s
  2. Create a PVC.
    $ kubectl apply -f example-pvc.yaml
    $ kubectl get pvc
    NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        AGE
    example-vanilla-rwo-pvc   Bound    pvc-2dc37ea0-dee0-4ad3-96ca-82f0159d7532   5Gi        RWO            example-vanilla-rwo-filesystem-sc   7s
  3. Create a VolumeSnapshotClass.
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshotClass
    metadata:
      name: example-vanilla-rwo-filesystem-snapshotclass
    driver: csi.vsphere.vmware.com
    deletionPolicy: Delete
    $ kubectl apply -f example-snapshotclass.yaml
    $ kubectl get volumesnapshotclass
    NAME                                           DRIVER                   DELETIONPOLICY   AGE
    example-vanilla-rwo-filesystem-snapshotclass   csi.vsphere.vmware.com   Delete           4s
  4. Create a VolumeSnapshot.
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    metadata:
      name: example-vanilla-rwo-filesystem-snapshot
    spec:
      volumeSnapshotClassName: example-vanilla-rwo-filesystem-snapshotclass
      source:
        persistentVolumeClaimName: example-vanilla-rwo-pvc
    $ kubectl apply -f example-snapshot.yaml
    $ kubectl get volumesnapshot
    NAME                                      READYTOUSE   SOURCEPVC                 SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                                  SNAPSHOTCONTENT                                    CREATIONTIME   AGE
    example-vanilla-rwo-filesystem-snapshot   true         example-vanilla-rwo-pvc                           5Gi           example-vanilla-rwo-filesystem-snapshotclass   snapcontent-a7c00b7f-f727-4010-9b1a-d546df9a8bab   57s            58s
    Note: For more information on the yaml files mentioned in the above steps, see https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/master/example/vanilla-k8s-RWO-filesystem-volumes.

Create Pre-Provisioned Snapshots

Pre-provision a snapshot for the vSphere Container Storage Plug-in.

Prerequisites

  • Ensure that an FCD snapshot is available in your vSphere environment.
    Note:
    • Pre-provisioned CSI snapshots are supported for CNS/FCD snapshots created using Kubernetes VolumeSnapshot APIs for vSphere 7.0 Update 3 and later.
    • Pre-provisioned CSI snapshots are not supported for FCD snapshots created using FCD APIs directly.
  • Construct the snapshot handle based on the combination of FCD Volume ID and FCD Snapshot ID of the snapshot. For example, if the FCD Volume ID and FCD Snapshot ID for a FCD snapshot are 4ef058e4-d941-447d-a427-438440b7d306 and 766f7158-b394-4cc1-891b-4667df0822fa, the snapshot handle constructed is 4ef058e4-d941-447d-a427-438440b7d306+766f7158-b394-4cc1-891b-4667df0822fa.
  • Update the spec.source.snapshotHandle field in the VolumeSnapshotContent object of the example-static-snapshot.yaml with the snapshot handle constructed in the above example.

Procedure

  • Create a pre-provisioned volume snapshot.
    $ kubectl apply -f example-static-snapshot.yaml
    $ kubectl get volumesnapshot static-vanilla-rwo-filesystem-snapshot
    NAME                                     READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT                           RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                 CREATIONTIME   AGE
    static-vanilla-rwo-filesystem-snapshot   true                     static-vanilla-rwo-filesystem-snapshotcontent   5Gi                           static-vanilla-rwo-filesystem-snapshotcontent   76m            22m

Restore Volume Snapshots

You can restore a volume snapshot that is already created with vSphere Container Storage Plug-in.

Procedure

  1. Ensure that the volume snapshot that you want to restore is available in the current Kubernetes cluster.
    $ kubectl get volumesnapshot
    NAME                                      READYTOUSE   SOURCEPVC                 SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                                  SNAPSHOTCONTENT                                    CREATIONTIME   AGE
    example-vanilla-rwo-filesystem-snapshot   true         example-vanilla-rwo-pvc                           5Gi           example-vanilla-rwo-filesystem-snapshotclass   snapcontent-a7c00b7f-f727-4010-9b1a-d546df9a8bab   22m            22m
  2. Create a PVC from a volume snapshot.
    $ kubectl create -f example-restore.yaml
    $ kubectl get pvc
    NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        AGE
    example-vanilla-rwo-filesystem-restore   Bound    pvc-202c1dfc-78be-4835-89d5-110f739a87dd   5Gi        RWO            example-vanilla-rwo-filesystem-sc   78s