Dynamic Storage

This topic explains how to use persistent volumes and storage classes to implement dynamic storage for Tanzu Kubernetes Grid (TKG) workload clusters.

Overview: PersistentVolume, PersistentVolumeClaim, and StorageClass

Within a Kubernetes cluster, PersistentVolume (PV) objects provide shared storage for cluster pods that is unaffected by pod lifecycles. Storage is provisioned to the PV through a PersistentVolumeClaim (PVC) object, which defines how much and how the pod accesses the underlying storage. For more information, see Persistent Volumes in the Kubernetes documentation.

Cluster administrators can define StorageClass objects that let cluster users dynamically create PVC and PV objects with different storage types and rules. Tanzu Kubernetes Grid also provides default StorageClass objects that let users provision persistent storage in a turnkey environment.

StorageClass objects include a provisioner field identifying the internal or external service plug-in that provisions PVs, and a parameters field that associates the Kubernetes storage class with storage options defined at the infrastructure level, such as VM Storage Policies in vSphere. For more information, see Storage Classes in the Kubernetes documentation.

Supported Storage Types

Tanzu Kubernetes Grid supports StorageClass objects for different storage types, provisioned by Kubernetes internal (“in-tree”) or external (“out-of-tree”) plug-ins.

Storage Types

  • vSphere Cloud Native Storage (CNS)
  • Amazon EBS
  • Azure Disk
  • Azure File
  • iSCSI
  • NFS
Note

vSphere CSI does not support Storage DRS, and supports Storage vMotion with conditions described in vSphere Functionality Supported by vSphere Container Storage Plug-in in the vSphere CSI documentation.

vSphere CSI support Storage vMotion with conditions, please read CSI doc for more details.

See Default Storage Classes for vSphere CNS, Amazon EBS, and Azure Disk default storage classes.

External Provisioning

In TKG v2.3, all default storage classes use external (“out-of-tree”) storage provisioning, not “in-tree” provisioning.

  • The storage classes do not ship with core Kubernetes.
  • StorageClass object provisioner values are not prefixed with kubernetes.io.
  • Provisioners follow the Container Storage Interface (CSI) standard for external storage.
  • Workload clusters with default storage classes deployed by previous versions of TKG may have in-tree storage provisioning. See Create Persistent Volumes with Storage Classes.

Default Storage Classes

Tanzu Kubernetes Grid provides default StorageClass objects that let workload cluster users provision persistent storage on their infrastructure in a turnkey environment, without needing StorageClass objects created by a cluster administrator.

The ENABLE_DEFAULT_STORAGE_CLASS variable is set to true by default in the cluster configuration file passed to --file option of tanzu cluster create, to enable the default storage class for a workload cluster.

Important

Do not modify the default storage class definitions. To customize a storage class, create a new StorageClass definition with a different name instead of modifying the default object created by TKG.

The Tanzu Kubernetes Grid default storage class definitions are:

vSphere CNS

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
    kapp.k14s.io/update-strategy: skip
provisioner: csi.vsphere.vmware.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagePolicyName: optional

See the vSphere CSI storage class parameters in the Kubernetes documentation.

Amazon EBS

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com

See the Amazon EBS CSI Driver storage class parameters in the AWS documentation.

Azure Disk

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: default
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: disk.csi.azure.com
parameters:
  kind: Managed
  storageaccounttype: Standard_LRS
  cachingmode: ReadOnly
volumeBindingMode: WaitForFirstConsumer

See the Azure Disk CSI Driver storage class parameters in the Azure documentation.

Azure File

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azure-file
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: file.csi.azure.com
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
allowVolumeExpansion: true
parameters:
  skuName: Premium_LRS

See the Azure File storage class parameters in the Kubernetes documentation.

The default storage class settings can be customized. Since most StorageClass fields are immutable, it is necessary to delete and re-create the default storage class:

  1. Prepare a StorageClass YAML file with the custom default storage class specification. The name must be default, and it must have the storageclass.kubernetes.io/is-default-class: "true" annotation.
  2. Delete the TKG-generated default storage class by running kubectl delete storageclass default.
  3. Create the custom default storage class by passing the file from step 1 to kubectl create -f.

Note, creation of the TKG default storage class can also be disabled during cluster creation by specifying ENABLE_DEFAULT_STORAGE_CLASS: false in the cluster configuration file.

Set Up CNS and Create a Storage Policy (vSphere)

vSphere administrators can set up vSphere CNS and create storage policies for virtual disk (VMDK) storage, based on the needs of Tanzu Kubernetes Grid cluster users.

You can use either vSAN or local VMFS (Virtual Machine File System) for persistent storage in a Kubernetes cluster, as follows:

vSAN Storage:

To create a storage policy for vSAN storage in the vSphere Client, browse to Home > Policies and Profiles > VM Storage Policies and click Create to launch the Create VM Storage Policy wizard.

Follow the instructions in Create a Storage Policy in the vSphere documentation. Make sure to:

  • In the Policy structure pane, under Datastore specific rules, select Enable rules for “vSAN” storage.
  • Configure other panes or accept defaults as needed.
  • Record the storage policy name for reference as the storagePolicyName value in StorageClass objects.

Local VMFS Storage:

To create a storage policy for local storage, apply a tag to the storage and create a storage policy based on the tag as follows:

  1. From the top-level vSphere menu, select Tags & Custom Attributes

  2. In the Tags pane, select Categories and click New.

  3. Enter a category name, such as tkg-storage. Use the checkboxes to associate it with Datacenter and the storage objects, Folder and Datastore. Click Create.

  4. From the top-level Storage view, select your VMFS volume, and in its Summary pane, click Tags > Assign….

  5. From the Assign Tag popup, click Add Tag.

  6. From the Create Tag popup, give the tag a name, such as tkg-storage-ds1 and assign it the Category you created. Click OK.

  7. From Assign Tag, select the tag and click Assign.

  8. From top-level vSphere, select VM Storage Policies > Create a Storage Policy. A configuration wizard starts.

  9. In the Name and description pane, enter a name for your storage policy. Record the storage policy name for reference as the storagePolicyName value in StorageClass objects.

  10. In the Policy structure pane, under Datastore specific rules, select Enable tag-based placement rules.

  11. In the Tag based placement pane, click Add Tag Rule and configure:

    • Tag category: Select your category name
    • Usage option: Use storage tagged with
    • Tags: Browse and select your tag name

  12. Confirm and configure other panes or accept defaults as needed, then click Review and finish. Finish to create the storage policy.

Create a Custom Storage Class

Cluster administrators can create a new storage class as follows:

  1. On vSphere, select or create the VM storage policy to use as the basis for the Kubernetes StorageClass.
  2. Create a StorageClass configuration .yaml with provisioner, parameters, and other options.
    • On vSphere, associate a Kubernetes storage class with a vSphere storage policy by setting its storagePolicyName parameter to the vSphere storage policy name, as a double-quoted string.
  3. Pass the file to kubectl create -f
  4. Verify the storage class by running kubectl describe storageclass <storageclass metadata.name>.

For example, see Enabling Dynamic Provisioning in the Kubernetes documentation.

For vSphere CSI information and resources, see VMware vSphere Container Storage Plug-in Documentation.

Use a Custom Storage Class in a Cluster

To provision persistent storage for their cluster nodes that does not use one of the Default Storage Classes described above, cluster users include a custom storage class in a pod configuration as follows:

  1. Set the context of kubectl to the cluster. For example:

    kubectl config use-context my-cluster-admin@my-cluster

  2. Select or create a storage class.

    • Select:
      • To list available storage classes, run kubectl get storageclass.
    • Create

  3. Create a PVC and its PV:

    1. Create a PersistentVolumeClaim configuration .yaml with spec.storageClassName set to the metadata.name value of your StorageClass object. For an example, see Enabling Dynamic Provisioning in the Kubernetes documentation.
    2. Pass the file to kubectl create -f
    3. Run kubectl describe pvc <pvc metadata.name> to verify the PVC.
    4. A PV is automatically created with the PVC. Record its name, listed in the kubectl describe pvc output after Successfully provisioned volume.
    5. Run kubectl describe pv <pv unique name> to verify the PV.

  4. Create a pod using the PVC:

    1. Create a Pod configuration .yaml that sets spec.volumes to include your PVC under persistentVolumeClaim.claimName. For an example, see Dynamically Provision a Block Volume with vSphere Container Storage Plug-in in the vSphere Container Storage Plug-in documentation.
    2. Pass the file to kubectl create -f
    3. Run kubectl get pod <pod metadata.name> to verify the pod.

Enable Volume Expansion for vSphere CSI (vSphere 7)

To enable volume expansion for vSphere CSI storage used by workload clusters, you need to add a csi-resizer sidecar pod to the cluster’s CSI processes.

The CSI configuration for workload clusters is encoded as a Kubernetes secret. This procedure adds the csi-resizer process by revising the CSI configuration secret. It adds to the secret a stringData definition that combines two encoded configuration data strings: a values.yaml string containing the secret’s prior CSI configuration data, and a new overlays.yaml string that deploys the csi-resizer pod.

Note

Online volume expansion is supported in vSphere 7.0 as of Update 2; see Volume Expansion in vSphere with Tanzu.

  1. Log into the management cluster for the workload cluster you are changing, and run tanzu cluster list if you need to retrieve the name of the workload cluster.

  2. Retrieve the name of the CSI secret for the workload cluster, using label selectors vsphere-csi and the cluster name:

    $ kubectl get secret \
  -l tkg.tanzu.vmware.com/cluster-name=NAME_OF_WORKLOAD_CLUSTER \
  -l tkg.tanzu.vmware.com/addon-name=vsphere-csi
my-wc-vsphere-csi-secret

  3. Save a backup of the secret’s content, in YAML format, to vsphere-csi-secret.yaml:

    kubectl get secret my-wc-vsphere-csi-secret -o yaml > vsphere-csi-secret.yaml

  4. Output the secret’s current content again, with the data.values values base64-decoded into plain YAML.

    $ kubectl get secret my-wc-vsphere-csi-secret -o jsonpath={.data.values\\.yaml} | base64 -d

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
vsphereCSI:
CSIAttacherImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/csi-attacher
  tag: v3.0.0_vmware.1
  pullPolicy: IfNotPresent
vsphereCSIControllerImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/vsphere-block-csi-driver
  tag: v2.1.1_vmware.1
  pullPolicy: IfNotPresent
livenessProbeImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/csi-livenessprobe
  tag: v2.1.1_vmware.1
  pullPolicy: IfNotPresent
vsphereSyncerImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/volume-metadata-syncer
  tag: v2.1.1_vmware.1
  pullPolicy: IfNotPresent
CSIProvisionerImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/csi-provisioner
  tag: v2.0.0_vmware.1
  pullPolicy: IfNotPresent
CSINodeDriverRegistrarImage:
  repository: projects.registry.vmware.com/tkg
  path: csi/csi-node-driver-registrar
  tag: v2.0.1_vmware.1
  pullPolicy: IfNotPresent
namespace: kube-system
clusterName: wc-1
server: 10.170.104.114
datacenter: /dc0
publicNetwork: VM Network
username: <MY-VSPHERE-USERNAME>
password: <MY-VSPHERE-PASSWORD>

  5. Open vsphere-csi-secret.yaml in an editor and do the following to make it look like the code below:

    1. Delete the existing definition for values.yaml, which is a long string.
    2. After the first line, add a line that defines stringData and indent values.yaml to make it the first element.
    3. Copy the data.values output from the previous step.
    4. After the third line, paste in the data.values output and indent it as the value of values.yaml.
    5. Immediately below the values.yaml definition, add another stringData definition for overlays.yaml as shown below. Do not modify other definitions in the file.
    apiVersion: v1
stringData:
  values.yaml: |
    #@data/values
    #@overlay/match-child-defaults missing_ok=True
    ---
    vsphereCSI:
      CSIAttacherImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/csi-attacher
        tag: v3.0.0_vmware.1
        pullPolicy: IfNotPresent
      vsphereCSIControllerImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/vsphere-block-csi-driver
        tag: v2.1.1_vmware.1
        pullPolicy: IfNotPresent
      livenessProbeImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/csi-livenessprobe
        tag: v2.1.1_vmware.1
        pullPolicy: IfNotPresent
      vsphereSyncerImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/volume-metadata-syncer
        tag: v2.1.1_vmware.1
        pullPolicy: IfNotPresent
      CSIProvisionerImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/csi-provisioner
        tag: v2.0.0_vmware.1
        pullPolicy: IfNotPresent
      CSINodeDriverRegistrarImage:
        repository: projects.registry.vmware.com/tkg
        path: csi/csi-node-driver-registrar
        tag: v2.0.1_vmware.1
        pullPolicy: IfNotPresent
      namespace: kube-system
      clusterName: wc-1
      server: 10.170.104.114
      datacenter: /dc0
      publicNetwork: VM Network
      username: <MY-VSPHERE-USERNAME>
      password: <MY-VSPHERE-PASSWORD>
  overlays.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"kind": "Deployment", "metadata": {"name": "vsphere-csi-controller"}})
    ---
    spec:
      template:
        spec:
          containers:
          #@overlay/append
            - name: csi-resizer
              image: projects.registry.vmware.com/tkg/kubernetes-csi_external-resizer:v1.0.0_vmware.1
              args:
                - "--v=4"
                - "--timeout=300s"
                - "--csi-address=$(ADDRESS)"
                - "--leader-election"
              env:
                - name: ADDRESS
                  value: /csi/csi.sock
              volumeMounts:
                - mountPath: /csi
                  name: socket-dir
kind: Secret
...

  6. Run kubectl apply to update the cluster’s secret with the revised definitions and re-create the csi-controller pod:

    kubectl apply -f vsphere-csi-secret.yaml

  7. To verify that the vsphere-csi-controller and external resizer are working on the cluster:

    1. Confirm that vsphere-csi-controller is running on the workload cluster with six healthy pods:

      $ kubectl get pods -n kube-system -l app=vsphere-csi-controller
NAME                                     READY   STATUS    RESTARTS   AGE
vsphere-csi-controller-<ID-HASH>   6/6     Running   0          6m49s

    2. Check the logs of the vsphere-csi-controller to see that the external resizer started.

      $ kubectl logs vsphere-csi-controller-<ID-HASH> -n kube-system -c csi-resizer
I0308 23:44:45.035254       1 main.go:79] Version : v1.0.0-0-gb22717d
I0308 23:44:45.037856       1 connection.go:153] Connecting to unix:///csi/csi.sock
I0308 23:44:45.038572       1 common.go:111] Probing CSI driver for readiness
I0308 23:44:45.040579       1 csi_resizer.go:77] CSI driver name: "csi.vsphere.vmware.com"
W0308 23:44:45.040612       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0308 23:44:45.042184       1 controller.go:117] Register Pod informer for resizer csi.vsphere.vmware.com
I0308 23:44:45.043182       1 leaderelection.go:243] attempting to acquire leader lease  kube-system/external-resizer-csi-vsphere-vmware-com...
I0308 23:44:45.073383       1 leaderelection.go:253] successfully acquired lease kube-system/external-resizer-csi-vsphere-vmware-com
I0308 23:44:45.076028       1 leader_election.go:172] new leader detected, current leader: vsphere-csi-controller-87d7dcf48-jcht2
I0308 23:44:45.079332       1 leader_election.go:165] became leader, starting
I0308 23:44:45.079638       1 controller.go:241] Starting external resizer csi.vsphere.vmware.com

For more information about expanding vSphere CSI storage volumes in online or offline mode, see Volume Expansion in vSphere with Tanzu.

Topology-Aware Volume Provisioning (vSphere)

For workload clusters created from a standalone management cluster, you can configure topology-aware local storage volume provisioning. Topology-aware volume provisioning allows Kubernetes to make intelligent decisions when dynamically provisioning volumes. Kubernetes gets scheduler input on the best place to provision a volume for a pod.

Create an availability zone (AZ):

  1. Follow the instruction in Running Clusters Across Multiple Availability Zones to create the following in vSphere:

    1. Add host group and VM group.

    2. Add rules to limit the VM group in the host group.

      One VM group and host group is used to run the AZ in the workload cluster.

      Note

      The limit rules are for local volume configurations only. You do not need to create limit rules if you are deploying to multiple availability zones.

    3. Deploy the custom resource definitions (CRDs) VSphereFailureDomain and VSphereDeploymentZone in the standalone management cluster.

      The AZ will map to a VSphereDeploymentZone and then to a host group in vSphere.

  2. Add a new AZ. You can add a new AZ using a ytt overlay configuration or using Tanzu CLI.

    ytt
    The following shows a ytt overlay configuration in a legacy cluster and a classy cluster. For information about how to download and install ytt, see Install the Carvel Tools.

    Legacy cluster

    #@overlay/match by=overlay.subset({"kind":"MachineDeployment", "metadata":{"name": "${CLUSTER_NAME}-md-0"}})
  ---
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: MachineDeployment
  metadata:
    name: #@ "{}-md-0".format(data.values.CLUSTER_NAME)
  spec:
    clusterName: #@ data.values.CLUSTER_NAME
    replicas: #@ data.values.WORKER_MACHINE_COUNT_0
    selector:
      matchLabels: null
    strategy:
      type: #@ verify_and_configure_machine_deployment_rollout_strategy(data.values.WORKER_ROLLOUT_STRATEGY)
    template:
      metadata:
        labels:
          node-pool: #@ "{}-worker-pool".format(data.values.CLUSTER_NAME)
      spec:
        clusterName: #@ data.values.CLUSTER_NAME
        version: #@ data.values.KUBERNETES_VERSION
        bootstrap:
          configRef:
            name: #@ "{}-md-0".format(data.values.CLUSTER_NAME)
            apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
            kind: KubeadmConfigTemplate
        infrastructureRef:
          name: #@ "{}-md-0".format(data.values.CLUSTER_NAME)
          apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
          kind: AWSMachineTemplate
        failureDomain: #@ default_az_0

    Classy cluster

    workers:
  machineDeployments:
  #@overlay/match by=overlay.index(0)
  - class: tkg-worker
    name: md-0
    replicas: #@ data.values.WORKER_MACHINE_COUNT_0
    #@overlay/match missing_ok=True
    failureDomain: #@ default_az_0
    metadata:
      annotations:
        run.tanzu.vmware.com/resolve-os-image: #@ "ami-region={},os-name={},os-arch={}".format(data.values.AWS_REGION, data.values.OS_NAME, data.values.OS_ARCH)
    Tanzu CLI
    You can create a new AZ to support topology aware local storage provisioning using Tanzu CLI.

    Legacy cluster

    tanzu cl node-pool set cl-name -f node-pool.yaml

# node-pool.yaml
name: vsphere-wc-1-manual-node-pool
replicas: 1
az: "rack4"

    Classy cluster

    See Set (Create) in Node Pools Configuration for ClusterClass-based clusters.

check-circle-line exclamation-circle-line close-line
Scroll to top icon