This topic explains how to use dynamic storage in Tanzu Kubernetes Grid clusters.

Overview: PersistentVolume, PersistentVolumeClaim, and StorageClass

Within a Kubernetes cluster, PersistentVolume (PV) objects provide shared storage for cluster pods that is unaffected by pod lifecycles. Storage is provisioned to the PV through a PersistentVolumeClaim (PVC) object, which defines how much and how the pod accesses the underlying storage. For more information, see Persistent Volumes in the Kubernetes documentation.

Cluster administrators can define StorageClass objects that let cluster users dynamically create PVC and PV objects with different storage types and rules. Tanzu Kubernetes Grid also provides default StorageClass objects that let users provision persistent storage in a turnkey environment.

StorageClass objects include a provisioner field identifying the internal or external service plug-in that provisions PVs, and a parameters field that configures the storage class based on provisioner options available. For more information, see Storage Classes in the Kubernetes documentation.

Supported Storage Types

Tanzu Kubernetes Grid supports StorageClass objects for different storage types, provisioned by Kubernetes internal ("in-tree") or external ("out-of-tree") plug-ins.

Storage Types

  • vSphere Cloud Native Storage (CNS)
  • Amazon EBS
  • Azure Disk
  • iSCSI
  • NFS

See Default Storage Classes below for vSphere CNS, Azure EBS, and Azure Disk default storage classes.

Plug-in Locations

  • Kubernetes internal ("in-tree") storage.
    • Ships with core Kubernetes; provider values are prefixed with kubernetes.io, e.g. kubernetes.io/aws-ebs.
  • External ("out-of-tree") storage.
    • Can be anywhere defined by provider value, e.g. csi.vsphere.vmware.com.
    • Follow the Container Storage Interface (CSI) standard for external storage.

Default Storage Classes

Tanzu Kubernetes Grid provides default StorageClass objects that let cluster users provision persistent storage on their infrastructure in a turnkey environment, without needing StorageClass objects created by a cluster administrator.

To use the default storage class for your infrastructure, define a PersistentVolumeClaim object with no spec.storageClassName setting, or with spec.storageClassName set to default.

The Tanzu Kubernetes Grid default storage class definitions are:

vSphere CNS

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
  storagePolicyName: optional

See the vSphere CSI storage class parameters in the Kubernetes documentation.

Amazon EBS

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: default
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs

See the Amazon EBS storage class parameters in the Kubernetes documentation.

Azure Disk

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: default
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/azure-disk
parameters:
  kind: Managed
  storageaccounttype: Standard_LRS
  cachingmode: ReadOnly
volumeBindingMode: WaitForFirstConsumer

See the Azure Disk storage class parameters in the Kubernetes documentation.

Set Up CNS and Create a Storage Policy (vSphere)

vSphere administrators can set up vSphere CNS and create storage policies for virtual disk (VMDK) storage, based on the needs of Tanzu Kubernetes Grid cluster users.

You can use either vSAN or local VMFS (Virtual Machine File System) for persistent storage in a Kubernetes cluster, as follows:

vSAN Storage:

To create a storage policy for vSAN storage, follow the instructions in Create a Storage Policy in the vSphere documentation. Make sure to:

  • In the Policy structure pane, under Datastore specific rules, select Enable rules for "vSAN" storage.
  • Configure other panes or accept defaults as needed.
  • Record the storage policy name for reference as the storagePolicyName value in StorageClass objects.

Local VMFS Storage:

To create a storage policy for local storage, apply a tag to the storage and create a storage policy based on the tag as follows:

  1. From the top-level vSphere menu, select Tags & Custom Attributes

  2. In the Tags pane, select Categories and click New.

  3. Enter a category name, such as tkg-storage. Use the checkboxes to associate it with Datacenter and the storage objects, Folder and Datastore. Click Create.

  4. From the top-level Storage view, select your VMFS volume, and in its Summary pane, click Tags > Assign....

  5. From the Assign Tag popup, click Add Tag.

  6. From the Create Tag popup, give the tag a name, such as tkg-storage-ds1 and assign it the Category you created. Click OK.

  7. From Assign Tag, select the tag and click Assign.

  8. From top-level vSphere, select VM Storage Policies > Create a Storage Policy. A configuration wizard starts.

  9. In the Name and description pane, enter a name for your storage policy. Record the storage policy name for reference as the storagePolicyName value in StorageClass objects.

  10. In the Policy structure pane, under Datastore specific rules, select Enable tag-based placement rules.

  11. In the Tag based placement pane, click Add Tag Rule and configure:

    • Tag category: Select your category name
    • Usage option: Use storage tagged with
    • Tags: Browse and select your tag name
  12. Confirm and configure other panes or accept defaults as needed, then click Review and finish. Finish to create the storage policy.

Create a Storage Class

Cluster administrators can create a new storage class as follows:

  1. On vSphere, select or create the VM storage policy to use as the basis for the Kubernetes StorageClass.
  2. Create a StorageClass configuration .yaml with provisioner, parameters, and other options.
    • On vSphere, associate a Kubernetes storage class with a vSphere storage policy by setting its storagePolicyName parameter to the vSphere storage policy name, as a double-quoted string.
  3. Pass the file to kubectl create -f
  4. Verify the storage class by running kubectl describe storageclass <storageclass metadata.name>.

Examples:

Use a Storage Class in a Cluster

Cluster users provision persistent storage for their cluster nodes by including it in a pod configuration as follows:

  1. Select or create a storage class.

    • Select:
      • You can always use the default storage class for your infrastructure, as described in Default Storage Classes, above.
      • To list available storage classes, run kubectl get storageclass.
    • Create
  2. Create a PVC and its PV:

    1. Create a PersistentVolumeClaim configuration .yaml with spec.storageClassName set to the metadata.name value of your StorageClass object. For an example, see Enabling Dynamic Provisioning in the Kubernetes documentation.
    2. Pass the file to kubectl create -f
    3. Run kubectl describe pvc <pvc metadata.name> to verify the PVC.
    4. A PV is automatically created with the PVC. Record its name, listed in the kubectl describe pvd output after Successfully provisioned volume.
    5. Run kubectl describe pv <pv unique name> to verify the PV.
  3. Create a pod using the PVC:

    1. Create a Pod configuration .yaml that sets spec.volumes to include your PVC under persistentVolumeClaim.claimName. For an example, see Dynamic Provisioning and StorageClass API in the vSphere Storage for Kubernetes documentation.
    2. Pass the file to kubectl create -f
    3. Run kubectl get pod <pod metadata.name> to verify the pod.
check-circle-line exclamation-circle-line close-line
Scroll to top icon