To back up and restore Tanzu Kubernetes (workload) clusters, you can use Velero, an open source community standard tool for backing up and restoring Kubernetes cluster objects and persistent volumes. Velero supports a variety of storage providers to store its backups.

If a workload cluster crashes and fails to recover, the infrastructure administrator can use a Velero backup to restore its contents to a new cluster, including cluster extensions and internal API objects for the workload clusters.

The following sections explain how to set up a Velero server on workload clusters, and direct it from the Velero CLI to back up and restore the clusters.

NOTE: You must create a new cluster to restore to; you cannot restore a cluster backup to an existing cluster.

Setup Overview

To back up and restore Tanzu Kubernetes Grid workload clusters, you need:

Install the Velero CLI

  1. Go to the Tanzu Kubernetes Grid downloads page and log in with your My VMware credentials.
  2. Under Product Downloads, click Go to Downloads.
  3. Scroll to the Velero entries and download the Velero CLI .gz file for your workstation OS. Its filename starts with velero-linux-, velero-mac-, or velero-windows64-.
  4. Use the gunzip command or the extraction tool of your choice to unpack the binary:

    gzip -d <RELEASE-TARBALL-NAME>.gz
    
  5. Rename the CLI binary for your platform to velero, make sure that it is executable, and add it to your PATH.

    • macOS and Linux platforms:

      1. Move the binary into the /usr/local/bin folder and rename it to velero.
      2. Make the file executable:
      chmod +x /usr/local/bin/velero
      
    • Windows platforms:

      1. Create a new Program Files\velero folder and copy the binary into it.
      2. Rename the binary to velero.exe.
      3. Right-click the velero folder, select Properties > Security, and make sure that your user account has the Full Control permission.
      4. Use Windows Search to search for env.
      5. Select Edit the system environment variables and click the Environment Variables button.
      6. Select the Path row under System variables, and click Edit.
      7. Click New to add a new row and enter the path to the velero binary.
  6. On vSphere with Tanzu:

Set Up a Storage Provider

To back up Tanzu Kubernetes Grid workload clusters, you need storage locations for:

  • Cluster object storage backups for Kubernetes metadata in workload clusters
  • Volume snapshots for data used by workload clusters

See Backup Storage Locations and Volume Snapshot Locations in the Velero documentation. Velero supports a variety of storage providers.

VMware recommends dedicating a unique storage bucket to each cluster.

Storage for vSphere

On vSphere, cluster object storage backups and volume snapshots save to the same storage location. This location must be S3-compatible external storage on Amazon Web Services (AWS), or an S3 provider such as MinIO.

To set up storage for Velero on vSphere, follow the installation procedures in the Velero Plugin for AWS repository, depending on what kind of cluster you are backing up:

Storage for and on AWS

To set up storage for Velero on AWS, follow the procedures in the Velero Plugins for AWS repository:

  1. Create an S3 bucket

  2. Set permissions for Velero

Set up S3 storage as needed for each plugin. The object store plugin stores and retrieves cluster object backups, and the volume snapshotter stores and retrieves data volumes.

Storage for and on Azure

To set up storage for Velero on Azure, follow the procedures in the Velero Plugins for Azure repository:

  1. Create an Azure storage account and blob container

  2. Get the resource group containing your VMs and disks

  3. Set permissions for Velero

Set up S3 storage as needed for each plugin. The object store plugin stores and retrieves cluster object backups, and the volume snapshotter stores and retrieves data volumes.

Deploy Velero Server to Clusters

To deploy the Velero Server to a cluster, you run the velero install command. This command creates a namespace called velero on the cluster, and places a deployment named velero in it.

velero install installs the Velero server to the current default cluster in your kubeconfig, or else you can specify a different cluster with the --kubeconfig flag.

How you run the velero install command and otherwise set up Velero on a cluster depends on your infrastructure and storage provider:

Velero Server on vSphere without Tanzu

This procedure applies to Tanzu Kubernetes Grid workload clusters. To deploy Velero to a vSphere with Tanzu Supervisor cluster that serves as your Tanzu Kubernetes Grid management cluster, see the Velero Server on vSphere with Tanzu below.

  1. Install the Velero server to the current default cluster in your kubeconfig by running velero install, as described in the Install section for Vanilla Kubernetes clusters in the Velero Plugin for vSphere v1.1.0 repository. Include option values as follows:

    • --provider aws
    • --plugins velero/velero-plugin-for-aws:v1.1.0
    • --bucket $BUCKET: name of your S3 bucket
    • --backup-location-config region=$REGION: region the bucket is in
    • --snapshot-location-config region=$REGION: region the bucket is in
    • For bucket access via username / password, include --secret-file ./velero-creds pointing to local file that looks like:

      [default]
      aws_access_key_id=<AWS_ACCESS_KEY_ID>
      aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
      
    • For bucket access via kube2iam:

      --pod-annotations iam.amazonaws.com/role=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<VELERO_ROLE_NAME>``
      --no-secret
      
    • (Optional) --kubeconfig to install the Velero server to a cluster other than the current default.

    • For additional options, see Install and start Velero.

    For example, to use MinIO as the object storage, following the MinIO server setup instructions in the Velero documentation:

    velero install --provider aws --plugins "velero/velero-plugin-for-aws:v1.1.0" --bucket velero --secret-file ./credentials-velero --backup-location-config "region=minio,s3ForcePathStyle=true,s3Url=minio_server_url" --snapshot-location-config region="default"
    

    Installing the Velero server to the cluster creates a namespace in the cluster called velero, and places a deployment named velero in it.

  2. For workload clusters with CSI-based volumes, add the Velero Plugin for vSphere. This plugin lets Velero use your S3 bucket to store CSI volume snapshots for workload data, in addition to storing cluster objects:

    1. Download the Velero Plugin for vSphere v1.1.0 image.
    2. Run velero plugin add PLUGIN-IMAGE with the plugin image name.
      • PLUGIN-IMAGE is the container image name listed in the Velero Plugin for vSphere repo v1.1.0, for example, vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0.
    3. Enable the plugin by adding the following VirtualMachine permissions to the role you created for the Tanzu Kubernetes Grid account, if you did not already include them when you created the account:
      • Configuration > Toggle disk change tracking
      • Provisioning > Allow read-only disk access
      • Provisioning > Allow virtual machine download
      • Snapshot management > Create snapshot
      • Snapshot management > Remove snapshot

Velero Server on vSphere with Tanzu

vSphere with Tanzu Supervisor clusters do not have the Kubernetes API server permissions required to retrieve Kubernetes cluster objects, so you need to install Velero with a Velero vSphere Operator that elevates Velero's permissions.

To install Velero on a Supervisor cluster, follow Installing Velero on a Supervisor Cluster in the Velero Plugin for vSphere v1.1.0 repository.

NOTE: Tanzu Kubernetes Grid does not support backing up the Kubernetes object metadata for the Supervisor cluster, which captures its state. You can use Velero to back up data volume snapshots for user workloads running on the Supervisor cluster, as well as objects and data from workload clusters managed by the Supervisor cluster.

Velero Server on AWS

To install Velero on workload clusters on AWS, follow the Install and start Velero procedure in the Velero Plugins for AWS repository.

Velero Server on Azure

To install Velero on workload clusters on Azure, follow the Install and start Velero procedure in the Velero Plugins for Azure repository.

vSphere Backup and Restore

These sections describe how to back up and restore workload clusters on vSphere.

Back Up a Cluster on vSphere

  1. Follow the Deploy Velero Server to Clusters instructions above to deploy a Velero server onto the cluster, along with the Velero Plugin for vSphere if needed.

  2. Back up the cluster:

    velero backup create your_backup_name --exclude-namespaces=tkg-system
    

    Excluding tkg-system objects avoids creating duplicate cluster API objects when restoring a cluster.

Restore a Cluster on vSphere

  1. Create a new cluster. You cannot restore a cluster backup to an existing cluster.

  2. Follow the Deploy Velero Server to Clusters instructions above to deploy a Velero server onto the new cluster, along with the Velero Plugin for vSphere if needed.

  3. Restore the cluster:

    velero restore create your_restore_name --from-backup your_backup_name
    

AWS Backup and Restore

These sections describe how to back up and restore clusters on AWS.

Back Up a Cluster on AWS

  1. Follow the Velero Plugin for AWS setup instructions to install Velero server on the cluster.

  2. Back up the cluster:

    velero backup create your_backup_name --exclude-namespaces=tkg-system
    

    Excluding tkg-system objects avoids creating duplicate cluster API objects when restoring a cluster.

Restore a Cluster on AWS

  1. Create a new cluster. You cannot restore a cluster backup to an existing cluster.

  2. Follow the Velero Plugin for AWS setup instructions to install Velero server on the new cluster.

  3. Restore the cluster:

    velero backup get
    velero restore create your_restore_name --from-backup your_backup_name
    

Azure Backup and Restore

These sections describe how to back up and restore clusters on Azure.

Back Up a Cluster on Azure

  1. Follow the Velero Plugin for Azure setup instructions to install Velero server on the cluster.

  2. Back up the cluster:

    velero backup create your_backup_name --exclude-namespaces=tkg-system
    

    Excluding tkg-system objects avoids creating duplicate cluster API objects when restoring a cluster.

  3. If velero backup returns a transport is closing error, try again after increasing the memory limit, as described in Update resource requests and limits after install in the Velero documentation.

Restore a Cluster on Azure

  1. Create a new cluster. You cannot restore a cluster backup to an existing cluster.

  2. Follow the Velero Plugin for Azure setup instructions to install Velero server on the new cluster.

  3. Restore the cluster:

    velero backup get
    velero restore create your_restore_name --from-backup your_backup_name
    
check-circle-line exclamation-circle-line close-line
Scroll to top icon