RSS
 

Back Up and Restore Cluster Workloads

To back up and restore the workloads hosted by workload clusters in Tanzu Kubernetes Grid, you can use Velero, an open source community standard tool for backing up and restoring Kubernetes cluster objects and persistent volumes. Velero supports a variety of storage providers to store its backups.

You can use Velero to back up and restore a workload cluster’s current workloads and persistent volumes state, for entire clusters or specific namespaces. You can configure pre- and post-hooks for backup and restore to run custom processes before or after backup and restore events. If some aspects of workload or cluster state are not well-suited to backup/restore, you can exclude them.

A Tanzu Kubernetes Grid subscription includes support for VMware’s tested, compatible distribution of Velero available from the Tanzu Kubernetes Grid downloads page.

If a cluster crashes and fails to recover, the infrastructure administrator can use a Velero backup to restore its contents to a new cluster. Velero does not back up or restore the cluster infrastructure itself.

To back up and restore vSphere with Tanzu clusters, including Supervisor Clusters and the workload clusters that they create, see Backing Up and Restoring vSphere with Tanzu.

Note

Backing up Tanzu Kubernetes Grid management clusters is not supported. For guidance backing up or restoring management clusters, please contact your VMware representative.

Setup Overview

The following sections explain how to set up a Velero server on workload clusters created by Tanzu Kubernetes Grid, and direct it from the Velero CLI to back up and restore the cluster’s workload and persistent volume contents.

To back up and restore Tanzu Kubernetes Grid workload cluster contents, you need:

Install the Velero CLI

To install the Velero CLI v1.9.2, do the following:

Caution

If you have already installed Velero CLI v1.8.1 or earlier, as distributed with prior versions of TKG, you need to upgrade to v1.9.2. Older Velero versions do not work with the CRDs used in v1.9 and later.

  1. Go to the Tanzu Kubernetes Grid downloads page and log in with your VMware Customer Connect credentials.
  2. Under Product Downloads, click Go to Downloads.
  3. Scroll to the Velero entries and download the Velero CLI .gz file for your workstation OS. Its filename starts with velero-linux-, velero-mac-, or velero-windows64-.

  4. Use the gunzip command or the extraction tool of your choice to unpack the binary:

    gzip -d <RELEASE-TARBALL-NAME>.gz

  5. Rename the CLI binary for your platform to velero, make sure that it is executable, and add it to your PATH.

    • macOS and Linux platforms:

      1. Move the binary into the /usr/local/bin folder and rename it to velero.
      2. Make the file executable:
      chmod +x /usr/local/bin/velero

    • Windows platforms:

      1. Create a new Program Files\velero folder and copy the binary into it.
      2. Rename the binary to velero.exe.
      3. Right-click the velero folder, select Properties > Security, and make sure that your user account has the Full Control permission.
      4. Use Windows Search to search for env.
      5. Select Edit the system environment variables and click the Environment Variables button.
      6. Select the Path row under System variables, and click Edit.
      7. Click New to add a new row and enter the path to the velero binary.

Set Up a Storage Provider

To back up Tanzu Kubernetes Grid workload cluster contents, you need storage locations for:

  • Cluster object storage backups for Kubernetes metadata in clusters
  • Volume snapshots for data used by clusters

See Backup Storage Locations and Volume Snapshot Locations in the Velero documentation. Velero supports a variety of storage providers.

VMware recommends dedicating a unique storage bucket to each cluster.

Storage for vSphere

On vSphere, cluster object storage backups and volume snapshots save to the same storage location. This location must be S3-compatible external storage on Amazon Web Services (AWS), or an S3 provider such as MinIO.

To set up storage for Velero on vSphere, see Velero Plugin for vSphere in Vanilla Kubernetes Cluster for the v1.4.0 plugin.

Storage for and on AWS

To set up storage for Velero on AWS, follow the procedures in the Velero Plugins for AWS repository:

  1. Create an S3 bucket

  2. Set permissions for Velero

Set up S3 storage as needed for each plugin. The object store plugin stores and retrieves cluster object backups, and the volume snapshotter stores and retrieves data volumes.

Storage for and on Azure

To set up storage for Velero on Azure, follow the procedures in the Velero Plugins for Azure repository:

  1. Create an Azure storage account and blob container

  2. Get the resource group containing your VMs and disks

  3. Set permissions for Velero

Set up S3 storage as needed for each plugin. The object store plugin stores and retrieves cluster object backups, and the volume snapshotter stores and retrieves data volumes.

Deploy Velero Server to Clusters

To deploy the Velero Server to a workload cluster, you run the velero install command. This command creates a namespace called velero on the cluster, and places a deployment named velero in it.

Update CRDs from Prior Velero Versions

If your workload clusters have Velero v1.8.1 or prior installed, do the following to update its CRDs to work with the Velero v1.9.2 CLI:

  1. Upgrade the CRDs:

    velero install --crds-only --dry-run -o yaml | kubectl apply -f -

  2. Update the Velero image location:

    kubectl -n velero set image deployment velero velero=velero/velero:v1.9.2

  3. Update the Velero plugin image locations by running one or more of the following commands, based on which plugins are installed:

    kubectl -n velero set image deployment velero velero-plugin-for-csi=velero/velero-plugin-for-csi:v0.3.1
kubectl -n velero set image deployment velero velero-plugin-for-aws=velero/velero-plugin-for-aws:v1.5.1
kubectl -n velero set image deployment velero velero-plugin-for-microsoft-azure=velero/velero-plugin-for-microsoft-azure:v1.5.1
kubectl -n velero set image deployment velero velero-plugin-for-vsphere=vsphereveleroplugin/velero-plugin-for-vsphere:v1.4.0

  4. Update the Restic DaemonSet image location:

    kubectl -n velero set image DaemonSet restic restic=velero/velero:v1.9.2

Velero Install Options

To install Velero, you run velero install with the following options:

  • --provider $PROVIDER: For example, aws
  • --plugins projects.registry.vmware.com/tkg/velero/velero-plugin-for-aws:v1.5.1_vmware.1
  • --bucket $BUCKET: The name of your S3 bucket
  • --backup-location-config region=$REGION: The AWS region the bucket is in
  • --snapshot-location-config region=$REGION: The AWS region the bucket is in
  • (Optional) --kubeconfig to install the Velero server to a cluster other than the current default.

  • (Optional) --secret-file ./VELERO-CREDS one way to give Velero access to an S3 bucket on AWS is to pass in to this option a local VELERO-CREDS file that looks like:

    [default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

  • For additional options, see Install and start Velero.

How you run the velero install command and otherwise set up Velero on a cluster depends on your infrastructure and storage provider, as described in the sections below.

Install Velero on Clusters on vSphere

This procedure applies to workload clusters created by a Tanzu Kubernetes Grid management cluster.

To deploy Velero to a vSphere with Tanzu Supervisor Cluster that serves as your Tanzu Kubernetes Grid management cluster, see Backing Up and Restoring vSphere with Tanzu.

  1. If the workload cluster already has Velero installed from TKG v1.6.x or earlier, follow the steps in Update CRDs from Prior Velero Versions.

  2. Install the Velero server to the current default cluster in your kubeconfig by running velero install with the options listed in Velero Install Options above.

    • For example, to use MinIO as the object storage, following the MinIO server setup instructions in the Velero documentation:

      velero install --provider aws --plugins "projects.registry.vmware.com/tkg/velero/velero-plugin-for-aws:v1.5.1_vmware.1" --bucket velero --secret-file ./credentials-velero --backup-location-config "region=minio,s3ForcePathStyle=true,s3Url=minio_server_url" --snapshot-location-config region="default"

    • For more information, see the Install section for Vanilla Kubernetes clusters in the Velero Plugin for vSphere v1.4.0 repository.

    • Installing the Velero server to the cluster creates a namespace in the cluster called velero, and places a deployment named velero in it.

  3. If you did not pass a credentials file to --secret-file with velero install above, configure grant access to Velero from your backup S3 bucket. On AWS, for example, attach a policy to the IAM role nodes.tkg.cloud.vmware.com, which governs hosted applications like Velero, to allow access to the S3 bucket.

  4. For workload clusters deployed to vSphere, add the Velero Plugin for vSphere. This plugin lets Velero use your S3 bucket to store CSI volume snapshots for workload data, in addition to storing cluster objects:

    1. Download the Velero Plugin for vSphere v1.4.0 image.

    2. Retrieve and decode the vSphere credentials used by your CSI driver into a secret configuration file csi-vsphere.conf:

      kubectl -n vmware-system-csi get secret vsphere-config-secret -o jsonpath='{.data.csi-vsphere\.conf}'| base64 -d > csi-vsphere.conf

    3. (Optional) Check and confirm the vCenter IP address, username, and password values in the secret configuration file csi-vsphere.conf, which looks like this:

      cluster-id = "CLUSTER-ID"

[VirtualCenter "VCENTER-IP"]
user = "USERNAME"
password = "PASSWORD"
port = "443"

    4. Use the configuration file to create the velero-vsphere-config-secret secret in the namespace velero:

      kubectl -n velero create secret generic velero-vsphere-config-secret --from-file=csi-vsphere.conf

    5. Create a ConfigMap file velero-plugin.conf for the Velero plugin that references the secret:

      apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-vsphere-plugin-config
data:
  cluster_flavor: VANILLA
  vsphere_secret_name: velero-vsphere-config-secret
  vsphere_secret_namespace: velero

    6. Apply the ConfigMap:

      kubectl -n velero apply -f velero-plugin.conf

    7. Add the plugin:

      velero plugin add PLUGIN-IMAGE

      Where PLUGIN-IMAGE is the registry path to the container image listed in the Velero Plugin for vSphere repo v1.4.0, for example, http://projects.registry.vmware.com/tkg/velero/velero-plugin-for-vsphere:v1.4.0_vmware.1.

    8. Enable the plugin by adding the following VirtualMachine permissions to the role you created for the Tanzu Kubernetes Grid account, if you did not already include them when you created the account:

      • Configuration > Toggle disk change tracking
      • Provisioning > Allow read-only disk access
      • Provisioning > Allow virtual machine download
      • Snapshot management > Create snapshot
      • Snapshot management > Remove snapshot

Install Velero on Clusters on AWS

To install Velero on workload clusters on AWS, follow the Install and start Velero procedure in the Velero Plugins for AWS repository.

  1. If the workload cluster already has Velero installed from TKG v1.6.x or earlier, follow the steps in Update CRDs from Prior Velero Versions.

  2. Run velero install with the options listed in Velero Install Options above.

Install Velero on Clusters on Azure

To install Velero on workload clusters on Azure, follow the Install and start Velero procedure in the Velero Plugins for Azure repository.

  1. If the workload cluster already has Velero installed from TKG v1.6.x or earlier, follow the steps in Update CRDs from Prior Velero Versions.

  2. Run velero install with the options listed in Velero Install Options above.

Back Up a Workload Cluster

To back up the contents of a workload cluster:

  1. Follow the Deploy Velero Server to Clusters instructions for your infrastructure, above, to deploy a Velero server onto the cluster, along with the Velero Plugin for vSphere if needed.

  2. Back up the cluster contents:

    velero backup create your_backup_name

  3. If velero backup returns a transport is closing error, try again after increasing the memory limit, as described in Update resource requests and limits after install in the Velero documentation.

Note

Backup and restore of Windows workload clusters is not supported.

Restore a Workload Cluster

To restore a workload cluster’s contents from backup:

  1. Create a new cluster. You cannot restore a cluster backup to an existing cluster.

  2. Follow the Deploy Velero Server to Clusters instructions for your infrastructure, above, to deploy a Velero server onto the new cluster, along with the Velero Plugin for vSphere if needed.

  3. Restore the cluster contents:

    velero backup get
velero restore create your_restore_name --from-backup your_backup_name
check-circle-line exclamation-circle-line close-line
Scroll to top icon