You can use the Velero Plugin for vSphere to backup and restore workloads running on vSphere Pods by installing the Velero Plugin for vSphere on the Supervisor.

Overview

The Velero Plugin for vSphere provides a solution for backing up and restoring vSphere with Tanzu workloads. The solution requires the installation and configuration of several components. Once you have the Velero Plugin for vSphere installed and configured on Supervisor, you can backup and restore TKG cluster workloads. For persistent workloads, the Velero Plugin for vSphere lets you take snapshots of the persistent volumes.

Prerequisites: Required

Before installing the Velero Plugin for vSphere on Supervisor, adhere to the following prerequisites:
  • Supervisor is enabled.
  • vSphere Namespace is created and configured.
  • You must be a member of the vSphere Administrator role, or have following vSphere privileges:
    • SupervisorServices.Manage
    • Namespaces.Manage
    • Namespaces.Configure
  • Create a Linux VM where you can run the Velero CLI. Or use an existing Linux jump host where you access Supervisor.
  • The Velero version numbers are presented as X.Y.Z. Refer to the Velero Compatibility Matrix for the specific versions to use, and substitute accordingly when running the commands.
The screenshot shows the end state of a Velero Plugin for vSphere installation.
  • NSX networking is used to support the deployment of vSphere Pods
  • A Data Manager VM is deployed
  • The Velero Operator is enabled and running in the velero-vsphere-domain-cXX namespace
  • A namespace called velero is configured
  • The Velero Plugin for vSphere is running as a vSphere Pod in the velero namespace

The screenshot shows Velero Plugin in the vSphere Client.

Step 0 (Optional): Create a Dedicated Network for Backup and Restore Traffic

Although not required, it is recommended that for production environments you separate the backup and restore traffic from the vSphere with Tanzu management network traffic. There are two aspects to this:
  • Tag the ESXi hosts to support network file copy (NFC)
  • Configure the backup and restore network using NSX-T Data Center

To configure vSphere ESXi hosts to support a dedicated network block device (NBD) transport, you add a VMkernel NIC on each ESXi host in the vCenter Server cluster where Workload Management is enabled and set the vSphereBackupNFC on that NIC. When the tag vSphereBackupNFC is applied to the NIC type for a VMkernel adapter, backup and restore traffic goes through the chosen virtual NIC.

To perform this configuration, you use the Virtual Disk Development Kit. See the NBD documentation.
Note: If the vSphereBackupNFC is not enabled on the VMkernel NIC, the backup and restore traffic will be not be sent on the backup and restore network, even if you configure one. If vSphereBackupNFC is not enabled, the traffic will travel on the vSphere management network.
Once the vSphereBackupNFC tag is enabled, configure the backup and restore network using NSX-T by updating the existing vSphere Distributed Switch (vDS) for the cluster as follows:
  • In the vSphere Client, select Menu > Networking.
  • Select the existing vDS for the cluster.
  • Right-click the vDS and select Distributed Port Group > New Distributed Port Group.
  • Create a new Distributed Port Group named BackupRestoreNetwork.
  • Add a VMkernel adapter to the BackupRestoreNetwork Distributed Port Group.
  • Attach all the ESXi hosts in the vCenter cluster where Workload Management is enabled to the BackupRestoreNetwork Distributed Port Group.
  • Enable the vSphereBackupNFC tag.

Step 1: Create an S3-Compatible Object Store

For backup and restore of persistent volumes, you need to provide an S3-compatible object store. Velero supports a number of object store providers.

To install the Velero Plugin for vSphere, you will need to provide the following information about your S3-compatible object store:

Data Item Example Value
s3Url http://my-s3-store.example.com
aws_access_key_id ACCESS-KEY-ID-STRING
aws_secret_access_key SECRET-ACCESS-KEY-STRING
Create a secrets file name s3-credentials with the following information. You will reference this file when you install the Velero Plugin for vSphere.
[default]
aws_access_key_id = ACCESS-KEY-ID-STRING
aws_secret_access_key = SECRET-ACCESS-KEY-STRING

MinIO is an S3-compatible object store that is easy to install and use. vSphere with Tanzu ships with a MinIO Supervisor Service that you can enable. For more information, refer to the vSphere with Tanzu Services and Workloads publication.

Alternatively, you can manually install a MinIO server on a Linux VM. For instructions, see Install and Configure Standalone Velero and Restic on TKG Clusters on Supervisor.

Step 2: Install and Configure Data Manager

Warning: The Data Manager is only functionally tested and it is not meant to work at scale and does not promise any performance expectations. It is not meant to backup business critical applications in production.

To facilitate backup and restore using the Velero Plugin for vSphere, you deploy one or more Data Manager VMs to move persistent volume backup data into and out of S3-compatible object storage. The Data Manager moves the volume snapshot data from the vSphere volume to the remote durable S3-compatible storage on backup, and from remote S3-compatible storage to a vSphere volume during restore.

In a vSphere with Tanzu environment, install the Data Manager as a virtual machine.
Note: Do not power on the Data Manager VM until after you have enabled the Velero vSphere Operator.
  1. Using the vSphere Client, right-click the Datacenter where Workload Management is enabled and select Deploy OVF Template.
  2. Enter the path to the Data Manager OVA file and upload it to the vCenter Server: https://vsphere-velero-datamgr.s3-us-west-1.amazonaws.com/datamgr-ob-17253392-photon-3-release-1.1.ova.
  3. Name the virtual machine DataManager, for example.
  4. Select the compute resource which is the vCenter cluster where Supervisor is configured.
  5. Review the VM deployment details and click Next.
  6. Accept the license agreements and click Next.
  7. Select the storage and click Next.
  8. Select the destination network for the Data Manager VM.
    • Select the BackupRestoreNetwork if you configured a dedicated backup and restore network.
    • Select the Management Network if you did not configure a dedicated backup and restore network.
  9. Confirm the selections and click Finish to complete the process.
  10. Use the Recent Tasks panel to monitor the progress of the deployment.
    Note: If you receive an error that the "OVF descriptor is not available," use Chrome browser.
  11. Once the Data Manager VM is deployed, configure the input parameters for the VM.
  12. Right-click the VM and select Edit Settings.
  13. In the Virtual Hardware tab, for CD/DVD Drive, change from Host Device to Client Device.
    Note: If you do not do this, you cannot save the required advanced configuration settings.
  14. In the Edit Settings > Advanced Parameters tab, select Advanced > Edit Configuration Parameters.
  15. Configure the input parameters for each of the following settings:
    Parameter Value
    guestinfo.cnsdp.vcUser

    Enter the vCenter Server user name with sufficient privileges to deploy VMs.

    If you do not specify a user with vSphere Administrator permissions, refer to the vSphere Permissions documentation for guidance. Or, create a dedicated user for Workload Management: Create a Dedicated Group and Role for vSphere with Tanzu Operators.

    guestinfo.cnsdp.vcAddress Enter the vCenter Server IP address or FQDN.
    guestinfo.cnsdp.vcPasswd Enter the vCenter Server user password.
    guestinfo.cnsdp.vcPort The default is 443. Do not change this value.
    guestinfo.cnsdp.wcpControlPlaneIP Enter the Supervisor Cluster IP address.

    Get this value by navigating to the vCenter cluster where Workload Management is enabled and selecting Configure > Namespaces > Network > Management Network > Starting IP Address

    guestinfo.cnsdp.updateKubectl The default is false. Do not change this value.
    guestinfo.cnsdp.veleroNamespace The default is velero and you should leave it as such unless you have a compelling reason to change it. Later in the process you will create a vSphere Namespace on the Supervisor Cluster with the name velero. These names must match.
    guestinfo.cnsdp.datamgrImage If not configured (unset), the system by default will pull the container image from Docker Hub at vsphereveleroplugin/data-manager-for-plugin:1.1.0
  16. Click OK to save the configuration and OK again to save the VM settings.
    Note: If you did not change the CD/DVD Drive from Host Device to Client Device, you cannot save the settings. If this is the case, cancel the operation, change the drive and repeat the advanced configuration settings.
  17. Do not power on the Data Manager VM until after you enable the Velero vSphere Operator (next section).

Step 3: Install the Velero vSphere Operator Service on Supervisor

vSphere with Tanzu provides the Velero vSphere Operator as a vSphere Service. The Velero vSphere Operator service works with the Velero Plugin for vSphere to support backup and restore of Kubernetes workloads, including snapshotting persistent volumes. For more information about Supervisor Services, refer to the publication vSphere with Tanzu Services and Workloads.

Complete the following operation to register the Velero vSphere Operator specification with the vCenter Server where Workload Management is enabled, and to install the Velero vSphere Operator as a service on Supervisor.

  1. Download the YAML for the Velero vSphere Operator from the following location:

    http://vmware.com/go/supervisor-service

    The service specification file is named velero-supervisorservice-x.x.x.yaml.

  2. From the vSphere Client home menu, select Workload Management.
  3. Select the Services tab.
  4. Select the target vCenter Server from the drop-down menu at the top.
  5. Drag and drop the service specification file velero-supervisorservice-x.x.x.yaml that you downloaded to the Add New Service card.

    Alternatively you can click Add and browse to and select the file velero-supervisorservice-x.x.x.yaml.

  6. Click Next and accept the licence agreement.
  7. Click Finish.

    The Velero vSphere Operator is registered with vCenter Server. Verify that the service is in an Active state. You cannot install the service if it is Deactivated.

  8. Locate the Velero vSphere Operator specification in the Services tab.
  9. Click Actions > Install on Supervisors.The screenshot shows the Actions menu options on the Services tab.
  10. Select the target Supervisor where you want to install the service.
    Note: If you do not see Supervisor, verify that you are using NSX networking.
  11. Configure the Velero vSphere Operator service installation as follows:
    1. Select the version from the drop-down: X.Y.Z.
    2. Do not specify a Repository endpoint.
    3. Do not enter a username or password.
    4. Click Next.
  12. Click Finish to complete the installation of the service.

Verify the Velero vSphere Operator service on the Supervisor and start the Data Manager VM.

  1. From the vSphere Client home menu, select Workload Management.
  2. Select Services.
  3. Verify that you see the Velero vSphere Operator installed and its status is Configured.
  4. In the Namespaces resource pool, verify that you see a new namespace named svc-velero-vsphere-domain-xxx, where xxx is a unique alphanumeric token. This is the namespace created by the system for the Velero vSphere Operator.
    Note: You do not need to configure this namespace and you should not edit it.
  5. In the Hosts and Clusters view, select the Data Manager VM.
  6. Right-click the Data Manager VM and power it on.

Step 4: Create a vSphere Namespace for the Velero Plugin for vSphere

Using the vSphere Client, manually create a vSphere Namespace on Supervisor. This namespace is required for the Velero Plugin for vSphere.
  • Name the namespace velero.
  • Select the velero namespace and configure it.
  • Specify the Storage for the velero namespace.
  • Grant a user with appropriate privileges the Edit permission on the velero namespace.

Step 5: Create the Velero Plugin Configmap

Create a configmap for the Velero plugin named velero-vsphere-plugin-config.yaml.
apiVersion: v1
kind: ConfigMap
metadata:
  name: velero-vsphere-plugin-config
data:
  cluster_flavor: SUPERVISOR
Apply the configmap on Supervisor.
kubectl apply -n <velero-namespace> -f velero-vsphere-plugin-config.yaml
If you do not install the configmap, you receive the following error when you try to install the Velero Plugin for vSphere.
Error received while retrieving cluster flavor from config, err: configmaps "velero-vsphere-plugin-config" not found
Falling back to retrieving cluster flavor from vSphere CSI Driver Deployment

Step 6: Install the Velero Plugin for vSphere

Now you are ready to install the Velero Plugin for vSphere. To do this, download and run the velero-vsphere CLI.
Note: This procedure requires a Linux VM. Download the velero-vsphere binary to the Linux jump host where you run the kubectl-vsphere and kubectl CLIs.
  1. Download the Velero Plugin for vSphere CLI.
    Check the compatibility matrix and download the target version from here: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/releases.
    Note: In the commands that follow, replace X.Y.Z with the versions of Velero CLI and Plugin that you download.
  2. Securely copy the CLI to the Linux jump host. For example:
    pscp -P 22 C:\temp\velero-vsphere-X.Y.Z-linux-amd64.tar.gz ubuntu@10.117.29.131:/home/ubuntu/tanzu
  3. Extract the velero-vsphere CLI and make it writable.
    tar -xf velero-vsphere-X.Y.Z-linux-amd64.tar.gz
    chmod +x velero-vsphere
  4. Add the CLI to your Path.
    export PATH="$(pwd)/velero-vsphere-X.Y.Z-linux-amd64:$PATH"
  5. Create the s3-credentials file with the following contents.
    aws_access_key_id = ACCESS-KEY-ID-STRING
    aws_secret_access_key = SECRET-ACCESS-KEY-STRING
  6. Obtain the region, URL, and bucket name for your S3-compatible object store.
  7. Log in to the Supervisor using the vSphere Plugin for kubectl.
  8. Switch context to the Supervisor.
    kubectl config use-context SUPERVISOR-CLUSTER-IP-ADDRESS
  9. Run the following velero-vsphere CLI command to install the Velero Plugin for vSphere into the velero namespace you created.
    Export the values for the AWS $BUCKET and $REGION. If you deviated from any of the preceding instructions, adjust those values as well, such as the name or location of the secrets file, the name of the manually created velero namespace, etc.
    export BUCKET=example-velero-sv && export REGION=us-east-1
     
    ./velero-vsphere install \
           --namespace velero \
           --version vX.X.X \
           --provider aws \
           --plugins harbor-repo.vmware.com/velero/velero-plugin-for-aws:vX.Y.Z,harbor-repo.vmware.com/velero/velero-plugin-for-vsphere:vX.Y.Z \
           --bucket $BUCKET \
           --secret-file ~/.aws/credentials \
           --snapshot-location-config region=$REGION \
           --backup-location-config region=$REGION
    Note: For example, the Velero CLI version is v1.8.1 if the Velero Plugin for vSphere v1.4.0 is used.
  10. Verify successful installation of the Velero Plugin for vSphere.
    On successful installation you should see the following message:
    Send the request to the operator about installing Velero in namespace velero
    Run the following command to further verify. You should see "Completed" and the version.
    kubectl -n velero get veleroservice default -o json | jq '.status'
    Expected result:
    {
      "enabled": true,
      "installphase": "Completed",
      "version": "v1.8.1"
    }
    Note: The above command assumes you have the jq utility installed, which formats JSON output sent to the terminal. If you do not have jq installed, either install it or remove that part of the command (everything after json).
  11. Troubleshoot as necessary.

    If the installation is not successful, remove the installation and try again. To remove the installation, complete the steps in the next section in the order listed.

Addendum: Uninstall the Velero Plugin for vSphere

Complete these steps to uninstall the Velero Plugin for vSphere. Complete these steps in the order listed.
  1. Run the velero-vsphere CLI to uninstall the Velero Plugin for vSphere.
    ./velero-vsphere uninstall -n velero
  2. Verify that the vSphere Pod named velero is removed.
    kubectl get pods -n velero

    If you see that the pod is "Terminating," wait for it to be removed before proceeding.

  3. Using the vSphere Client, delete the vSphere Namespace named velero that you manually created.
    Note: Do not proceed with the next step until the namespace deletion is complete. You can use kubectl to verify that the velero namespace is removed (but do not use kubectl to remove the velero namespace).
  4. Using the vSphere Client, uninstall the Velero vSphere Operator from the Supervisor.
    1. Select the vCenter cluster where Workload Management is enabled.
    2. Select Configure > vSphere Services > Overview.
    3. Select the Velero vSphere Operator.
    4. Click Uninstall.

      This action uninstalls the Velero vSphere Operator from the Supervisor. The operator remains available for re-installation at the Workload Management > Services page. To remove the service entirely, select Actions > Delete.

Addendum: Install the Velero Plugin in an Air-gapped Environment

If you plan to install the Velero Plugin for vSphere in an air-gapped environment, you must install it with customized images. You must make sure that the matching images of backup-driver and data-manager-for-plugin of the customized images are available in the expected registry and are accessible from the Kubernetes cluster. In an air-gapped environment, customized images from private registry are expected since the released images in docker hub are not accessible.

To install the plugin, perform the following steps:
  1. Download the released images of velero-plugin-for-vsphere, backup-driver, and data-manager-for-plugin.
  2. Rename the images, that is, tag them with matching <Registry endpoint and path> and <Version tag> and upload them in customized repositories.
  3. Install the plugin using the velero-plugin-for-vsphere image you customized.

    When you install Velero Plugin for vSphere in a vanilla cluster, it deploys two additional components, a backup-driver deployment and a data-manager-for-plugin DaemonSet in the background. In the Supervisor and Tanzu Kubernetes clusters, only a backup-driver deployment is deployed.

    When you provide the container image of velero-plugin-for-vsphere, the matching backup-driver and data-manager-for-plugin images are parsed using an image parsing mechanism.

    Container images are formalized into the following pattern:
    <Registry endpoint and path>/<Container name>:<Version tag>

    When you provide the velero-plugin-for-vsphere container image, the corresponding images of backup-driver and data-manager-for-plugin with matching <Registry endpoint and path> and <Version tag> are parsed.

    For example, consider the following velero-plugin-for-vsphere container image:
    abc.io:8989/x/y/.../z/velero-plugin-for-vsphere:vX.Y.Z
    The following matching images of backup-driver and data-manager-for-plugin are expected to be pulled:
    abc.io:8989/x/y/.../z/backup-driver:vX.Y.Z
    abc.io:8989/x/y/.../z/data-manager-for-plugin:vX.Y.Z
  4. Troubleshoot the installation.
    If there are any issues or errors in parsing the matching images of backup-driver and data-manager-for-plugin, the installation falls back to corresponding images from the official velerovsphereplugin repositories in Docker hub. The following issues trigger the fall-back mechanism:
    1. An unexpected container name is used in the customized velero-plugin-for-vsphere image in the user input.

      For example, x/y/velero-velero-plugin-for-vsphere:vX.Y.Z is used.

    2. The Velero deployment name is customized to anything other than velero. For example, an issue is triggered if the Velero deployment name is updated to velero-server in the Velero manifests file before deploying Velero.

      The existing image parsing mechanism in velero-plugin-for-vsphere can only recognize the Velero deployment with the fixed name, velero.