Install and Configure Standalone Velero and Restic on TKG Clusters

To backup and restore workloads running on TKG clusters on Supervisor, create a data store and install Velero with Restic on the Kubernetes cluster.

Overview

TKG clusters run on virtual machine nodes. To backup and restore TKG cluster workloads, you install Velero and Restic on the cluster.

Prerequisites

Ensure that your environment meets the following prerequisites for installing Velero and Restic to back up and restore workloads running on Tanzu Kubernetes clusters.

A Linux VM with sufficient storage to store several workload backups. You will install MinIO on this VM.
A Linux VM where the Kubernetes CLI Tools for vSphere are installed, including the vSphere Plugin for kubectl and kubectl. You will install the Velero CLI on this client VM. If you do not have such a VM, you can install the Velero CLI locally, but you must adjust the installation steps accordingly.
The Kubernetes environment has internet access and can be reached by the client VM.

Install and Configure MinIO Object Store

Velero requires an S3-compatible object store as the destination for Kubernetes workload backups. Velero supports several such object store providers. For simplicity, these instructions use MinIO, an S3-compatible storage service that runs locally on the object store VM.

Install MinIO.

wget https://dl.min.io/server/minio/release/linux-amd64/minio

Grant execute permissions to MinIO.
```
chmod +x minio
```
Create a directory on the file system for MinIO.
```
mkdir /DATA-MINIO
```
Start the MinIO server.
```
./minio server /DATA-MINIO
```

After the MinIO server starts, you are provided with important data store instance details, including the Endpoint URL, AccessKey, and SecretKey. Record the Endpoint URL, AccessKey, and SecretKey in the table.

Data Store Metadata	Value
Endpoint URL
AccessKey
SecretKey

Browse to the MinIO data store by opening a browser to the MinIO server endpoint URL.
Log in to the MinIO server and provide the AccessKey and SecretKey.
To enable MinIO as a service, configure MinIO for automatic start-up by download the minio.service script.
```
curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service
```
Edit the minio.service script and add the following value for ExecStart.
```
ExecStart=/usr/local/bin/minio server /DATA-MINIO path
```
Save the revised script.

Configure the MinIO service by running the following commands.

cp minio.service /etc/systemd/system 
cp minio /usr/local/bin/ 
systemctl daemon-reload 
systemctl start minio 
systemctl status minio 
systemctl enable minio

Create a MinIO bucket for backup and restore by launching the MinIO browser and logging in to your object store.
Click the Create Bucket icon.
Enter the bucket name, for example: my-cluster-backups.
Verify that the bucket was created.
By default, a new MinIO bucket is read-only. For Velero standalone backup and restore, the MinIO bucket must be read-write. To set the bucket to read-write, select the bucket and click on the ellipses (dots) link.
Select Edit Policy.
Change the policy to Read and Write.
Click Add.
To close the dialog box, click X.

Install the Velero CLI

Install the Velero CLI on the VM client or on your local machine.

The version used for this documentation is Velero 1.9.7 for Tanzu Kubernetes Grid 2.2.0.

Download Velero from the Tanzu Kubernetes Grid product download page at the VMware Customer Connect portal.
Note: You must use the Velero binary signed by VMware to be eligible for support from VMware.
Open a command line and change directory to the Velero CLI download.
Unzip the download file. For example:
```
gunzip velero-linux-vX.X.X_vmware.1.gz
```
Check for the Velero binary.
```
ls -l
```
Grant execute permissions to the Velero CLI.
```
chmod +x velero-linux-vX.X.X_vmware.1 
```
Make the Velero CLI globally available by moving it to the system path:
```
cp velero-linux-vX.X.X_vmware.1 /usr/local/bin/velero 
```
Verify the installation.
```
velero version
```

Install Velero and Restic on the Tanzu Kubernetes Cluster

The Velero CLI context will automatically follow the kubectl context. Before running Velero CLI commands to install Velero and Restic on the target cluster, set the kubectl context.

Retrieve the name of the MinIO bucket. For example, my-cluster-backups.
Get the AccessKey and SecretKey for the MinIO bucket.
Set the context for the target Kubernetes cluster so that the Velero CLI knows which cluster to work on.
```
kubectl config use-context tkgs-cluster-name
```
Create a secrets file named credentials-minio. Update the file with the MinIO server access credentials that you collected. For example:
```
aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ
aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
```
Note: If you receive an error message "Error getting a backup store" with the description "NoCredentialProviders: no valid providers in chain," prepend the line [default] to the beginning of the credentials file. For example:
```
[default]
aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ
aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
```
Save the file and verify that the file is in place.
```
ls
```

Run the following command to install Velero and Restic on the target Kubernetes cluster. Replace both URLs with the URL of your MinIO instance.

velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.0.0 \
--bucket tkgs-velero \
--secret-file ./credentials-minio \
--use-volume-snapshots=false \
--use-restic \
--backup-location-config \
region=minio,s3ForcePathStyle="true",s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000

Verify the installation of Velero and Restic.
```
kubectl logs deployment/velero -n velero
```
Verify the velero namespace.
```
kubectl get ns
```
Verify the velero and restic pods.
```
kubectl get all -n velero
```

Troubleshoot Restic DaemonSet (If Necessary)

To run the three-pod Restic DaemonSet on a Kubernetes cluster, you may have to update the Restic DaemonSet spec and modify the hostPath. For more information about this issue, see Restic Integration in the Velero documentation.

Verify the three-pod Restic DaemonSet.
```
kubectl get pod -n velero
```
If the pods are in a CrashLoopBackOff status, edit them as follows.
Run the edit command.
```
kubectl edit daemonset restic -n velero
```
Change hostPath from /var/lib/kubelet/pods to /var/vcap/data/kubelet/pods.
```
- hostPath:
    path: /var/vcap/data/kubelet/pods
```
Save the file.

Verify the three-pod Restic DaemonSet.

kubectl get pod -n velero

NAME                      READY   STATUS    RESTARTS   AGE
restic-5jln8              1/1     Running   0          73s
restic-bpvtq              1/1     Running   0          73s
restic-vg8j7              1/1     Running   0          73s
velero-72c84322d9-le7bd   1/1     Running   0          10m

Adjust Velero Memory Limits (If Necessary)

If your Velero backup returns status=InProgress for many hours, increase the limits and requests memory settings.

Run the following command.

kubectl edit deployment/velero -n velero

Change the limits and request memory settings from the default of 256Mi and 128Mi to 512Mi and 256Mi.

    
ports:
- containerPort: 8085
  name: metrics
  protocol: TCP
resources:
  limits:
    cpu: "1"
    memory: 512Mi
  requests:
    cpu: 500m
    memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File