This topic describes how to install Velero and Restic for backing up and restoring Tanzu Kubernetes Grid Integrated Edition (TKGI)-provisioned Kubernetes workloads. This topic also describes how to install MinIO for Velero with Restic backup storage.
Ensure the following before installing Velero and Restic for backing up and restoring TKGI:
Restic requires an object store as the backup destination for workload backups.
To deploy and configure a MinIO Server on a Linux Ubuntu VM as the Velero and Restic backend object store:
For more information about MinIO, see the MinIO Quick Start Guide.
To install MinIO:
Install the MinIO app:
wget https://dl.min.io/server/minio/release/linux-amd64/minio
Grant execute permissions to the MinIO app:
chmod +x minio
Create a directory where MinIO data will be stored:
mkdir /DATA-MINIO
To prepare the MinIO server:
Start the MinIO server:
./minio server /DATA-MINIO
After the MinIO server has started, you are provided with the data store instance endpoint URL, AccessKey, and SecretKey.
Record the MinIO server endpoint URL, AccessKey, and SecretKey information for the data store instance.
Browse to the MinIO data store by opening a browser to the MinIO server endpoint URL. For example: http://10.199.17.63:9000/minio/login/.
Log in to the MinIO server and provide the AccessKey and SecretKey.
To enable MinIO as a service, configure MinIO for automatic startup:
Download the minio.service
script:
curl -O https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service
Edit the minio.service
script and add the following value for ExecStart
:
ExecStart=/usr/local/bin/minio server /DATA-MINIO path
Save the revised script.
Configure the MinIO service by running the following commands:
cp minio.service /etc/systemd/system
cp minio /usr/local/bin/
systemctl daemon-reload
systemctl start minio
systemctl status minio
systemctl enable minio
To create a MinIO bucket for TKGI workload backup and restore:
Click the Create Bucket icon.
Enter the bucket name, for example: tkgi-velero
.
By default, a new MinIO bucket is read-only, but for our Velero backup and restore the MinIO bucket must be read-write.
To set the new tkgi-velero
bucket to read-write:
Read and Write
. To install the Velero CLI on your workstation:
To download the Velero CLI Binary:
Download the supported version of the signed Velero binary for your version of TKGI from the TKGI product downloads page at VMware Customer Connect. For more information about the currently supported Velero versions, see the Product Snapshot section of the Release Notes.
Note: You must use the Velero binary signed by VMware to be eligible for support from VMware.
To install the Velero CLI on the TKGI client or on your local machine:
Unzip the download file:
gunzip velero-linux-v1.6.3_vmware.1.gz
To check for the Velero binary:
ls -l
For example:
$ ls -l-rw-r–r– 1 root root 57819586 Sep 14 16:19 velero-linux-v1.6.3_vmware.1
Grant execute permissions to the Velero CLI:
chmod +x velero-linux-v1.6.3_vmware.1
Make the Velero CLI globally available by moving it to the system path:
cp velero-linux-v1.6.3_vmware.1 /usr/local/bin/velero
Verify the installation:
velero version
For example:
$ velero versionClient: Version: v1.6.3
To install the Velero and Restic pods on each Kubernetes cluster whose workloads you want to backup, complete the following:
The following steps require that:
The Velero CLI context will automatically follow the kubectl context. Before running Velero CLI commands to install Velero and Restic on the target cluster, set the kubectl context:
tkgi-velero
.0XXNO8JCCGV41QZBV0RQ
and SecretKey: clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
.kubectl
works against the cluster. If needed, use tkgi get-credentials
.Set the context for the target Kubernetes cluster so that the Velero CLI knows which cluster to work on by running:
tkgi get-credentials CLUSTER-NAME
Where CLUSTER-NAME
is the name of the cluster. For example:
$ tkgi get-credentials cluster-1Fetching credentials for cluster cluster-1. Password: ******** Context set for cluster cluster-1.
You can now switch between clusters by using: $kubectl config use-context <cluster-name>
You can also run kubectl config use-context CLUSTER-NAME
to set context.
To create a secrets file, create a file named credentials-minio
. Update the file with the MinIO server access credentials that you collected above:
[default]
aws_access_key_id = ACCESS-KEY
aws_secret_access_key = SECRET-KEY
Where:
ACCESS-KEY
is the AccessKey that you collected above.SECRET-KEY
is the SecretKey that you collected above.For example:
[default]
aws_access_key_id = 0XXNO8JCCGV41QZBV0RQ
aws_secret_access_key = clZ1bf8Ljkvkmq7fHucrKCkxV39BRbcycGeXQDfx
Save the file.
Verify that the file is in place:
ls
For example:
$ lscredentials-minio
To install Velero and Restic:
Run the following command to install Velero and Restic on the target Kubernetes cluster:
velero install \
--provider aws \
--plugins projects.registry.vmware.com/tkg/velero/velero-plugin-for-aws:v1.2.1_vmware.1 \
--bucket tkgi-velero \
--secret-file ./credentials-minio \
--use-volume-snapshots=false \
--use-restic \
--backup-location-config \
region=minio,s3ForcePathStyle="true",s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000
For example:
$ velero install –provider aws –plugins projects.registry.vmware.com/tkg/velero/velero-plugin-for-aws-v1.2.1_vmware.1
–bucket tkgi-velero –secret-file ./credentials-minio –use-volume-snapshots=false
–use-restic –backup-location-config
region=minio,s3ForcePathStyle=“true”,s3Url=http://10.199.17.63:9000,publicUrl=http://10.199.17.63:9000CustomResourceDefinition/backups.velero.io: created … Waiting for resources to be ready in cluster… … DaemonSet/restic: created Velero is installed! Use ‘kubectl logs deployment/velero -n velero’ to view the status.
Verify the installation of Velero and Restic:
kubectl logs deployment/velero -n velero
Verify the velero
namespace:
kubectl get ns
For example:
$ kubectl get nsNAME STATUS AGE default Active 13d kube-node-lease Active 13d kube-public Active 13d kube-system Active 13d pks-system Active 13d velero Active 2m38s
Verify the velero
and restic
pods.
kubectl get all -n velero
For example:
$ kubectl get all -n veleroNAME READY STATUS RESTARTS AGE pod/restic-25chx 0/1 CrashLoopBackOff 1 30s pod/restic-rpxcp 0/1 CrashLoopBackOff 1 30s pod/restic-wfxmg 0/1 CrashLoopBackOff 1 30s pod/velero-8dc7498d9-9v7x4 1/1 Running 0 30s
To run the three-pod Restic DaemonSet on a Kubernetes cluster in TKGI, you must modify the Restic DaemonSet spec and modify the hostpath.
To modify the Restic DaemonSet:
Verify the three-pod Restic DaemonSet:
kubectl get pod -n velero
For example:
$ kubectl get pod -n veleroNAME READY STATUS RESTARTS AGE pod/restic-p5bdz 0/1 CrashLoopBackOff 4 3m8s pod/restic-rbmnd 0/1 CrashLoopBackOff 4 3m8s pod/restic-vcpjm 0/1 CrashLoopBackOff 4 3m8s pod/velero-68f47744f5-lb5df 1/1 Running 0 3m8s
Run the following command:
kubectl edit daemonset restic -n velero
Change hostPath from /var/lib/kubelet/pods
to /var/vcap/data/kubelet/pods
:
- hostPath:
path: /var/vcap/data/kubelet/pods
Save the file.
To verify the three-pod Restic DaemonSet:
kubectl get pod -n velero
For example:
kubectl get pod -n veleroNAME READY STATUS RESTARTS AGE restic-5jln8 1/1 Running 0 73s restic-bpvtq 1/1 Running 0 73s restic-vg8j7 1/1 Running 0 73s velero-68f47744f5-lb5df 1/1 Running 0 10m
For information about the need to modify the Restic DaemonSet spec, see Restic Integration in the Velero documentation.
If your Velero backup returns status=InProgress
for many hours, increase the limits and requests memory settings.
To increase limits and requests memory settings:
Run the following command:
kubectl edit deployment/velero -n velero
Change the limits and request memory settings from the default of 256Mi
and 128Mi
to 512Mi
and 256Mi
:
ports:
- containerPort: 8085
name: metrics
protocol: TCP
resources:
limits:
cpu: "1"
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
If you are working in an air-gapped environment, you can install Velero and Restic using an internal registry. For more information, see Air-gapped deployments in the Velero documentation.
credentials-minio
file exists. For more information, see Set Up the kubectl Context above.Download the Velero CLI and Velero with Restic Docker images for your version of TKGI:
Note: You must use the container images signed by VMware to be eligible for support from VMware.
Push the Docker images into the internal registry. Adjust the variables as needed for your registry instance and preferences.
docker login harbor.example.com
docker load -i velero-plugin-for-aws-v1.2.1_vmware.1.tar.gz
docker tag vmware-tanzu/velero-plugin-for-aws-v1.2.1_vmware.1 harbor.example.com/vmware-tanzu/velero-plugin-for-aws-v1.2.1_vmware.1
docker load -i velero-restic-restore-helper-v1.6.3_vmware.1.tar.gz
docker tag vmware-tanzu/velero-restic-restore-helper-v1.6.3_vmware.1 harbor.example.com/vmware-tanzu/velero-restic-restore-helper-v1.6.3_vmware.1
docker load -i velero-linux-v1.6.3_vmware.1.gz
docker tag vmware-tanzu/velero:v1.6.3_vmware.1 harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.6.3_vmware.1
docker push harbor.example.com/harbor-ci/vmware-tanzu/velero-plugin-for-aws-v1.2.1_vmware.1
docker push harbor.example.vmware.com/vmware-tanzu/velero-restic-restore-helper-v1.6.3_vmware.1
docker push harbor.example.vmware.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.6.3_vmware.1
Install Velero with Restic:
velero install --image harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.6.3_vmware.1 \
--plugins harbor.example.com/vmware-tanzu/velero-plugin-for-aws-v1.2.1_vmware.1 \
--provider aws --bucket velero --secret-file ./credentials-minio \
--use-volume-snapshots=false \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://20.20.224.27:9000,publicUrl=http://20.20.224.27:9000 --use-restic
For example:
$ velero install –image harbor.example.com/vmware-tanzu/harbor.example.com/vmware-tanzu/velero:v1.6.3_vmware.1 –plugins harbor.example.com/vmware-tanzu/velero-plugin-for-aws-v1.2.1_vmware.1 –provider aws –bucket velero –secret-file ./credentials-minio –use-volume-snapshots=false –backup-location-config region=minio,s3ForcePathStyle=“true”,s3Url=http://20.20.224.27:9000,publicUrl=http://20.20.224.27:9000 –use-restic Velero is installed! Use ‘kubectl logs deployment/velero -n velero’ to view the status.
For more information about installing Velero and Restic, see On-Premises Environments and Restic Integration in the Velero documentation.
After installing, configure the Restic post-installation settings:
kubectl apply -f restic-cm.yaml -n velero
. Download the example configmap restic-cm.yaml provided for this purpose./var/lib/kubelet/pods
with /var/vcap/data/kubelet/pods
. Verify that the Restic pods are running. For more information, see Modify the Host Path above.'- --restic-timeout=900m'
to spec.template.spec.containers
.(Optional) Adjust your Restic Pod CPU and memory reserves: Depending on your requirements, you can adjust the CPU and memory reserves and limits for your Velero and Restic Pods. For more information, see Adjust Velero Memory Limits (if necessary) above.
restic pod
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi
velero pod
resources:
limits:
cpu: "1"
memory: 256Mi
requests:
cpu: 500m
memory: 128Mi