This topic describes how to use Velero to backup and restore a statefulset application with label.
This example demonstrates Velero backup and restore for a statefulset application with label selector. The CassandraDB app is used for demonstrating stateful backup and restore with Velero.
When restoring a stateful application using Velero, the Storage Class that was used by the PVC in the application must be present on the Kubernetes cluster. If the PVC was using the default storage class, then the default storage class must also be present prior to initiating the restore operation with Velero.
Install and configure Minio, Velero, and Restic.
The application we are going to use is the CassandraDB statefulset app. Download the CassandraDB YAML files to a local known directory:
Modify the storage class cassandra-storageclass.yaml:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: demo-sts-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: thin
Apply the storage class YAML file:
kubectl apply -f cassandra-storageclass.yaml
storageclass.storage.k8s.io/demo-sts-sc created
Verify the storage class:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
demo-sts-sc kubernetes.io/vsphere-volume Delete Immediate false 3s
By default all the objects related to CassandraDB app have the same and common label: app=cassandra
. Unlike the Guestbook app, you do not need to create a new label that is common to all Kubernetes objects for this app.
Create the namespace:
kubectl create ns cassandra
namespace/cassandra created
Create the service:
kubectl apply -f headless-cassandra-service.yaml -n cassandra
service/cassandra created
Deploy CassandraDB app:
kubectl apply -f cassandra-statefulset.yaml -n cassandra
statefulset.apps/cassandra created
Verify the CassandraDB app:
kubectl get all -n cassandra --show-labels
NAME READY STATUS RESTARTS AGE LABELS
pod/cassandra-0 1/1 Running 0 2m24s app=cassandra,controller-revision-hash=cassandra-559df6f5c,statefulset.kubernetes.io/pod-name=cassandra-0
pod/cassandra-1 1/1 Running 0 103s app=cassandra,controller-revision-hash=cassandra-559df6f5c,statefulset.kubernetes.io/pod-name=cassandra-1
pod/cassandra-2 0/1 Running 0 29s app=cassandra,controller-revision-hash=cassandra-559df6f5c,statefulset.kubernetes.io/pod-name=cassandra-2
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS
service/cassandra ClusterIP None <none> <none> 2m28s app=cassandra
NAME READY AGE LABELS
statefulset.apps/cassandra 2/3 2m24s app=cassandra
Verify PVC and PV:
kubectl get pvc,pv -n cassandra
Make sure CassandraDB instances are fully participating in the cluster:
kubectl exec -it cassandra-0 -n cassandra -- nodetool status
Datacenter: Demo-DataCenter
===========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.1.2 104.41 KiB 32 68.0% 06df61ac-135c-406c-a1f1-d9937de38b9d Demo-Rack
UN 172.16.1.3 94.96 KiB 32 75.7% a49a5c78-85be-411f-a0b9-e28ff35fd918 Demo-Rack
UN 172.16.1.4 81.1 KiB 32 56.4% 85d124ee-0217-49e8-b56c-e13736038f02 Demo-Rack
Create the DB:
kubectl exec -it cassandra-0 -n cassandra -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
Create and populate table:
cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { ‘class’ : ‘SimpleStrategy’, ‘replication_factor’ : 3 }; cqlsh> use demodb; cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, ‘Tom’, ‘Cork’, 999, 1000000); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (101, ‘Andrew’, ‘NY’, 1000, 1000000); cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (102, ‘Lara’, ‘Paris’, 1001, 1000000); cqlsh:demodb> select * from emp;
Verify:
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
(3 rows)
Verify that the other Cassandra DB instances have the same information:
kubectl exec -it cassandra-1 -n cassandra – cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
(3 rows)
kubectl exec -it cassandra-2 -n cassandra – cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
(3 rows)
## <a id='cassandra-annotations'></a> Add Annotations
Add annotations for the stateful pods with the volume name `cassandra-data`.
kubectl get pod -n cassandra
NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 19m cassandra-1 1/1 Running 0 19m cassandra-2 1/1 Running 0 17m
The pods `cassandra-0`, `cassandra-1` and `cassandra-2` must be annotated with `cassandra-data`.
kubectl -n cassandra annotate pod/cassandra-0 backup.velero.io/backup-volumes=cassandra-data pod/cassandra-0 annotated
kubectl -n cassandra annotate pod/cassandra-1 backup.velero.io/backup-volumes=cassandra-data pod/cassandra-1 annotated
kubectl -n cassandra annotate pod/cassandra-2 backup.velero.io/backup-volumes=cassandra-data pod/cassandra-2 annotated
Verify the annotations:
kubectl -n cassandra describe pod/cassandra-0 | grep Annotations Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra describe pod/cassandra-1 | grep Annotations Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra describe pod/cassandra-2 | grep Annotations Annotations: backup.velero.io/backup-volumes: cassandra-data
## <a id='cassandra-backup'></a> Backup the CassandraDB App Using Label
Perform the Velero backup:
velero backup create cassandra-label-backup –selector app=cassandra Backup request “cassandra-label-backup” submitted successfully.
Run velero backup describe cassandra-label-backup
or velero backup logs cassandra-label-backup
for more details.
Verify the backup:
velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR cassandra-label-backup Completed 0 0 2020-07-29 09:48:49 -0700 PDT 29d default app=cassandra
View backup details:
velero backup describe cassandra-label-backup –details
Name: cassandra-label-backup Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/source-cluster-k8s-gitversion=v1.17.8+vmware.1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=17
Phase: Completed
…
Velero-Native Snapshots:
Restic Backups: Completed: cassandra/cassandra-0: cassandra-data cassandra/cassandra-1: cassandra-data cassandra/cassandra-2: cassandra-data
Use Velero CRD commands to further verify the backup:
kubectl get crd
kubectl get backups.velero.io -n velero
NAME AGE cassandra-label-backup 96s
kubectl describe backups.velero.io cassandra-label-backup -n velero
## <a id='cassandra-restore'></a> Restore the CassandraDB App
Restore the CassandraDB app from the Velero backup. Note the following about the restore operation:
- Pod annotation is required to restore a Velero backup of a persistent volume (PV)
- The namespace 'cassandra' is automatically recreated during the restoration
Delete the namespace:
kubectl delete ns cassandra namespace “cassandra” deleted
Confirm that Cassandra resources are deleted:
kubectl get ns
NAME STATUS AGE default Active 21d kube-node-lease Active 21d kube-public Active 21d kube-system Active 21d pks-system Active 21d velero Active 7d18h
kubectl get pvc,pv –all-namespaces
No resources found
Verify that the storage class used by the application is still present:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE demo-sts-sc kubernetes.io/vsphere-volume Delete Immediate false 36m
Restore the CassandraDB app from the Velero backup:
velero restore create –from-backup cassandra-label-backup
Restore request “cassandra-label-backup-20200729095415” submitted successfully. Run velero restore describe cassandra-label-backup-20200729095415
or velero restore logs cassandra-label-backup-20200729095415
for more details.
Verify that the app is restored:
velero restore get
NAME BACKUP STATUS ERRORS WARNINGS CREATED SELECTOR cassandra-label-backup-20200729095415 cassandra-label-backup Completed 0 0 2020-07-29 09:54:15 -0700 PDT
velero restore describe cassandra-label-backup-20200729095415
Name: cassandra-label-backup-20200729095415 Namespace: velero Labels:
Phase: Completed
Backup: cassandra-label-backup
Namespaces: Included: all namespaces found in the backup Excluded:
Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io Cluster-scoped: auto
Namespace mappings:
Label selector:
Restore PVs: auto
Restic Restores (specify –details for more information): Completed: 3
kubectl get ns
NAME STATUS AGE cassandra Active 79s default Active 21d kube-node-lease Active 21d kube-public Active 21d kube-system Active 21d pks-system Active 21d velero Active 7d19h
kubectl get all -n cassandra –show-labels
Verify the persistent volume:
kubectl get pvc,pv -n cassandra –show-labels
Check the content of each Cassandra DB instance:
kubectl exec -it cassandra-0 -n cassandra – cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
kubectl exec -it cassandra-1 -n cassandra – cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
(3 rows)
kubectl exec -it cassandra-2 -n cassandra – cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal ——–+———-+———-+———–+——— 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000
(3 rows)