This topic describes how to use Velero to back up and restore a StatefulSet application with namespace.
Overview
This example demonstrates Velero back up and restore for a StatefulSet application with namespace. The Cassandra database app is used for demonstrating back up and restore with Velero.
When restoring a stateful application using Velero, the storage class that was used by the PersistentVolumeClaim (PVC) in the application must be present on the Kubernetes cluster. If the PVC was using the default storage class, then the default storage class must also be present prior to initiating the restore operation with Velero.
Prerequisites
Install and configure MinIO, Velero, and restic.
The application we are going to use is the Cassandra database StatefulSet app. Download the Cassandra Database YAML files to a local known directory:
- cassandra-statefulset.yaml
- cassandra-storageclass.yaml
- headless-cassandra-service.yaml
Create the Storage Class
Modify the storage class cassandra-storageclass.yaml to use the generic Container Storage Interface (CSI) provisioner:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cassandra-sc-csi
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
datastoreurl: "ds:///vmfs/volumes/vsan:52d8eb4842dbf493-41523be9cd4ff7b7/"
Apply the storage class YAML file:
kubectl apply -f 0-cassandra-storageclass.yaml
storageclass.storage.k8s.io/cassandra-sc-csi created
Verify the storage class:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cassandra-sc-csi (default) csi.vsphere.vmware.com Delete Immediate false 98m
Deploy Cassandra Database App
Create the namespace:
kubectl create ns cassandra-csi
namespace/cassandra-csi created
Create the service:
kubectl apply -f 1-headless-cassandra-service.yaml -n cassandra
service/cassandra created
Deploy the Cassandra database app:
kubectl apply -f 2-cassandra-statefulset.yaml -n cassandra
statefulset.apps/cassandra created
Verify:
kubectl get all -n cassandra-csi
NAME READY STATUS RESTARTS AGE
pod/cassandra-0 1/1 Running 0 101m
pod/cassandra-1 1/1 Running 0 100m
pod/cassandra-2 1/1 Running 0 99m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cassandra ClusterIP None <none> <none> 101m
NAME READY AGE
statefulset.apps/cassandra 3/3 101m
kubectl get pvc,pv -n cassandra-cs
Make sure Cassandra database instances are fully participating in the cluster:
kubectl exec -it cassandra-0 -n cassandra-csi -- nodetool status
Datacenter: Demo-DataCenter
===========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.1.2 134.43 KiB 32 55.6% f4ae812c-fd83-4268-a225-890df1ca84d6 Demo-Rack
UN 172.16.1.3 132.68 KiB 32 73.0% deb9625e-89f5-45d0-9d0b-c8ee3dcb610a Demo-Rack
UN 172.16.1.4 109.58 KiB 32 71.4% 4599bc2e-6f7b-4093-9397-53184a83a92c Demo-Rack
Create and Populate a Database and Table in Cassandra
Create the DB:
kubectl exec -it cassandra-0 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
Create and populate the table:
cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
cqlsh> use demodb;
cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, 'Tom', 'Cork', 999, 1000000);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (101, 'Andrew', 'NY', 1000, 1000000);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (102, 'Lara', 'Paris', 1001, 1000000);
Verify:
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
Verify that the other Cassandra DB instances have the same information:
kubectl exec -it cassandra-1 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
kubectl exec -it cassandra-2 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
Add Annotations
Add annotations for the stateful pods with the volume name cassandra-data:
kubectl get pod -n cassandra-csi
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 105m
cassandra-1 1/1 Running 0 105m
cassandra-2 1/1 Running 0 103m
The pods cassandra-0, cassandra-1 and cassandra-2 must be annotated with cassandra-data.
kubectl -n cassandra-csi annotate pod/cassandra-0 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-0 annotated
kubectl -n cassandra-csi annotate pod/cassandra-1 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-1 annotated
kubectl -n cassandra-csi annotate pod/cassandra-2 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-2 annotated
Verify the annotations:
kubectl -n cassandra-csi describe pod/cassandra-0 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra-csi describe pod/cassandra-1 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra-csi describe pod/cassandra-2 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
Back Up the Cassandra Database App Using Namespace
Perform the Velero back up:
velero backup create cassandra-csi-backup --include-namespaces cassandra-csi
Backup request "cassandra-csi-backup" submitted successfully.
Run `velero backup describe cassandra-csi-backup` or `velero backup logs cassandra-csi-backup` for more details.
Verify the backup:
velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
cassandra-csi-backup Completed 0 0 2020-07-30 12:02:43 -0700 PDT 29d default <none>
Get backup details:
velero backup describe cassandra-csi-backup --details
Use the Velero Kubernetes CustomResourceDefinition (CRD) commands to verify:
kubectl get crd
kubectl get backups.velero.io -n velero
kubectl describe backups.velero.io cassandra-backup -n velero
Restore the Cassandra Database App
Restore the Cassandra database app from the Velero backup. Note the following about the restore operation:
- Pod annotation is still required for Velero backup of PV.
- The namespace
cassandrawas automatically re-created.
Delete the namespace:
kubectl delete ns cassandra-csi
namespace "cassandra-csi" deleted
Verify that Cassandra resources are deleted:
kubectl get ns
NAME STATUS AGE
default Active 22d
kube-node-lease Active 22d
kube-public Active 22d
kube-system Active 22d
pks-system Active 22d
velero Active 8d
kubectl get pvc,pv --all-namespaces
No resources found
Verify that the storage class used by the application is still present:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cassandra-sc-csi (default) csi.vsphere.vmware.com Delete Immediate false 175m
Restore the Cassandra database app from the backup:
velero restore create --from-backup cassandra-csi-backup
Restore request "cassandra-csi-backup-20200730130959" submitted successfully.
Run `velero restore describe cassandra-csi-backup-20200730130959` or `velero restore logs cassandra-csi-backup-20200730130959` for more details.
Verify that the app is restored:
velero restore get
NAME BACKUP STATUS ERRORS WARNINGS CREATED SELECTOR
cassandra-csi-backup-20200730130959 cassandra-csi-backup Completed 0 0 2020-07-30 13:09:59 -0700 PDT <none>
Check restoration details:
velero restore describe cassandra-csi-backup-20200730130959
Verify the namespace:
kubectl get ns
NAME STATUS AGE
cassandra-csi Active 75s
default Active 22d
kube-node-lease Active 22d
kube-public Active 22d
kube-system Active 22d
pks-system Active 22d
velero Active 8d
Verify the pods:
kubectl get all -n cassandra-csi
NAME READY STATUS RESTARTS AGE
pod/cassandra-0 1/1 Running 0 105s
pod/cassandra-1 0/1 Running 3 105s
pod/cassandra-2 0/1 Running 3 105s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cassandra ClusterIP None <none> <none> 105s
NAME READY AGE
statefulset.apps/cassandra 1/3 105s
Verify the persistent volume:
kubectl get pvc,pv -n cassandra-csi
Check the content of each Cassandra DB instance:
kubectl exec -it cassandra-0 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
kubectl exec -it cassandra-1 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
kubectl exec -it cassandra-2 -n cassandra-csi -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
Conclusions
Key takeaways from the restore operation:
- Pod annotation is still required for Velero to back up a PV.
- The namespace
cassandra-csiwas automatically re-created. - Restic works fine with CSI.
