This topic describes how to use Velero to back up and restore a StatefulSet application with namespace.
Overview
This example demonstrates Velero back up and restore for a StatefulSet application with namespace. The Cassandra database app is used for demonstrating back up and restore with Velero.
When restoring a stateful application using Velero, the Storage Class that was used by the PVC in the application must be present on the Kubernetes cluster. If the PVC was using the default storage class, then the default storage class must also be present prior to initiating the restore operation with Velero.
Prerequisites
Install and configure MinIO, Velero, and restic.
The application we are going to use is the Cassandra database StatefulSet app. Download the Cassandra Database YAML files to a local known directory:
- cassandra-statefulset.yaml
- cassandra-storageclass.yaml
- headless-cassandra-service.yaml
Create the Storage Class
Modify the storage class cassandra-storageclass.yaml:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: demo-sts-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
diskformat: thin
Apply the storage class YAML file:
kubectl apply -f cassandra-storageclass.yaml
storageclass.storage.k8s.io/demo-sts-sc created
Verify the storage class:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
demo-sts-sc kubernetes.io/vsphere-volume Delete Immediate false 3s
Deploy Cassandra Database App
Create the namespace:
kubectl create ns cassandra
namespace/cassandra created
Create the service:
kubectl apply -f headless-cassandra-service.yaml -n cassandra
service/cassandra created
Deploy Cassandra database app:
kubectl apply -f cassandra-statefulset.yaml -n cassandra
statefulset.apps/cassandra created
Verify:
kubectl get all -n cassandra
NAME READY STATUS RESTARTS AGE
pod/cassandra-0 1/1 Running 0 5m16s
pod/cassandra-1 1/1 Running 0 4m25s
pod/cassandra-2 1/1 Running 0 2m52s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cassandra ClusterIP None <none> <none> 5m21s
NAME READY AGE
statefulset.apps/cassandra 3/3 5m16s
kubectl get pvc,pv -n cassandra
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/cassandra-data-cassandra-0 Bound pvc-394770b3-64ba-4c35-af0c-f76a52695b25 100Gi RWO demo-sts-sc 5m46s
persistentvolumeclaim/cassandra-data-cassandra-1 Bound pvc-5e0e4935-472e-46de-bba2-91d06d301f3b 100Gi RWO demo-sts-sc 4m55s
persistentvolumeclaim/cassandra-data-cassandra-2 Bound pvc-cc4d55c4-b794-4a9f-871f-365ec0c3ad50 100Gi RWO demo-sts-sc 3m22s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-394770b3-64ba-4c35-af0c-f76a52695b25 100Gi RWO Delete Bound cassandra/cassandra-data-cassandra-0 demo-sts-sc 5m46s
persistentvolume/pvc-5e0e4935-472e-46de-bba2-91d06d301f3b 100Gi RWO Delete Bound cassandra/cassandra-data-cassandra-1 demo-sts-sc 4m54s
persistentvolume/pvc-cc4d55c4-b794-4a9f-871f-365ec0c3ad50 100Gi RWO Delete Bound cassandra/cassandra-data-cassandra-2 demo-sts-sc 3m22s
Make sure Cassandra database instances are fully participating in the cluster:
kubectl exec -it cassandra-0 -n cassandra -- nodetool status
Datacenter: Demo-DataCenter
===========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.16.1.2 104.42 KiB 32 69.1% 6670d501-ce1a-41a6-b0ab-e5654be90d05 Demo-Rack
UN 172.16.1.3 75.87 KiB 32 60.8% d4c65f54-6ba0-4caa-8b0c-9a2b945cb05c Demo-Rack
UN 172.16.1.4 81.09 KiB 32 70.1% fb0cca97-eb35-4e69-ad87-09ab18f739b2 Demo-Rack
Create and Populate a Database and Table in Cassandra
Create the DB:
kubectl exec -it cassandra-0 -n cassandra -- cqlsh
Connected to Demo-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
Create and populate table:
cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
cqlsh> use demodb;
cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, 'Tom', 'Cork', 999, 1000000);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (101, 'Andrew', 'NY', 1000, 1000000);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (102, 'Lara', 'Paris', 1001, 1000000);
cqlsh:demodb> select * from emp;
Verify:
emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
100 | Cork | Tom | 999 | 1000000
102 | Paris | Lara | 1001 | 1000000
101 | NY | Andrew | 1000 | 1000000
(3 rows)
Verify that the other Cassandra DB instances have the same information:
-
Cassandra DB instance 1
kubectl exec -it cassandra-1 -n cassandra -- cqlsh Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000 (3 rows) -
Cassandra DB instance 2
kubectl exec -it cassandra-2 -n cassandra -- cqlsh Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000 (3 rows)
Add Annotations
Add annotations for the stateful pods with the volume name cassandra-data.
kubectl get pod -n cassandra
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 19m
cassandra-1 1/1 Running 0 19m
cassandra-2 1/1 Running 0 17m
The pods cassandra-0, cassandra-1 and cassandra-2 must be annotated with cassandra-data.
kubectl -n cassandra annotate pod/cassandra-0 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-0 annotated
kubectl -n cassandra annotate pod/cassandra-1 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-1 annotated
kubectl -n cassandra annotate pod/cassandra-2 backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-2 annotated
Verify the annotations:
kubectl -n cassandra describe pod/cassandra-0 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra describe pod/cassandra-1 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
kubectl -n cassandra describe pod/cassandra-2 | grep Annotations
Annotations: backup.velero.io/backup-volumes: cassandra-data
Back Up the Cassandra Database App using Namespace
Perform the Velero back up:
velero backup create cassandra-backup --include-namespaces cassandra
Backup request "cassandra-backup" submitted successfully.
Run `velero backup describe cassandra-backup` or `velero backup logs cassandra-backup` for more details.
Verify the backup:
velero backup create cassandra-backup --include-namespaces cassandra
Backup request "cassandra-backup" submitted successfully.
Run `velero backup describe cassandra-backup` or `velero backup logs cassandra-backup` for more details.
velero backup describe cassandra-backup --details
Name: cassandra-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.17.8+vmware.1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=17
Phase: Completed
...
Velero-Native Snapshots: <none included>
Restic Backups:
Completed:
cassandra/cassandra-0: cassandra-data
cassandra/cassandra-1: cassandra-data
cassandra/cassandra-2: cassandra-data
Get the backup:
velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
cassandra-backup Completed 0 0 2020-07-29 08:26:18 -0700 PDT 29d default <none>
Use the Velero Kubernetes CustomResourceDefinition (CRD) commands to verify:
kubectl get crd
kubectl get backups.velero.io -n velero
NAME AGE
cassandra-backup 94s
kubectl describe backups.velero.io cassandra-backup -n velero
Restore the Cassandra Database App
Restore the Cassandra database app from the Velero backup. Note the following about the restore operation:
- Pod annotation is still required for Velero backup of PV.
- The namespace
cassandrawas automatically re-created.
Delete the namespace:
kubectl delete ns cassandra
namespace "cassandra" deleted
Verify that Cassandra resources are deleted:
kubectl get ns
NAME STATUS AGE
default Active 21d
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
pks-system Active 21d
velero Active 7d18h
kubectl get pvc,pv --all-namespaces
No resources found
Verify that the storage class used by the application is still present:
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
demo-sts-sc kubernetes.io/vsphere-volume Delete Immediate false 36m
Restore the Cassandra database app from the backup:
velero restore create --from-backup cassandra-backup
Restore request "cassandra-backup-20200729083742" submitted successfully.
Run `velero restore describe cassandra-backup-20200729083742` or `velero restore logs cassandra-backup-20200729083742` for more details.
Verify that the app is restored:
velero restore describe cassandra-backup-20200729083742
Name: cassandra-backup-20200729083742
Namespace: velero
Labels: <none>
Annotations: <none>
Phase: Completed
Backup: cassandra-backup
Namespaces:
Included: all namespaces found in the backup
Excluded: <none>
Resources:
Included: *
Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
Cluster-scoped: auto
Namespace mappings: <none>
Label selector: <none>
Restore PVs: auto
Restic Restores (specify --details for more information):
Completed: 3
velero restore get
NAME BACKUP STATUS ERRORS WARNINGS CREATED SELECTOR
cassandra-backup-20200729083742 cassandra-backup Completed 0 0 2020-07-29 08:37:42 -0700 PDT <none>
kubectl get ns
NAME STATUS AGE
cassandra Active 80s
default Active 21d
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
pks-system Active 21d
velero Active 7d18h
kubectl get all -n cassandra
NAME READY STATUS RESTARTS AGE
pod/cassandra-0 1/1 Running 0 2m6s
pod/cassandra-1 1/1 Running 2 2m6s
pod/cassandra-2 1/1 Running 2 2m6s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cassandra ClusterIP None <none> <none> 2m6s
NAME READY AGE
statefulset.apps/cassandra 3/3 2m6s
Verify the persistent volume:
kubectl get pvc,pv -n cassandra
Check the content of each Cassandra DB instance:
-
Cassandra DB instance 0
kubectl exec -it cassandra-0 -n cassandra -- cqlsh Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000 (3 rows) -
Cassandra DB instance 1
kubectl exec -it cassandra-1 -n cassandra -- cqlsh Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000 (3 rows) -
Cassandra DB instance 2
kubectl exec -it cassandra-2 -n cassandra -- cqlsh Connected to Demo-Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4] Use HELP for help. cqlsh> use demodb; cqlsh:demodb> select * from emp; emp_id | emp_city | emp_name | emp_phone | emp_sal --------+----------+----------+-----------+--------- 100 | Cork | Tom | 999 | 1000000 102 | Paris | Lara | 1001 | 1000000 101 | NY | Andrew | 1000 | 1000000 (3 rows)
