Source File Not Found During Backup

You can use these options to deal with missing source files during backups.

Problem

When using filesystem level volume backups (FSB), you observe something similar in the backup logs as shown below, especially for applications like ElasticSearch.

time="2023-07-07T02:11:30Z" level=error msg="Error backing up item" backup=velero/sharedservice-full-cls-backup-daily-20230707000048 error="pod volume backup failed: running backup, stderr=error running restic backup command restic backup --repo=s3:https://s3-ap-southeast-1.amazonaws.com/tmc-cluster-backup-prod/01GB1KBTPXVZACM1NKZPXWD2YR/restic/prod-o11y --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=backup=sharedservice-full-cls-backup-daily-20230707000048 --tag=backup-uid=ab09902b-9d65-49b9-bc9b-3153a30d01ba --tag=ns=prod-o11y --tag=pod=elasticsearch-data-1 --tag=pod-uid=8a1bd1f5-ea5c-41ce-a170-28cf4ad18a8b --tag=pvc-uid=4a8ba939-8bd0-443b-91aa-cb0a36066586 --tag=volume=data --host=velero --json --parent=a9f71e50 with error: exit status 3 stderr: {\"message_type\":\"error\",\"error\":{\"Op\":\"lstat\",\"Path\":\"indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1.fnm\",\"Err\":2},\"during\":\"archival\",\"item\":\"/host_pods/8a1bd1f5-ea5c-41ce-a170-28cf4ad18a8b/volumes/kubernetes.io~csi/pvc-4a8ba939-8bd0-443b-91aa-cb0a36066586/mount/indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1.fnm\"}\n{\"message_type\":\"error\",\"error\":{\"Op\":\"lstat\",\"Path\":\"indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1_Lucene90_0.dvd\",\"Err\":2},\"during\":\"archival\",\"item\":\"/host_pods/8a1bd1f5-ea5c-41ce-a170-28cf4ad18a8b/volumes/kubernetes.io~csi/pvc-4a8ba939-8bd0-443b-91aa-cb0a36066586/mount/indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1_Lucene90_0.dvd\"}\n{\"message_type\":\"error\",\"error\":{\"Op\":\"lstat\",\"Path\":\"indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1_Lucene90_0.dvm\",\"Err\":2},\"during\":\"archival\",\"item\":\"/host_pods/8a1bd1f5-ea5c-41ce-a170-28cf4ad18a8b/volumes/kubernetes.io~csi/pvc-4a8ba939-8bd0-443b-91aa-cb0a36066586/mount/indices/IF-3K2QkQLSG3-ZyKKa1Nw/0/index/_2x79j_1_Lucene90_0.dvm\"}\nWarning: at least one source file could not be read\n: error running restic

Cause

This happens when there is churn in the filesystem and the file is no longer present while performing pod volume backup.

Solution

• ♦ Use one of the following options:

  Option	Description
  Use CSI volume snapshots.	You may choose to back up your volumes using CSI volume snapshots instead of filesystem level volume backups. When creating a backup, select the option to perform CSI volume snapshots and use the opt-in approach for filesystem backups.
  Use the opt-out approach.	If CSI volume snapshot is not supported in your cluster, then exclude the volume using the opt-out approach for filesystem backups.
  Use an application specific backup method.	Some applications provide their own method to perform a backup. For example, backup of ElasticSearch using filesystem backup is not recommended. For information about backing up ElasticSearch, see https://www.elastic.co/guide/en/elasticsearch/reference/8.2/snapshot-restore.html#other-backup-methods.