Troubleshooting BOSH Backup and Restore

This topic tells you how to solve common problems that you might encounter when using BOSH Backup and Restore (BBR) to back up and restore your deployments.

Troubleshooting during a restore

This section provides solutions for errors that occur during a restore.

Restore fails with a MySQL Monit start timeout

Symptom

While running the BBR restore command, restoring the job mysql-restore fails with the following error:

1 error occurred:

* restore script for job mysql-restore failed on mysql/0.
...
"Monit start failed: Timed out waiting for monit": 2m0s

Explanation

This happens when the MySQL job fails to start within the timeout period. It ends up in an Execution Failed state and monit never tries to start it again.

Solution

Ensure that your MySQL Server cluster has only one instance. If there is more than one instances of MySQL Server, the restore fails with a monit start timeout. Scale down to one instance and retry.

If your MySQL Server cluster is already scaled down to one node, it might have taken longer than normal to restart the cluster. To manually verify and retry:

List the VMs in your deployment by running:
```
bosh -e BOSH-DIRECTOR-IP --ca-cert /var/tempest/workspaces/default/root_ca_certificate \
-d DEPLOYMENT-NAME \
ssh
```
Where:
- BOSH-DIRECTOR-IP is the BOSH Director IP. To retrieve this IP address, see Retrieve BOSH Director Address in Restoring Deployments from Backup with BBR.
- DEPLOYMENT-NAME is the deployment name. To retrieve the deployment name, see Identify Your Deployment in Restoring Deployments from Backup with BBR.
Locate the mysql VM.
SSH into the mysql VM.
To check that the MySQL job process is running, run:
```
ps aux | grep galera-init
```
For more information, see Configuring TAS for VMs.
Check that monit reports that the MySQL job process is in an Execution Failed state by running:
```
sudo monit summary
```
If so, deactivate monitoring by running:
```
monit unmonitor
```
Enable monitoring by running:
```
monit monitor
```
After a few minutes, run:
```
monit summary
```
The command should report that all the processes are running.
Re-attempt the restore with BBR.

Deployment does not match the structure of the backup

Symptom

The following error displays:

Deployment 'deployment-name' does not match the structure of the provided backup

Explanation

The instance groups with the restore scripts in the destination environment do not match the backup metadata. For example, they might have the wrong number of instances of a particular instance group, or the metadata names an instance group that does not exist in the destination environment.

Solution

BBR only supports restoring to an environment that matches your original environment. VMware recommends altering the destination environment to match the structure of the backup.

General troubleshooting

This section provides solutions for general errors.

Connection error

Symptom

BBR displays an error message containing “SSH Dial Error” or another connection error.

Explanation

The jumpbox and the VMs in the deployment are experiencing connection problems.

Solution

To resolve connection problems between the jumpbox and the VMs in the deployment:

Ensure your deployment is healthy by running:
```
bosh vms
```
Clean up the data from the failed backup on the instances by running:
```
bbr deployment backup-cleanup
```
This step is required. Otherwise, further BBR commands fail.
Repeat the BBR operation.

Error running metadata script

Symptom

BBR backup or restore fails with a metadata error:

1 error occurred:
error 1:
An error occurred while running metadata script for job redis-server on redis/0ce9f81f-1756-480b-8e3e-a4609b14b6a6: error from metadata

Explanation

There is a problem with your deployment install.

Solution

Contact Support.