Restoring lost BOSH Director persistent disk

You can use BOSH Backup and Restore (BBR) to restore a BOSH Director that has lost its persistent disk.

Prerequisite

Restoring a BOSH Director persistent disk with BBR is only possible if you have previously taken a backup using the bbr director backup command.

This command produces a backup artifact in the format DIRECTOR_IP_TIMESTAMP/.

BOSH Director backups require identical references to all BOSH-deployed VMs. If BOSH re-creates your deployment VMs for any reason, such as changes to stemcells, networks, or availability zones (AZs), your BBR backup artifact is no longer compatible for performing a restore.

Prepare your environment

Your BOSH Director must be in a healthy state before you can perform a BOSH Director restore. The BOSH Director is in a healthy state when all jobs are running.

If your BOSH Director has lost its persistent disk, you must create a new disk to return your BOSH Director to a healthy state.

Create a new BOSH Director persistent disk

To create a new persistent disk for the BOSH Director:

Create a new persistent disk using your IaaS console.
Log in to VMware Tanzu Operations Manager VM. For more information, see Log Into the Tanzu Operations Manager VM with SSH.
Open the bosh-state.json file in a text editor.
Edit the disks.cid value to match the new persistent disk you created in the first step.
Update the persistent disk size in Tanzu Operations Manager and then click Apply Changes. This creates a new BOSH Director VM and attaches the new persistent disk to it.

Perform the restore

Caution BBR restore is a destructive process which removes any current data in your deployment. Performing a restore overwrites all new data since you created your most recent backup artifact.

Perform a BOSH Director restore by following the BBR instructions in Step 8: Restore the BOSH Director in Restoring Deployments From Backup with BBR.

Consolidate your environment

After you successfully restore your BOSH Director, the BOSH Director uses the VM references that were stored in the backup artifact. If you clicked Apply Changes in Tanzu Operations Manager between when the backup and the restore took place, check if any BOSH-deployed VMs were deleted or added.

If changes have occurred, ensure that the BOSH Director database is consistent with the current state of your IaaS after the restore. If VMs were deleted, the BOSH Director still has references to these deleted VMs after the restore. If VMs were added, the BOSH Director has no knowledge of the new VMs.

To ensure that the BOSH Director database matches your IaaS console:

Find and delete outdated VM references for an environment by running:
```
bosh --deployment DEPLOYMENT-NAME cloud-check
```
Where DEPLOYMENT-NAME is the name of your BOSH deployment.
Log in to your IaaS console and delete any orphaned VMs that your BOSH Director does not reference.

Your BOSH Director should now be in a healthy state with the new persistent disk attached. Your BOSH deployment VMs should be aligned with the current state of your IaaS.