Recovering from Tanzu Operations Manager and TAS for VMs upgrade failure

You can use BOSH Backup and Restore (BBR) to restore VMware Tanzu Operations Manager and VMware Tanzu Application Service for VMs (TAS for VMs) to a healthy state after a failed upgrade between TAS for VMs minor versions.

BBR can only restore across the same TAS for VMs minor versions. To verify which version of TAS for VMs was used to create the backup artifact, see metadata/cf-VERSION.yml in the backup artifact.

Evaluate your deployment

You can restore Tanzu Operations Manager and TAS for VMs to a healthy state after a failed upgrade between TAS for VMs minor versions if your backups, BOSH Director, and TAS for VMs deployment meet certain conditions. Follow the procedures in this section to confirm that your deployment can be restored by BBR.

Review your backup

Before you perform a restore, you must:

Run the following bbr commands from your Tanzu Operations Manager VM:
- Back up your deployments by running:
```
bbr deployment --deployment DEPLOYMENT-NAME backup
```
  Where DEPLOYMENT-NAME is the name of your TAS for VMs deployment.
- Back up your BOSH Director by running:
```
bbr director backup
```
  The above bbr commands produce the following backup artifacts:
  - DEPLOYMENT_NAME_TIMESTAMP/
  - DIRECTOR_IP_TIMESTAMP/
  - installation.zip
Export your Tanzu Operations Manager settings. For more information, see Settings Page in Using the Tanzu Operations Manager Interface.

Validate your environment for restore

Before you can restore after a failed upgrade between TAS for VMs minor versions, you must confirm that your BOSH Director and TAS for VMs deployment are healthy.

Confirm that your BOSH Director is healthy

To verify that the BOSH Director is in a healthy state for a bbr restore, SSH to the Tanzu Operations Manager VM and ensure that all jobs are running.

To ensure that all jobs are running:

SSH into your BOSH Director. For more information, see Log Into the BOSH Director VM with SSH in Advanced Troubleshooting with the BOSH CLI.
View all deployment job instances by running:
```
  monit summary
```
Monit should report that all jobs are running.
Ensure that the BOSH API is responsive by running the following command:
```
bosh configs
```
If this command runs successfully, then the BOSH Director API is in a healthy and responsive state and can be restored by BBR.

If your BOSH Director is unhealthy, see Restoring an Unhealthy BOSH Director.

Confirm that your TAS for VMs deployment is healthy

To confirm that TAS for VMs is in a healthy state:

Run:
```
bosh -d DEPLOYMENT-NAME instances
```
Where DEPLOYMENT-NAME is the name of your TAS for VMs BOSH deployment.

If your TAS for VMs deployment is healthy, then the output states that all of your instances are in the running state. If all instances are running, TAS for VMs can be restored by BBR.

If your TAS for VMs deployment is unhealthy, see Restoring an Unhealthy TAS for VMs Deployment.

Restore your deployment

After you confirm that your BOSH Director is healthy with its data intact, and all instances in your TAS for VMs deployment are responsive, you can restore TAS for VMs with BBR.

Restoring TAS for VMs

If your BOSH Director is healthy, you can restore across the same TAS for VMs minor version by following the procedures in Step 14: Restore TAS for VMs in Restoring Deployments from Backup with BBR.

Recover to a healthy state

If either the BOSH Director or TAS for VMs are not in a healthy state, they must be recovered before you can run bbr restore. If you need guidance on the following subsections, contact Support.

Restoring an unhealthy BOSH Director

If the BOSH Director is not in a healthy state, VMware recommends following the full restore workflow to regain your Tanzu Operations Manager, BOSH Director, and BOSH deployments. For more information, see Restore Workflow in Restoring Deployments from Backup with BBR.

Restoring an unhealthy TAS for VMs deployment

If your BOSH Director is healthy, you can restore your previous BOSH Director state by following the procedure in Step 8: Restore the BOSH Director in Restoring Deployments from Backup with BBR.

After your BOSH Director is restored, you can then restore your deployments by following the procedure in Step 14: Restore TAS for VMs in Restoring Deployments from Backup with BBR.