Recovering from a Single Provider Failure

If your single Provider VM fails, you can recover it from the latest backup of its internal vPostgres database only if you have configured the Provider Backup repository in your Data Management for VMware Tanzu installation.

Recovering involves deploying a new Provider VM. You must deploy the new VM with the same version of the .ova, and the same IP address, as that of the original.

Data Management for VMware Tanzu automatically transfers all trusted certificates to the new Provider VM after the backup is restored.

Audience

The procedure in this topic is performed by a Data Management for VMware Tanzu user in the Provider Administrator role.

Prerequisites

Before you recover the Provider VM, ensure that you can identify:

The version of the original Provider VM, and that you have the corresponding .ova.
The IP address of the original Provider VM.
The S3 location and credentials for the Provider Backup Repo.

If you are restoring from an TLS-enabled S3 repository, also ensure that you have the CA certificate on hand, and that you can identify a location on the Provider VM for this file.

Procedure

Perform the following procedure to recover the Provider VM in your Data Management for VMware Tanzu installation:

Generate a JSON-format recovery file that specifies the Provider Backup Repo object repository access credentials.
1. Open a file named provider_recovery.json in the editor of your choice.
2. Copy/paste the following text into the file:
```
{
"AWS_ACCESS_KEY_ID": "<placeholder>",
"AWS_SECRET_ACCESS_KEY": "<placeholder>",
"WALG_S3_PREFIX": "<placeholder>",
"AWS_ENDPOINT": "<placeholder>",
"AWS_S3_FORCE_PATH_STYLE": "true",
"AWS_REGION": "<placeholder>"
}
```
3. Replace the <placeholder>s with the appropriate values for your environment.
4. If you are accessing an SSL-enabled S3:
  1. Identify the file system location on the Provider VM for the CA certificate.
  2. Add a comma after the AWS_REGION line.
  3. Add the following text to identify the cert file location (you will copy the CA certificate to this location in a later step):
```
"WALG_S3_CA_CERT_FILE": "</path/to/custom/ca/file>"
```
5. Save the file and exit the editor.
Deploy a new Provider VM; you must specify the same .ova and IP address as that of the original VM.
Power off the old VM.
Power on the newly-deployed VM.
Copy the provider_recovery.json file that you previously created to /opt/.walg.json on the Provider VM. For example:
```
$ scp provider_recovery.json root@providervm://opt/.walg.json
```
Log in to the new Provider VM using root credentials.
Verify that the vpostgresql service is running by using the command:
```
root@providervm$ service vpostgresql status
```
If the vpostgres service is not running, then destroy the new Provider VM and deploy another Provider VM with the same .version of the .ova and follow the steps from Step 4.
Run the restore-provider command to restore the internal Provider vPostgres database:
```
root@providervm$ restore-provider -c /opt/.walg.json
```
Wait until the restoration process is complete.
Log out of the VM.

Note: Recovery make take some time.
Log in to the Data Management for VMware Tanzu Provider console to verify the recovery process.

Note: If log in fails, recovery may still be in-progress. Wait a few minutes and try again.
Select System Audit from the left navigation pane.

This actions displays the System Audit Events view, a table that lists system audit events.
Examine recent events listed in the table, and locate an Event Type that specifies Provider Restore Success that has an Event Time near the time of recovery.
Manually trigger a Provider VM backup by updating the Provider backup settings.