Backing up your Tanzu Operations Manager deployments with BBR

You can back up critical back-end components with BOSH Backup and Restore (BBR), a command-line tool for backing up and restoring BOSH deployments. For information about restoring your backup, see Restoring Tanzu Operations Manager deployments from backup with BBR.

In order to create a consistent backup, BBR stops the Cloud Controller API and the Cloud Controller workers during backup. This affects API functionality, such as when pushing apps or using the Cloud Foundry Command Line Interface (cf CLI). The deployed apps do not experience downtime.

Caution

• Security: Backup artifacts can contain secrets. Secure backup artifacts by using encryption or other means.
• Compatibility: A BBR backup can only be restored to environments matching its source environment. To ensure that a restore environment is compatible with a backup, see Compatibility of Restore.
• Failed Restore: Restore is a destructive operation. BBR is designed to restore deployments after a disaster. If a restore fails, the environment might be left in an unusable state and require re-provisioning. For more information, see Restoring Deployments from Backup with BBR.
• Service Data: BBR does not back up any service data.
• API Outage: The Cloud Controller API does not send or receive calls during restoration.

Component availability during backup

If the Cloud Controller API and workers are stopped during a backup, any services that rely on the API and workers are not functional during that backup.

Most of the effected components are stopped and restarted after a successful backup.

As of TAS for VMs v2.11, these components are:

• cc_deployment_updater
• cloud-controller-backup
• cloud_controller_clock
• cloud_controller_ng
• cloud_controller_worker
• credhub
• pcf-autoscaling
• policy-server
• routing-api
• s3-unversioned-blobstore-backup-restorer
• tps
• uaa
• usage-servicedb

For examples of the effect of component unavailability:

• Autoscaler is unable to scale apps during a backup.

• Any component within the TAS for VMs deployment that contains lock or unlock scripts for backup is temporarily unavailable during backup process.

• UAA is in read-only mode for the duration of the backup.

Recommendations

VMware recommends:

• Follow the full procedure documented in this topic when creating a backup. This ensures that you always have a consistent backup to restore from.

• Back up frequently, especially before making any changes to your deployment, such as the configuration of any tiles in Tanzu Operations Manager.

• You can prune the BOSH blobstore by running bosh clean-up --all before you run a backup of the BOSH Director. The command removes all unused resources including packages compiled against older stemcell versions. If a lot of unused resources have accumulated over time, you create a smaller, faster backup of the BOSH Director. For more information see Clean-Up in the BOSH documentation.

Running the bosh clean-up --all command is a destructive operation and can remove resources that are unused but needed. For example, if an On-Demand Service Broker is deployed and no service instances have been created, the releases needed to create a service instance are categorized as unused.

Supported components

BBR is a binary that can back up and restore BOSH deployments and BOSH Directors. BBR requires that the backup targets supply scripts that implement the backup and restore functions. BBR can communicate securely with external blobstores and databases, using TLS, if these are configured accordingly.

BBR can back up VMware Tanzu Application Service for VMs (TAS for VMs) and BOSH Director.

TAS for VMs

BBR can back up and restore TAS for VMs configured with:

• An internal MySQL database or a supported external database. For a list of supported external databases, see Supported External Databases in Configuring Cloud Foundry for BOSH Backup and Restore in the Cloud Foundry documentation.

• An internal WebDAV/NFS blobstore, an external Amazon S3 or S3-compatible blobstore, or an external Azure blobstore.

For guidance about backing up unsupported databases and blobstores, see Unsupported External Blobstores and Databases in TAS for VMs.

BOSH Director

BBR can back up and restore the BOSH Director configured with:

• An internal PostgreSQL database or a supported external database. As part of backing up the BOSH Director, BBR backs up the BOSH UAA database and the CredHub database. For a list of supported external databases, see Supported External Databases in Configuring Cloud Foundry for BOSH Backup and Restore in the Cloud Foundry documentation.

• An internal blobstore or an external versioned S3 blobstore.

BBR support for backing up and restoring external databases and blobstores varies across Tanzu Operations Manager versions. For more information, see External Storage Support Across Tanzu Operations Manager Versions.

External storage support across Tanzu Operations Manager versions

You can use BBR to back up and restore the following types of external database and blobstore configurations:

Deployment Type	External Storage Type
TAS for VMs	External Database Amazon S3 Blobstore* S3-Compatible Blobstore Azure Blobstore GCS Blobstore
BOSH Director	External Database Amazon S3 Blobstore* S3-Compatible Blobstore GCS Blobstore

^*Any S3 clone that supports the versioning API.

Unsupported external blobstores and databases in TAS for VMs

If you configured an unsupported external blobstore or an unsupported external database in TAS for VMs, see the following guidelines to perform a backup successfully:

Scenario	Action
You configured a supported database and an unsupported external blobstore in TAS for VMs.	Follow the backup procedures and copy the external blobstore by using the IaaS-specific tool.
You configured an unsupported external database and a supported blobstore in TAS for VMs.	Follow the backup procedures and copy the external database by using the IaaS-specific tool.
You configured both an unsupported external database and an unsupported external blobstore in TAS for VMs.	Follow the backup procedures but skip Back Up Your TAS for VMs Deployment. Copy the external database and blobstore by using the IaaS-specific tool.

You might encounter inconsistencies between the blobstore and database. If apps do not appear during a restore, re-push those apps.

Backing up services

BBR does not currently back up any service data.

When backing up services:

• BBR backs up and restores the service bindings, but not the service data.

• You can redeploy on-demand service instances manually during restore, but the data in the instance is not backed up.

• BBR does not back up managed services or their data.

• You can back up and restore brokered services with the procedures documented in this topic and in Restoring deployments from backup with BBR.

Backup workflow

BBR examines the jobs in the BOSH deployment, and triggers the scripts in the following stages:

1. Pre-backup lock: The pre-backup lock scripts locks the job so backups are consistent across the cluster.

2. Backup: The backup script backs up the release.

3. Post-backup unlock: The post-backup unlock script unlocks the job after the backup is complete.

Scripts in the same stage are all triggered together. For instance, BBR triggers all pre-backup lock scripts before any backup scripts. Scripts within a stage can be triggered in any order.

The backup artifacts are drained to the jumpbox, where the operator can transfer them to storage and use them to restore the deployment.

The following diagram shows a sample backup flow. A detailed description follows.

1. You run bbr backup pcf-deployment on the jumpbox.

2. BBR calls the following backup scripts on the deployment:

3. backup-lock

4. backup

5. backup-unlock

6. BBR transfers the backup artifacts from the deployment to the jumpbox.

7. BBR transfers the backup artifacts from the jumpbox to a secondary backup store that is outside of the network or Availability Zone.

Preparing to create your backup

Step 1: Set up your jumpbox

You must have a jumpbox before you can install BBR to the jumpbox. A jumpbox is a separate, hardened server on your network that provides a controlled means of access to the VMs other computers on your network.

VMware recommends using the Tanzu Operations Manager VM as your jumpbox. If you use the Tanzu Operations Manager VM as your jumpbox, the path to the root certificate authority (CA) is:

/var/tempest/workspaces/default/root_ca_certificate

You must use this path to run BBR commands.

As an alternative to using the Tanzu Operations Manager VM as your jumpbox, you can:

• Use Concourse to back up a deployment using BBR. For more information, see the BBR PCF Pipeline Tasks repository on GitHub.

• Configure a custom jumpbox. For more information, see Configuring a custom jumpbox for BBR.

Step 2: Record the Cloud Controller database encryption credentials

To retrieve the Cloud Controller database encryption credentials from the TAS for VMs tile:

1. Go to Tanzu Operations Manager Installation Dashboard.

2. Click the TAS for VMs tile.

3. Click Credentials.

4. Locate the Cloud Controller section.

5. Beside Db Encryption Credentials, click Link to Credential, and record it. You must provide these credentials when you contact Support for help restoring your installation.

   

Step 3: Retrieve BOSH Director credentials

To use BBR, you must retrieve and record the following credentials:

• Bosh Director Credentials
• Bbr Ssh Credentials
• Uaa Bbr Client Credentials

There are two ways to retrieve BOSH Director credentials:

• Tanzu Operations Manager Installation Dashboard
• Tanzu Operations Manager API

Option 1: Tanzu Operations Manager Installation Dashboard

To retrieve your BOSH Director credentials using the Tanzu Operations Manager Installation Dashboard:

1. Go to Tanzu Operations Manager Installation Dashboard.

2. Click the BOSH Director tile.

3. Click the Credentials tab.

   

4. Locate Director Credentials.

   1. Click Link to Credentials next to it.
   2. Verify the value of the identity text box. It can be director.
   3. Copy and record the value of the password text box.

5. Locate Bbr Ssh Credentials.

   1. Click Link to Credentials next to it.
   2. Copy the value of the private_key_pem text box.

   3. Reformat the copied value and save it in the current directory to a file named PRIVATE-KEY-FILE. Run:

      printf -- "YOUR-PRIVATE-KEY" > PRIVATE-KEY-FILE
      

      Where:

      • YOUR-PRIVATE-KEY is the text of your private key.
      • PRIVATE-KEY-FILE is the path to the private key file you are creating.

      For example:

      $ printf --  "-----BEGIN RSA PRIVATE KEY----- MIIEkeycontents ----END RSA PRIVATE KEY-----" > bbr_key.pem
      

6. Locate Uaa Bbr Client Credentials.

   1. Click Link to Credentials next to it.
   2. Verify the value of the identity text box. It can be bbr_client.
   3. Record the value of the password text box.

Option 2: Tanzu Operations Manager API

To retrieve BOSH Director credentials using the Tanzu Operations Manager API:

1. Obtain your UAA access token. For more information, see Access the API in Using the Tanzu Operations Manager API.

2. Retrieve the Director Credentials.

   1. Run:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/director_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Tanzu Operations Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.

   2. Verify the value of the identity text box. It can be director.

   3. Record the value of the password text box.

3. Retrieve Bbr Ssh Credentials.

   1. Run:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/bbr_ssh_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Tanzu Operations Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.

   2. Copy the value of the private_key_pem text box.

   3. Reformat the copied value and save it in the current directory to a file named PRIVATE-KEY-FILE. Run:

      printf -- "YOUR-PRIVATE-KEY" > PRIVATE-KEY-FILE
      

      Where:

      • YOUR-PRIVATE-KEY is the text of your private key.
      • PRIVATE-KEY-FILE is the path to the private key file you are creating.

      For example:

      $ printf --  "-----BEGIN RSA PRIVATE KEY----- MIIEkeycontents ----END RSA PRIVATE KEY-----" > bbr_key.pem
      

4. Retrieve Uaa Bbr Client Credentials.

   1. Run:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/uaa_bbr_client_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Tanzu Operations Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.

   2. Verify the value of the identity text box. It can be bbr_client.

   3. Record the value of the password text box.

For more information about using the Tanzu Operations Manager API, see Using the Tanzu Operations Manager API.

Step 4: Retrieve BOSH Director address

To retrieve the IP address of your BOSH Director from the BOSH Director tile:

1. If you are not using the Tanzu Operations Manager VM as your jumpbox, install the latest BOSH CLI on your jumpbox.

2. From the Installation Dashboard in Tanzu Operations Manager, click BOSH Director, then Status, and record the IP address listed for the Director. You access the BOSH Director using this IP address.

   

3. From the command line, log in to the BOSH Director, using the IP address that you previously used. Run:

   bosh -e DIRECTOR-IP \
   --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE log-in
   

   Where:

   • DIRECTOR-IP is the BOSH Director IP address recorded above.
   • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the root Certificate Authority (CA) certificate as outlined in Set Up your Jumpbox.

4. When prompted for Email (), specify director.

5. When prompted for Password (), enter the Director Credentials that you obtained in Retrieve BOSH Director Credentials.

   For example:

   $ bosh -e 10.0.0.3 \
   --ca-cert /var/tempest/workspaces/default/root_ca_certificate log-in
   Email (): director
   Password (): *******************
   Successfully authenticated with UAA
   Succeeded
   

Step 5: Check your BOSH Director

This section describes the procedures you must perform to confirm that your BOSH Director is reachable and can be backed up.

Connect to your jumpbox

You can establish a connection to your jumpbox in one of the following ways:

• Option 1: Connect with SSH
• Option 2: Connect with BOSH_ALL_PROXY

These procedures require that you have a jumpbox configured for SSH access. If you do not have a jumpbox, see Enabling SSH Access and Tunneling in the BOSH documentation.

Option 1: Connect with SSH

To connect to your jumpbox with SSH, perform one of these steps:

• If you are using the Tanzu Operations Manager VM as your jumpbox, log in to the Tanzu Operations Manager VM. For more information, see Log Into the Tanzu Operations Manager VM with SSH.

• To connect to your jumpbox using the command line, run:

  ssh -i PATH-TO-KEY JUMPBOX-USERNAME@JUMPBOX-ADDRESS
  

  Where:

  • PATH-TO-KEY is the local path to your private key file for the jumpbox host.
  • JUMPBOX-USERNAME is your jumpbox username.
  • JUMPBOX-ADDRESS is the IP address or hostname of the jumpbox.

If you connect to your jumpbox with SSH, you must run the BBR commands in this topic from within your jumpbox.

Option 2: Connect with BOSH_ALL_PROXY

You can use the environment variable BOSH_ALL_PROXY to open an SSH tunnel with SOCKS5 to the jumpbox. This tunnel enables you to forward requests from your local machine to the BOSH Director through the jumpbox. When BOSH_ALL_PROXY is set, the BBR CLI uses its value to forward requests to the BOSH Director.

For the following procedures to work, ensure the SOCKS port is not already in use by a different tunnel or process.

Using BOSH_ALL_PROXY can result in longer backup and restore times due to network performance degradation. All operations must pass through the proxy, so moving backup artifacts can be significantly slower.

To connect with BOSH_ALL_PROXY, perform one of the following procedures:

• Establish the tunnel separately from the BOSH CLI.
• Establish the tunnel using the BOSH CLI.

To establish the tunnel separately from the BOSH CLI:

1. Establish the tunnel and make it available on a local port by running:

   ssh -4 -D SOCKS-PORT -fNC JUMPBOX-USERNAME@JUMPBOX-ADDRESS -i JUMPBOX-KEY-FILE -o ServerAliveInterval=60
   

   Where:

   • SOCKS-PORT is the local SOCKS port.
   • JUMPBOX-USERNAME is your jumpbox username.
   • JUMPBOX-ADDRESS is the IP address or hostname of the jumpbox.
   • JUMPBOX-KEY-FILE is the local SSH private key for accessing the jumpbox.

2. Provide the BOSH CLI with access to the tunnel using the BOSH_ALL_PROXY environment variable by running:

   export BOSH_ALL_PROXY=socks5://localhost:SOCKS-PORT
   

   Where is SOCKS-PORT is your local SOCKS port.

To establish the tunnel using the BOSH CLI:

1. Provide the BOSH CLI with the necessary SSH credentials to create the tunnel by running:

   export BOSH_ALL_PROXY=ssh+socks5://JUMPBOX-USERNAME@JUMPBOX-ADDRESS:SOCKS-PORT?private_key=JUMPBOX-KEY-FILE
   

   Where:

   • JUMPBOX-USERNAME is your jumpbox username.
   • JUMPBOX-ADDRESS is the IP address or hostname of the jumpbox.
   • SOCKS-PORT is the local SOCKS port.
   • JUMPBOX-KEY-FILE is the local SSH private key for accessing the jumpbox.

   In BBR v1.5.0 and earlier, the tunnel created by the BOSH CLI does not include the ServerAliveInterval flag. This might cause your SSH connection to time out when transferring large artifacts.

   In BBR v1.5.1, the ServerAliveInterval flag is included. For more information, see bosh-backup-and-restore on GitHub.

Check your BOSH Director

To check your BOSH Director:

1. Run:

   bbr director \
   --private-key-path PRIVATE-KEY-FILE \
   --username bbr \
   --host HOST \
   pre-backup-check
   

   Where:

   • PRIVATE-KEY-FILE is the path to the private key file that you created from Bbr Ssh Credentials in Step 3: Retrieve BOSH Director Credentials.
   • HOST is the address of the BOSH Director. If the BOSH Director is public, HOST is a URL, such as https://bosh.xxx.cf-app.com. Otherwise, HOST is the BOSH-DIRECTOR-IP you retrieved in Step 4: Retrieve BOSH Director Address.

   To enable debug logs, use the optional --debug flag. For more information, see Logging.

2. After the pre-backup verification succeeds, continue to Step 6: Confirm backup Restore Node is deployed. If the pre-backup check fails, the BOSH Director might not have the correct backup scripts or the connection to the BOSH Director might have failed.

Step 6: Confirm backup restore node is deployed

When BBR backs up TAS for VMs, it needs a Backup Restore node.

To confirm that the Backup Restore node is deployed:

1. Go to Tanzu Operations Manager Installation Dashboard.

2. Click the TAS for VMs tile.

3. Click Resource Config.

4. Find the Backup Restore Node job and check that the Instances dropdown is set to 1.

   

5. If the Instances dropdown is not set to 1:

6. Set the Instances dropdown to 1.
7. Click Save.
8. Go to Tanzu Operations Manager Installation Dashboard.
9. Click Review Pending Changes.
10. Review your changes. For more information, see Reviewing pending product changes.
11. Click Apply Changes.

Step 7: Identify your deployment

To identify the name of the BOSH deployment:

1. Log in to your BOSH Director.

2. To identify the name of the BOSH deployment, run:

   bosh -e BOSH-DIRECTOR-IP --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE deployments
   

   Where:

   • BOSH-DIRECTOR-IP is the BOSH Director IP you retrieved in Step 4: Retrieve BOSH Director Address.
   • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the CA for the BOSH Director.

Step 8: Check your deployment

To check that your BOSH Director is reachable and has a deployment that can be backed up:

1. Run:

   bbr deployment \
   --target BOSH-DIRECTOR-IP \
   --username BOSH-CLIENT \
   --password BOSH-PASSWORD \
   --deployment DEPLOYMENT-NAME \
   --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
   pre-backup-check
   

   Where:

   • BOSH-DIRECTOR-IP is the BOSH Director IP you retrieved in Step 4: Retrieve BOSH Director Address.
   • BOSH-CLIENT, BOSH-PASSWORD are the Uaa Bbr Client Credentials, identity and password, that you retrieved in Step 3: Retrieve BOSH Director Credentials.
   • DEPLOYMENT-NAME is the deployment name you retrieved in Step 6: Identify Your Deployment.
   • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the root CA for the BOSH Director. If you are using the Tanzu Operations Manager VM as your jumpbox, the path is /var/tempest/workspaces/default/root_ca_certificate.

2. After the pre-backup check succeeds, continue to Create Your Backup. If the pre-backup check fails, the deployment you selected might not have the correct backup scripts, or the connection to the BOSH Director might have failed.

Create your backup

This section describes the procedures for creating your backup with BBR.

Step 9: Export installation settings

VMware recommends that you back up your Tanzu Operations Manager installation settings by exporting frequently.

When exporting your installation settings, keep in mind:

• This option is only available after you have deployed at least once.

• Always export your installation settings before following the procedure in Deploy Tanzu Operations Manager and Import Installation Settings in Restoring Deployments from Backup with BBR.

• Exporting your installation settings only backs up the settings you configured in Tanzu Operations Manager. It does not back up your VMs or any external MySQL databases.

• Your Tanzu Operations Manager settings are encrypted. Ensure you keep track of your Decryption Passphrase, because you need it to restore the Tanzu Operations Manager settings.

When you export installation settings for Tanzu Operations Manager v1.12 and later, releases are not included in the output file. Releases are now included in the BOSH Director backup, so the output file is much smaller than in previous Tanzu Operations Manager versions.

There are two ways to export Tanzu Operations Manager installation settings:

• Use the Tanzu Operations Manager UI
• Use the Tanzu Operations Manager API

Option 1: Use the Tanzu Operations Manager UI

To export your installation settings using the Tanzu Operations Manager UI:

1. From the Tanzu Operations Manager Installation Dashboard, click your username at the top right of the navigation.

2. Click Settings.

3. Click Export Installation Settings.

4. Click Export Installation Settings. Note the warning: “Upgrading Ops Manager may block the upgrade path for certain products. Review Upgrading Ops Manager and verify that an upgrade path exists for your installed products.”

   

Option 2: Use the Tanzu Operations Manager API

If you want to automate the backup process, you can use the Tanzu Operations Manager API to export your installation settings.

To export your installation settings using the Tanzu Operations Manager API:

1. Run:

   curl https://OPS-MANAGER-FQDN/api/v0/installation_asset_collection \
     -H "Authorization: Bearer UAA-ACCESS-TOKEN" > installation.zip
   

   Where:

   • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) for your Tanzu Operations Manager deployment.
   • UAA-ACCESS-TOKEN is your UAA access token. For more information, see Access the API in Using the Tanzu Operations Manager API.

Step 10: Back up BOSH Director

The time it takes to back up the BOSH Director varies depending on conditions such as network infrastructure and underlying hardware. For large foundations, this step can take a considerable amount of time because the BBR backup command can take a long time to complete. VMware recommends you run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. Use nohup, a screen, or a tmux session.

To back up BOSH Director:

1. Run:

   bbr director \
   --private-key-path PRIVATE-KEY-FILE \
   --username bbr \
   --host HOST \
   backup
   

   Where:

   • PRIVATE-KEY-FILE is the path to the private key file that you created from Bbr Ssh Credentials in Step 3: Retrieve BOSH Director Credentials.
   • HOST is the address of the BOSH Director:
     • If the BOSH Director is public, HOST is a URL, such as https://bosh.xxx.cf-app.com.
     • If the BOSH Director is not public, HOST is the BOSH-DIRECTOR-IP retrieved in Step 4: Retrieve BOSH Director Address.

   To enable debug logs, use the optional --debug flag. For more information, see Logging.

If your backup terminates early or fails, you must perform a cleanup.

To clean up:

1. Run:

   bbr director \
   --private-key-path PRIVATE-KEY-FILE \
   --username bbr \
   --host HOST \
   backup-cleanup
   

   Where:

   • PRIVATE-KEY-FILE is the path to the private key file that you created from Bbr Ssh Credentials in Step 3: Retrieve BOSH Director Credentials.
   • HOST is the address of the BOSH Director:
     • If the BOSH Director is public, HOST is a URL, such as https://bosh.xxx.cf-app.com.
     • If the BOSH Director is not public, HOST is the BOSH-DIRECTOR-IP retrieved in Step 4: Retrieve BOSH Director Address.

Step 11: Back up your TAS for VMs deployment

Backing up TAS for VMs can take a considerable amount of time, and varies depending on network and hardware infrastructure. In controlled conditions, backup durations have been observed ranging from a few minutes to several hours. To reduce your backup lock times, consider using selective backups. For more information, see File storage backup level. Part of the backup process causes downtime for Cloud Controller, during which you cannot push, scale, or delete apps. Your apps are not affected. Because the BBR backup command can take a long time to complete, VMware recommends you run it independently of the SSH session, so that the process can continue running even if your connection to the jumpbox fails. The prededing command uses nohup, but you could also run the command in a screen or tmux session.

To back up your TAS for VMs deployment:

1. (Optional) If you are not using an internal blobstore or a versioned S3-compatible external blobstore, create a copy of the blobstore with your IaaS-specific tool. Your blobstore backup might be slightly inconsistent with your TAS for VMs backup depending on the duration of time between performing the backups.

2. Run:

   bbr deployment \
   --target BOSH-DIRECTOR-IP \
   --username BOSH-CLIENT \
   --password BOSH-PASSWORD \
   --deployment DEPLOYMENT-NAME \
   --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
   backup
   

   Where:

   • BOSH-DIRECTOR-IP is the BOSH Director IP you retrieved in Step 4: Retrieve BOSH Director Address.
   • BOSH-CLIENT, BOSH-PASSWORD are the Uaa Bbr Client Credentials, or the identity and password, that you retrieved in Step 3: Retrieve BOSH Director Credentials.
   • DEPLOYMENT-NAME is the deployment name you retrieved in Step 6: Identify Your Deployment.
   • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the CA for the BOSH Director.
   • To enable debug logs, use the optional --debug flag. For more information, see Logging.
   • To ensure that BBR downloads your current deployment manifest when backing up, use the optional --with-manifest flag. These manifests are included in the Tanzu Operations Manager export, but are useful for reference.

3. After the command successfully completes, continue to Step 12: Transfer Backup Artifacts. If the command fails:

4. Run:

   ```
   bbr deployment \
   --target BOSH-DIRECTOR-IP \
   --username BOSH-CLIENT \
   --password BOSH-PASSWORD \
   --deployment DEPLOYMENT-NAME \
   --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
   backup-cleanup
   ```
   
   Where:
   
   - `BOSH-DIRECTOR-IP` is the BOSH Director IP you retrieved in [Step 4: Retrieve BOSH Director Address](#retrieve-address).
   - `BOSH-CLIENT`, `BOSH-PASSWORD` are the **Uaa Bbr Client Credentials**, or the identity and password, that you retrieved in [Step 3: Retrieve BOSH Director Credentials](#retrieve-credentials).
   - `DEPLOYMENT-NAME` is the deployment name you retrieved in [Step 6: Identify Your Deployment](#identify-deployment).
   - `PATH-TO-BOSH-SERVER-CERTIFICATE` is the path to the CA for the BOSH Director.
   - To enable debug logs, use the optional `--debug` flag. For more information, see [Logging](#logging).
   - To ensure that BBR downloads your current deployment manifest when backing up, use the optional `--with-manifest` flag. These manifests are included in the Tanzu Operations Manager export, but are useful for reference.
   

5. Run the BBR backup command again after ensuring that:

   • All the parameters in the command are set.
   • The BOSH Director credentials are valid.
   • The specified deployment exists.

After taking your backup

This section describes the procedures you must perform after taking your backup.

Step 12: Transfer backup artifacts

After creating your backup successfully:

1. Move the backup artifacts off the jumpbox to your preferred storage space. The backup created by BBR consists of a directory with the backup artifacts and metadata files. VMware recommends compressing and encrypting the files.

2. Make redundant copies of your backup artifacts and store them in multiple locations to minimize the risk of losing backups in the event of a disaster.

3. Attempt to restore every backup to validate the artifacts. For more information, see Step 13: Validate Your Backup.

Backup artifacts might contain data covered by European Union regulations. For more information, see European Union's General Data Protection Regulation (GDPR).

Step 13: (Optional) Validate your backup

After you back up Tanzu Operations Manager, consider validating your backup by restoring it and checking the apps. BBR is designed for disaster recovery and BBR backups are only compatible with environments matching their source environment’s configuration.

The VMs and disks from the backed-up BOSH Director must not be visible to the new BOSH Director when validating your backup. As a result, VMware recommends that you deploy the new BOSH Director to a different IaaS network and account for the VMs and disks of the backed-up BOSH Director. Data services outside of Tanzu Operations Manager might produce unexpected side effects during restoration. Restored apps and services might attempt to connect to data services when you restore to a new environment. For example, consider an app that processes mail queues and connects to an external database. When you validate your backup in a test environment, the app might start processing the queue, and this work might be lost.

Validate your backup in a second environment

To spin up a second environment and test the backup:

1. Export your Tanzu Operations Manager installation by performing the procedure in Step 8: Export Installation Settings.

2. Create a new Tanzu Operations Manager VM in a different network to the original. Ensure that the Tanzu Operations Manager VM has enough persistent disk to accommodate the files exported in the previous step. For more information, see IaaS-Specific Deployment Guidelines in Deploying BOSH and Tanzu Operations Manager.

3. Ensure that the restore environment is compatible with the backup. For more information, see Compatibility of Restore in Restoring Deployments from Backup with BBR.

4. Follow the procedures in Restoring Deployments from Backup with BBR.

Validate your TAS for VMs backup only

For a sandbox or other non-production environment, you can optionally perform an in-place restore of TAS for VMs only. In this case, you restore the TAS for VMs backup to the same Tanzu Operations Manager environment that the backup was created from. For more information, see Step 12: Redeploy TAS for VMs in Restoring Deployments from Backup with BBR.

Logging

BBR outputs logs to stdout. By default, BBR logs:

• The backup and restore scripts that it finds
• When it starts or finishes a stage, such as pre-backup scripts or backup scripts
• When the process is complete
• When any error occurs

If you require additional logging, use the optional --debug flag to print:

• Logs about the API requests made to the BOSH server
• All commands executed on remote instances
• All commands executed on local environment
• Standard in and standard out streams for the backup and restore scripts when they are executed

Canceling a backup

To cancel a backup:

1. Enter CTRL-C to terminate the BBR process, then enter yes to confirm.

2. Run the BBR backup-cleanup command.

   • For a canceled director backup, run:

     bbr director \
     --private-key-path PRIVATE-KEY-FILE \
     --username bbr \
     --host HOST \
     backup-cleanup
     

     Where:

     • PRIVATE-KEY-FILE is the path to the private key file that you created from Bbr Ssh Credentials in Step 3: Retrieve BOSH Director Credentials.
     • HOST is the address of the BOSH Director:
       • If the BOSH Director is public, HOST is a URL, such as https://bosh.xxx.cf-app.com.
       • If the BOSH Director is not public, HOST is the BOSH-DIRECTOR-IP retrieved in Step 4: Retrieve BOSH Director Address.

   • For a canceled deployment backup, run:

     bbr deployment \
     --target BOSH-DIRECTOR-IP \
     --username BOSH-CLIENT \
     --password BOSH-PASSWORD \
     --deployment DEPLOYMENT-NAME \
     --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
     backup-cleanup
     

     Where:

     • BOSH-DIRECTOR-IP is the BOSH Director IP you retrieved in Step 4: Retrieve BOSH Director Address.
     • BOSH-CLIENT, BOSH-PASSWORD are the Uaa Bbr Client Credentials, or the identity and password, that you retrieved in Step 3: Retrieve BOSH Director Credentials.
     • DEPLOYMENT-NAME is the deployment name you retrieved in Step 6: Identify Your Deployment.
     • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the CA for the BOSH Director.
     • To enable debug logs, use the optional --debug flag. For more information, see Logging.
     • To ensure that BBR downloads your current deployment manifest when backing up, use the optional --with-manifest flag. These manifests are included in the Tanzu Operations Manager export, but are useful for reference.

European Union General Data Protection Regulation

Backup artifacts might contain personal and other data covered by the European Union’s General Data Protection Regulation (GDPR).

For example, a TAS for VMs backup could contain user email addresses.

Handle backup artifacts in accordance with your organizational policies and applicable laws as they pertain to security, confidentiality, and privacy.