As a best practice, you must back up all the data on the Client nodes within the Client group to perform operations such as restoring or cloning.

Backing up Client node data lets you retrieve the data if an operation fails, and you must restore the lost data.

The Client node backup data is stored in a repository such as a Network File System (NFS) or SSHFS file server. The file server is mounted in the Client node VM pre-defined /mnt/client-backups/<node-id> directory.

Note:

As a best practice, it is recommended to secure the NFS file server data to avoid a data breach.

Prerequisites

  • Verify that you have the deployed blockchain ID information.
  • Familiarize yourself with the backup and restore consideration for your VMware Blockchain nodes. See VMware Blockchain Node Backup and Restore Considerations on vSphere.
  • Verify that you have access to the latest version of VMware Blockchain.

  • Verify that you have captured the IP addresses of all the Client node VMs and have access to them. You can find the information in the VMware Blockchain Orchestrator descriptor file.

Procedure

  1. Set up a secure NFS file server to store the Client node backup data.
  2. Mount the NFS to the target directory /mnt/client-backups.

    The mount point configuration involves updating /etc/fstab and running the mount command.

  3. Validate that the Daml index database container has read and write access to the backup directory.
    1. Get the Daml index database image ID.

      sudo docker images

      Sample image ID, 873681a0fd91.

    2. Launch a container with the Daml index database image. 
      sudo docker run -it --rm --name=daml_index_db_mount_test -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/11/bin -e GOSU_VERSION=1.12 -e LANG=en_US.utf8 -e PG_MAJOR=11 -e PG_VERSION=11.12-1.pgdg90+1 -e PGDATA=/var/lib/postgresql/data -e POSTGRES_USER=indexdb -e POSTGRES_PASSWORD=indexdb -e POSTGRES_MULTIPLE_SCHEMAS=daml_ledger_api -e POSTGRES_CONFIG_FILE= -v /config/daml-index-db:/config/daml-index-db:rw -v /config/generic:/config/generic:rw -v /mnt/data/db:/var/lib/postgresql/data:rw  -v /config/pgbackrest:/etc/pgbackrest -v /mnt/client-backups:/mnt/client-backups --network=blockchain-fabric --expose=5432 --entrypoint /bin/bash <daml-index-db-image>

      Substitute <daml-index-db-image> with the image ID from the previous step.

      A Bash shell prompt appears.

    3. Switch to a postgres user.

      su - postgres

    4. Switch to the mounted directory.

      cd /mnt/client-backups

    5. With the postgres user permissions, create test files, directories, nested files, and directories, and validate that you have read permission.

      If you receive a read or write permission error, your mount point is not configured correctly. Remount the NFS file server with the correct permissions. The remount might require some reconfiguration on the NFS server.

  4. Configure the mounted file server for the Client node backup.
    1. Stop all the applications that invoke connection requests to the Daml Ledger.
    2. Stop the Client node components. 
      curl -X POST 127.0.0.1:8546/api/node/management?action=stop
      vmbc@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop vmbc@localhost [ ~ ]# sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.0.234  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.0.234  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent vmbc@localhost [ ~ ]#
    3. Start all the Client node components. 
      curl -X POST 127.0.0.1:8546/api/node/management?action=start
    4. Restart the agent. 
      sudo docker restart agent
    5. If a backup was configured after the Client node deployment using the agent REST API, reissue the API. 
      POST http://127.0.0.1:8546/api/backup
{
    "retention_days": 33,
    "schedule_frequency": "DAILY"
}
    6. Verify that the backup was configured properly. 
      GET http://127.0.0.1:8546/api/backup/status

      The call returns the following API response from the Client node.

      {
    "backup_state": {
        "state": "ENABLED",
        "state_change_time": "2021-11-29T22:40:33.828981Z[UTC]"
    },
    "execution_error": "",
    "execution_message": "... some message ...",
    "execution_status_code": 0,
    "last_run_end_time": "2021-11-29T22:29:25.71632Z[UTC]",
    "last_run_start_time": "2021-11-29T22:29:09.876718Z[UTC]"
    "in_progress": false
}
    7. Verify that the backup_state value is ENABLED and execution_status_code value is 0.
  5. Capture the semantic data to validate the restore process for point-in-time restore.

    When a backup process runs regularly and at intermediate times, you must capture data during the restore process to make sure that the restored data is consistent.

    1. Access the Docker container.

      sudo docker exec -it daml_index_db bash

    2. Switch to a postgres user.

      su - postgres

    3. Start the psql client.

      psql -U indexdb -daml_ledger_api

    4. Run the count(*) command for the following tables. 
      select count(1) from configuration_entries;
select count(1) from flyway_schema_history;
select count(1) from package_entries;
select count(1) from packages;
select count(1) from parameters;
select count(1) from participant_command_completions;
select count(1) from participant_command_submissions;
select count(1) from participant_events;
select count(1) from participant_events_consuming_exercise;
select count(1) from participant_events_create;
select count(1) from participant_events_divulgence;
select count(1) from participant_migration_history_v100;
select count(1) from parties;
select count(1) from party_entries;
    5. Capture all the data and the timestamps and save them for further validation during restore.