You can optionally restore a failed Replica node from the backup data.

Prerequisites

  • Verify that the backup was created from a healthy Replica node. See Back-Up Replica Nodes on vSphere.

  • Verify that you have captured the IP addresses of all the Replica and Client node VMs and have access to them. You can find the information in the VMware Blockchain Orchestrator descriptor file.

Procedure

  1. Stop all the applications that invoke connection requests from the Daml Ledger.
  2. Stop the Client node components. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    vmbc@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop vmbc@localhost [ ~ ]# sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.0.234  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.0.234  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent vmbc@localhost [ ~ ]#
  3. Repeat the stop operation on each Client node.
  4. Restore the backup data on each Client node. 
    sudo rm -rf /mnt/data/db
#1: change the owner of /mnt/data to vmbc user in new client sudo chown vmbc:users /mnt/data   

#2: Copy the backup data from the old client to new client rsync -avh /mnt/data/<BackupName> <destination>

#Untar the backup DB in the new client for small DB
sudo tar xvzf <BackupName> --directory /   

#Untar the backup DB in the new client for large DB 
cd /mnt/data/ 
sudo nohup tar xvzf db-backup.tar.gz --directory . &

    Use the backup that you created recently. The <backup_name> must end in .tar.gz. For example, rsync -avh /mnt/data/db-backup.tar.gz [email protected]:/mnt/data/

  5. Stop the Replica node. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    vmbc@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop
vmbc@localhost [ ~ ]# sudo docker ps -a
CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                      NAMES
3b7135c677cf        vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.0.234    "java -jar node-agen…"   20 hours ago        Up 20 hours         127.0.0.1:8546->8546/tcp   agent
  6. Repeat the stop operation on each Replica node in the Replica Network.
  7. Restore the backup data on each Replica node. 
    sudo rm -rf /mnt/data/rocksdbdata/
#1: change the owner of /mnt/data to vmbc user in new client sudo chown vmbc:users /mnt/data   

#2: Copy the backup data from the old client to new client rsync -avh /mnt/data/<BackupName> <destination>

#Untar the backup DB in the new client for small DB
sudo tar xvzf <BackupName> --directory /   

#Untar the backup DB in the new client for large DB 
cd /mnt/data/ 
sudo nohup tar xvzf db-backup.tar.gz --directory . &

    Use the backup that you created recently. The <backup_name> must end in .tar.gz. For example, rsync -avh /mnt/data/db-backup.tar.gz [email protected]:/mnt/data/

  8. (Optional) If you apply the backup data to restore a different node where the configuration is refreshed, sanitize the Replica node you have identified to restore. 
    sudo docker run -it --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <image_name> /concord/kv_blockchain_db_editor /concord/rocksdbdata removeMetadata

    The <image_name> is the Concord-core image name in the blockchain.

    vmwaresaas.jfrog.io/vmwblockchain/concord-core:1.6.0.0.234

  9. Start all the Replica nodes. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=start
  10. Verify that all the containers, such as daml_execution_engine and concord, are running.

    sudo docker ps -a

    If the containers are not running, use the command to restart the containers.

    sudo docker ps -aq | grep -v $(sudo docker ps -aq --filter='name=^/agent') | xargs 
sudo docker rm -rf
sudo docker restart agent
  11. Start all the Client node components. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=start