As a best practice, you must back up the data on the Client nodes within the Client group and Replica nodes in the Replica Network to perform operations such as upgrade, restore, or cloning.

Backing up the Client and Replica nodes data ensures that you can access the data in case an operation fails and you must restore the lost data.

Prerequisites

  • Verify that you have the deployed blockchain ID information.
  • Verify that you have access to the latest version of VMware Blockchain.
  • Verify that you have captured the IP addresses of all the Client and Replica node VMs and have access to them. You can find the information in the VMware Blockchain Orchestrator descriptor file.

Procedure

  1. Stop all the applications that invoke connection requests the DAML Ledger.
  2. Stop the Client node.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop root@localhost [ ~ ]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmware-docker-blockchainsaas.bintray.io/vmwblockchain/operator:1.1.0.0.11 "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmware-docker-blockchainsaas.bintray.io/vmwblockchain/agent:1.1.0.0.11 "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent root@localhost [ ~ ]#
  3. Repeat the stop operation on each Client node in the Client group.
  4. Verify that all the containers except for agent and deployed operator keys are stopped.

    docker ps

    If the docker ps command shows that some containers, with the exception agent and deployed operator keys are still running, then rerun the command or use the docker stop <container_name> command to stop the containers.

  5. Back up the data on each Client node.
    tar cvzf <backup_name> /mnt/data/db 
    #For data greater than 64GB 
    rsync -avh /mnt/data/db <destination_folder> 

    The <backup_name> must end in .tar.gz. For example, db-backup.tar.gz.

    The rsync command might time out due to SSH inactivity, incrementally rerun the command.

  6. (Optional) For the upgrade scenario, perform a configuration backup on each Client node.
    tar cvzf <backup_name> /config/daml-ledger-api/environment-vars /config/daml-index-db/environment-vars /config/telegraf/telegraf.conf

    The <backup_name> is user-defined. The backup can be on the VM or external mounted storage.

  7. (Optional) If operator keys were deployed, pause all the Replica nodes at the same checkpoint from the operator container and check the status periodically until all the Replica nodes' status is true.
    docker exec -it operator sh -c './concop wedge stop' {"succ":true} docker exec -it operator sh -c './concop wedge status' {"192.168.100.107":true,"192.168.100.108":true,"192.168.100.109":true,"192.168.100.110":true} 
  8. Stop the Replica node.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# docker ps
    CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                      NAMES
    3b7135c677cf        vmware-docker-blockchainsaas.bintray.io/vmwblockchain/agent:1.1.0.0.11   "java -jar node-agen…"   20 hours ago        Up 20 hours         127.0.0.1:8546->8546/tcp   agent
  9. Repeat the stop operation on each Replica node in the Replica Network.
  10. Verify that all the containers except for the agent are stopped.

    docker ps

    If the docker ps command shows that some containers beside the agent are running, then rerun the command or use the docker stop <container_name> command to stop the containers.

  11. Check that all the Replica nodes are stopped in the same state.

    Verifying that the LastReacheableBlockID and LastBlockID sequence number of each Replica node stopped helps determine if any nodes are lagging.

    If there is a lag, when you power on the Replica Network some Replica nodes in the state-transfer mode might have to catch up. Otherwise, it can result in a failed consensus and require restoring each Replica node from the latest single copy.

    docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <ImageName> /concord/sparse_merkle_db_editor /concord/rocksdbdata getLastBlockID
    docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <image_name> /concord/sparse_merkle_db_editor /concord/rocksdbdata getLastReachableBlockID

    The <image_name> is the Concord-core image name in the blockchain.

    vmware-docker-blockchainsaas.bintray.io/vmwblockchain/concord-core:<ImageVersion>

  12. Back up the data on each Replica node.
    tar cvzf <backup_name> /mnt/data/db 
    #For data greater than 64GB 
    rsync -avh /mnt/data/db <destination_folder> 

    The <backup_name> must end in .tar.gz. For example, db-backup.tar.gz.

    The rsync command might time out due to SSH inactivity, incrementally rerun the command.

  13. Start all the Replica nodes.
    curl -X POST 127.0.0.1:8546/api/node/management?action=start 
  14. Start all the Client nodes.
    curl -X POST 127.0.0.1:8546/api/node/management?action=start
  15. Start all the applications that invoke connection requests the DAML Ledger

What to do next

If there is a failure or you want to clone a deployment, you can restore it from the backup data. See Restore VMware Blockchain from the Backup Data.