Clone VMware Blockchain Nodes on vSphere

To clone a VMware Blockchain deployment, you must back up the existing blockchain state and create a clone from the newly saved backup.

Prerequisites

Familiarize yourself with the cloning implementation details. See the Getting Started with VMware Blockchain guide.
Verify that the deployment descriptor parameters for cloning are configured. See Configuring the Deployment Descriptor Parameters for Cloning on vSphere.
Verify that you have successfully deployed VMware Blockchain nodes. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator on vSphere.
If the Concord operator containers were deployed, verify that the Concord operator container is running. See Instantiate the Concord Operator Container for vSphere.
Verify that you back up an existing blockchain state. See VMware Blockchain Back Up and Restore on vSphere.

Procedure

Configure the deployment descriptor files in the descriptor directory.

Sample deployment_descriptor.json file to deploy cloned the Replica, Client, and Full Copy Client nodes.

{
  "populatedReplicas": [
    {
      "zoneName": "replicas"
      "providedIp": "192.168.10.12"
    },
    {
      "zoneName": "replicas"
      "providedIp": "192.168.10.13"
    },
    {
      "zoneName": "replicas"
      "providedIp": "192.168.10.14"
    },
    {
      "zoneName": "replicas"
      "providedIp": "192.168.10.15"
  ],
  "populatedClients": [
    {
      "zoneName": "replica",
      "providedIp": "192.168.10.19"
      "groupName": "Group2",
      "clientGroupId": "709ca448-c6bf-4cb3-b9f3-5964a6cd9fe9",
      "damlDbPassword": "<password>"
    }
    "populatedFullCopyClients": [
        {
            "accessKey": "ACCESS_KEY_1",

            "bucketName": "s3BUCKET_1",

            "protocol": "HTTP",

            "secretKey": "SECRET_KEY_1",

            "url": "109.72.217.75:9000",

            "zoneName": "replicas",

            "providedIp": "192.168.10.16",

            "nodeId": "2c935ce1-9300-4101-a8e2-a157a1a392ab"

        }
  ],
  "blockchain": {
    "consortiumName": "finance",
    "blockchainType": "Daml"
  "blockchainId": "6d5ba56b-2ae7-4bdd-bff3-27294b64c370"
 }
}

Stop the Client node components.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop

vmbc@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop root@localhost [ ~ ]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.8.0.0.53  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.8.0.0.53  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent vmbc@localhost [ ~ ]#

Pause all the Replica nodes at the same checkpoint from the operator container and check the status periodically until all the Replica nodes' status is true.
Any blockchain node or nodes in state transfer or down for other reasons cause the wedge status command to return false. The wedge status command returns true when state transfer completes and all Replica nodes are healthy, allowing all Replica nodes to stop at the same checkpoint successfully.

Wedge command might take some time to complete. The metrics dashboards indicate nodes that have stopped processing blocks as they have been wedged. If you notice a false report in the dashboard, contact the VMware Blockchain support to diagnose the Replica nodes experiencing the problem. If the Wedge command times out, the system operator must execute the Wedge command again.
```
./concop wedge stop
# Stop all replicas on the next next checkpoint {'additional_data': 'set stop flag', 'succ': True} or {'succ': False}
 
./concop wedge status
# Check the wedge status of the replicas list
 
Keep trying the status command periodically until all replicas return true.
```
Check that all the Replica nodes are stopped in the same state.
Verifying that the LastReacheableBlockID and LastBlockID sequence number of each Replica node stopped helps determine if any nodes lag.

If there is a lag when you power on the Replica Network, some Replica nodes in the state-transfer mode might have to catch up. Otherwise, it can result in a failed consensus and require restoring each Replica node from the latest single copy.
```
docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <ImageName> /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastBlockID
docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <image_name> /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastReachableBlockID
```
The <image_name> is the Concord-core image name in the blockchain.
vmwaresaas.jfrog.io/vmwblockchain/concord-core:1.8.0.0.53
Encrypt and redirect the infrastructure and the deployment descriptor files for added security.
1. Encrypt the infrastructure_descriptor.json file.
```
$HOME/descriptors > ansible-vault encrypt infrastructure_descriptor.json
New Vault password:
Confirm New Vault password:
Encryption successful
```
2. Encrypt the deployment_descriptor.json file.
```
$HOME/descriptors > ansible-vault encrypt deployment_descriptor.json
New Vault password:
Confirm New Vault password:
Encryption successful
```
3. Configure the two environment variable values.
  - ORCHESTRATOR_OUTPUT_DIR - The output directory where the output file is written.
  - ORCHESTRATOR_DEPLOYMENT_TYPE - Set deployment type to PROVISION.
4. Run the secure-orchestrator.sh script from the orchestrator_runtime directory.
```
ORCHESTRATOR_OUTPUT_DIR=$HOME/output ORCHESTRATOR_DEPLOYMENT_TYPE=PROVISION ./secure-orchestrator.sh
```
  The script creates temporary files.
  - /dev/shm/orchestrator-awsIGoa0JA/infra_descriptor
  - /dev/shm/orchestrator-awsIGoa0JA/deployment_descriptor
5. Redirect the decrypted infrastructure_descriptor.json to the infrastructure_descriptor file location.
  Use the vault password used to encrypt the infrastructure_descriptor.json file.
```
ansible-vault view $HOME/descriptors/infrastructure_descriptor.json > /dev/shm/orchestrator-awsIGoa0JA/infra_descriptor
```
6. Redirect the decrypted deployment_descriptor.json to the deployment_descriptor file location.
  Use the vault password used to encrypt the deployment_descriptor.json file.
```
ansible-vault view $HOME/descriptors/deployment_descriptor.json > /dev/shm/orchestrator-awsIGoa0JA/deployment_descriptor
```
  After the script completes running, the temporary files are deleted.
7. (Optional) If the script fails or the secure_orchestrator.sh script is terminated, delete the temporary folder under the /dev/shm/orchestrator-* directory.

Run the VMware Blockchain Orchestrator cloning script.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors
INFRA_DESC_FILENAME=infrastructure_descriptor_clone.json
DEPLOY_DESC_FILENAME=deployment_descriptor_clone.json 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=CLONE
docker-compose -f docker-compose-orchestrator.yml up

Change the COMPONENT_NO_LAUNCH parameter in the /config/agent/config.json file to False on all the Replica and Client nodes.
```
sudo sed -i 's/"COMPONENT_NO_LAUNCH": "True"/"COMPONENT_NO_LAUNCH": "False"/g' /config/agent/config.json
```
Restart the agent.
```
docker restart agent
```