Bind the Newly Deployed VMware Blockchain Nodes on vSphere

After deploying the newly scaled VMware Blockchain nodes, you must bind these newly created nodes to the existing VMware Blockchain nodes.

Prerequisites

Verify that all the newly deployed Replica and the Client nodes are available and running.

Note:

If a node is unavailable after the scale-up process, that node is no longer be able to participate in the new blockchain configuration. See Scale-Up Deployed VMware Blockchain Nodes on vSphere.

Procedure

Stop all the applications that invoke the connection requests to the Daml Ledger.

Instantiate the operator container.

You can only use one Client node, which is part of the original blockchain, as an operator container.

SSH into a Client node.
Verify that the Client node has an operator container image and identify the operator image ID.

sudo docker images | grep "operator"
Verify that the operator container configuration file has content.
```
sudo cat /config/daml-ledger-api/concord-operator/operator.config
```
For deployments with unencrypted configuration, copy the private key content into the following location.
```
sudo vi /config/daml-ledger-api/concord-operator/operator_priv.pem
```

Deploy the encrypted operator private key content into the following location for deployments with encrypted configuration.

sudo docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public <Operator_IMAGE_ID>

Launch the Client node operator container.

sudo docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public <OP_IMAGE_ID>

A sample operator image is vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.0.234

Access the Client node operator container.
```
sudo docker exec -it operator bash
```

Navigate to the /home/blockchain/descriptors directory.

Create a reconfigure descriptor JSON file and set the parameter values in the descriptor directory.

The reconfigure descriptor file contains combined details of the nodes from the output files of the original blockchain nodes and scaled nodes. The file semantics and structure must adhere to the rules of a reconfiguration descriptor.

Note:

Use the same operator specification keys used during the initial deployment.

Values such as zoneName, providedIp, and nodeId are available in the original VMware Blockchain Orchestrator appliance output directory, /home/blockchain/output.

Sample reconfigure_deployment_descriptor.json file.

{
   "blockchain": {
        "consortiumName": "reconfigure",
        "blockchainType": "DAML",
        "blockchainId": "72a0f104-6845-46df-98d4-5ab0bf5623b"
    },
    "populatedReplicas": [
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.239",
            "nodeId": "2aea8345-9b94-4d19-b106-717de2d0d6fc"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.240",
            "nodeId": "1d3af60b-5921-4665-84cd-e2850a05c906"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.241",
            "nodeId": "f4a0025f-0415-484c-8975-382c7773d120"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.242",
            "nodeId": "3208e7d6-7e83-43d2-8136-645c79889c78"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.243",
            "nodeId": "915910ab-bc78-4b00-a049-acc504c7c01f"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.244",
            "nodeId": "f3539c24-6909-404c-9f9f-88b14a0aff58"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.245",
            "nodeId": "39c7e39a-323d-465e-a9b3-af9c369f7679"
       }
    ],
    "replicaNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 32,
        "diskSizeGb": 64
    },
    "populatedClients": [
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.246",
            "damlDbPassword": "x3lvSnUm-7V_3H0",
            "clientGroupId" : "73292c2d-fc0e-4629-8291-676051c8c398",
            "groupName" : "GROUP01",
            "nodeId": "25b5b049-290f-406f-8c5a-0c43d197950"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.247",
            "damlDbPassword": "x3lvSnUm-7V_3H0",
            "clientGroupId" : "73292c2d-fc0e-4629-8291-676051c8c398",
            "groupName" : "GROUP01",
            "nodeId": "f44c9431-9651-4935-a70c-8e1141ae6bbc"
        }
    ],
    "clientNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 32,
        "diskSizeGb": 64
    },
    "operatorSpecifications": {
        "operatorPublicKey": ""-----BEGIN PUBLIC KEY-----
          \nMFkwEwYHKoZ\n
         -----END PUBLIC KEY-----\n""
    }
}

Note:

The blockchain ID refers to the initial deployment blockchain ID. The blockchainId value must match the blockchain ID to which you want to add the nodes.

Validate that the credentials, network name, and folder name are correctly populated to avoid errors.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  
INFRA_DESC_FILENAME=infrastructure_descriptor.json 
DEPLOY_DESC_FILENAME=reconfigure_deployment_descriptor.json 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE 
docker-compose -f docker-compose-orchestrator.yml up

Run the VMware Blockchain Orchestratorreconfigure script.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
INFRA_DESC_FILENAME=infrastructure_descriptor.json
DEPLOY_DESC_FILENAME=reconfigure_deployment_descriptor.json  
ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
docker-compose -f docker-compose-orchestrator.yml up

The new node reconfiguration ID is available in the reconfiguration output file.

Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb

Stop the original Client nodes in the Client group.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop

vmbc@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop vmbc@localhost [ ~ ]# sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.0.234  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.6.0.0.234  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent vmbc@localhost [ ~ ]#

Verify that all the containers except the agent ,CRE, and deployed operator container are running on the selected Client node.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop
sudo docker ps -a

If the sudo docker ps -a command shows that some containers, with the exception agent and deployed operator container, are still running, rerun the command or use the sudo docker stop <container_name> command to stop the containers.

sudo docker start cre
image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator");docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
sudo docker ps

SSH into the VMware Blockchain Orchestrator appliance.
Enter the login credentials for the VMware Blockchain user account.

Log into the Client running the operator node configuration.

# sudo docker exec operator sh -c './concop scale --clients execute --configuration <new-configuration-id>'
# sudo docker exec operator sh -c './concop scale --replicas execute --configuration <new-configuration-id>'

The new reconfiguration ID is available in the reconfiguration output file.

Log into the new node configuration and verify the new configuration.

# sudo docker exec operator sh -c './concop scale --replicas status'
{"10.72.95.242":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.243":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.244":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.245":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true}}
# sudo docker exec operator sh -c './concop scale --clients status'
{"10.72.95.242":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.243":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.244":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.245":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}]}

Stop the original Replica nodes at the same state.

# sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastBlockID
{ "lastBlockID": "47197" }
# sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastReachableBlockID
{ "lastReachableBlockID": "47197" }

Repeat the stop operation on each Replica node in the Replica Network.
Back up the Client nodes data.
1. On the Client node from the initial deployment, create a /mnt/data/db/ archive.
2. Transfer the archive to the new Client nodes and extract the data.
```
#on a client node from the original deployment:
sudo tar cvzf </path/><bckp-name> /mnt/data/db
sudo scp -r </path/><bckp-name> vmbc@<the-client-ip-to-be-added>:/config
 
#on the newly created client node:
sudo tar -zxvf </path/><bckp-name> -C /
```
3. Use the original Client password.
  
  If you do not want to reuse the password, you can change the password.
Restore the RocksDB data from the original Replicanode to the new Replica nodes.

Update the agent configuration for all the newly created Replica and Client nodes.

sudo sed -i '/COMPONENT_NO_LAUNCH\|SKIP_CONFIG_RETRIEVAL/d' /config/agent/config.json
sudo sed -i 's/inactive/<new-configuration-id>/g' /config/agent/config.json

The new configuration ID is available in the reconfiguration output file.

Download the new configuration for the newly created Replica and Client nodes.
```
curl -ik -X POST http://localhost:8546/api/node/reconfigure/<new-configuration-id>
```
The new configuration ID is available in the reconfiguration output file.

Restart all the Replica nodes in the Replica Network.

curl -ik -X POST -H "content-type: application/json" --data '{ "containerNames" : ["all"] }' http://localhost:8546/api/node/restart

Restart all the Client nodes.

curl -ik -X POST -H "content-type: application/json" --data '{ "containerNames" : ["all"] }' http://localhost:8546/api/node/restart

Remove the old operator and create another operator.

# sudo docker rm -f operator
operator
 
# sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
Paste the private key and press ctrl+d on a blank line:
-----BEGIN PRIVATE KEY-----
MHcCAQEEINRW1u1O6g0pls==
-----END PRIVATE KEY-----Provision successful
Key installed successfully
 
# sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator");docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
b7918e8bac6c9ff99ca6dc95cd9bffab80084117396fa89d6a8508a2bffeea8c

Verify that all the existing and newly scaled nodes appear in Wavefront.

Verify that the following metrics indicate that your blockchain network is operating properly.

Option	Description
Metrics	Description
Blocks per second metrics	All the blockchain nodes must process blocks because time blocks are constantly being added. The nodes should be a positive number to be considered in a healthy state.
FastPaths	All blockchain nodes must report in a fast path, and none reporting in a slow path. When the Blocks per second metrics indicate an unhealthy state, the wedge status is always false until all the nodes have stopped at the same checkpoint.