After deploying the newly scaled VMware Blockchain nodes, you must generate the configuration for the new and existing nodes, and bind these newly created nodes to the VMware Blockchain topology.

Note:

The new node reconfiguration is available up to 45 minutes. You must finalize your reconfiguration within the time frame for the new deployment to succeed.

Prerequisites

Verify that all the newly deployed Replica and the Client nodes are available and running.

Note:

If a Client node is unavailable before the scale-up process, that node is no longer be able to participate in the new blockchain configuration. You must reconfigure the Client node to scale up. See Scale-Up Deployed VMware Blockchain Nodes on vSphere.

Procedure

  1. Navigate to the /home/blockchain/descriptors directory.
  2. Create a reconfigure descriptor JSON file and set the parameter values in the descriptor directory.

    The reconfigure descriptor file contains combined details of the nodes from the output files of the original blockchain nodes and scaled nodes. The file semantics and structure must adhere to the rules of a reconfiguration descriptor.

    Note:

    You must add the operator specification keys used to the reconfiguration descriptor.

    Values such as zoneName, providedIp, and nodeId are available in the original VMware Blockchain Orchestrator appliance output directory, /home/blockchain/output.

    Sample reconfigure_deployment_descriptor.json file.

    {
   "blockchain": {
        "consortiumName": "reconfigure",
        "blockchainType": "DAML",
        "blockchainId": "72a0f104-6845-46df-98d4-5ab0bf5623b"
    },
    "populatedReplicas": [
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.239",
            "nodeId": "2aea8345-9b94-4d19-b106-717de2d0d6fc"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.240",
            "nodeId": "1d3af60b-5921-4665-84cd-e2850a05c906"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.241",
            "nodeId": "f4a0025f-0415-484c-8975-382c7773d120"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.242",
            "nodeId": "3208e7d6-7e83-43d2-8136-645c79889c78"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.243",
            "nodeId": "915910ab-bc78-4b00-a049-acc504c7c01f"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.244",
            "nodeId": "f3539c24-6909-404c-9f9f-88b14a0aff58"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.245",
            "nodeId": "39c7e39a-323d-465e-a9b3-af9c369f7679"
       }
    ],
    "replicaNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 32,
        "diskSizeGb": 64
    },
    "populatedClients": [
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.246",
            "damlDbPassword": "x3lvSnUm-7V_3H0",
            "clientGroupId" : "73292c2d-fc0e-4629-8291-676051c8c398",
            "groupName" : "GROUP01",
            "nodeId": "25b5b049-290f-406f-8c5a-0c43d197950"
        },
        {
            "zoneName": "SDDC8",
            "providedIp": "10.72.95.247",
            "damlDbPassword": "x3lvSnUm-7V_3H0",
            "clientGroupId" : "73292c2d-fc0e-4629-8291-676051c8c398",
            "groupName" : "GROUP01",
            "nodeId": "f44c9431-9651-4935-a70c-8e1141ae6bbc"
        }
    ],
    "clientNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 32,
        "diskSizeGb": 64
    },
    "operatorSpecifications": {
        "operatorPublicKey": ""-----BEGIN PUBLIC KEY-----
          \nMFkwEwYHKoZ\n
         -----END PUBLIC KEY-----\n""
    }
}
    Note:

    The blockchain ID refers to the initial deployment blockchain ID. The blockchainId value must match the blockchain ID to which you want to add the nodes.

  3. Validate that the credentials, network name, and folder name are correctly populated to avoid errors. 
    ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  
INFRA_DESC_FILENAME=infrastructure_descriptor.json 
DEPLOY_DESC_FILENAME=reconfigure_deployment_descriptor.json 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE 
docker-compose -f docker-compose-orchestrator.yml up
  4. Run the VMware Blockchain Orchestratorreconfigure script. 
    ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
INFRA_DESC_FILENAME=infrastructure_descriptor.json
DEPLOY_DESC_FILENAME=reconfigure_deployment_descriptor.json  
ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
docker-compose -f docker-compose-orchestrator.yml up

    The new node reconfiguration ID is available in the reconfiguration output file.

  5. Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

    Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb

  6. Stop all the applications that invoke the connection requests to the Daml Ledger in the original deployment.
  7. Stop the original Client nodes in the Client group. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
  8. Instantiate the Concord operator container.

    You can only use one Client node, which is part of the original blockchain, as an Concord operator container.

    1. SSH into a Client node designated to host the operator Client node.
      Note:

      Reboot the Concord container before using an operator container from a different Client node.

    2. Verify that the Client node has a Concord operator container image and identify the operator image ID.

      sudo docker images | grep "operator"

    3. Verify that the Concord operator container configuration file is available. 
      sudo cat /config/daml-ledger-api/concord-operator/operator.config
    4. For deployments with unencrypted configuration, copy the private key content into the following location. 
      sudo vi /config/daml-ledger-api/concord-operator/operator_priv.pem
    5. For deployments with encrypted configuration, deploy the encrypted operator private key content into the following location.

      The private key is encrypted and stored on disk because the deployment secure store configuration requires private keys not to be saved unprotected on disk.

      # Replace <OP_IMAGE_ID> and paste the operator private key when prompted
sudo docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public <Operator_IMAGE_ID>
    6. Launch the Concord operator container Client node. 
      curl -X POST 127.0.0.1:8546/api/node/start-operator
    7. Access the Concord operator container Client node. 
      sudo docker exec -it operator bash
  9. Verify that all the containers except the agent, CRE, and deployed Concord operator container are running on the selected Client node. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
sudo docker ps -a

    If the sudo docker ps -a command shows that some containers, with the exception agent and deployed operator container, are still running, rerun the command or use the sudo docker stop <container_name> command to stop the containers.

    sudo docker start cre
image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator"); docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "operator");docker run -d --network=blockchain-fabric --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public $image
sudo docker ps
  10. Log into the Client node running the operator node configuration. 
    sudo docker exec -it operator bash
./concop scale --clients execute --configuration d5c28fc4-4c58-4864-88bb-219c16f4609b  --tokens   "{ \"53a061d5-1109-45b8-9938-20259aaef8c4\" : \"U2FsdGVkX1/gvoD7/gjnjuU+peYd8ycSOiZRUYJYzDBtPbxBpwxmuYZ9NclBX7GrJgWghKqychJjGCol4telgg3pCOjDrM6eS1bt+L4/5aAupPKu2bDiKT9l/2lLiTU7\", \"5abcd2cb-da6e-4d41-a77a-6a7ab3d4ef9d\" : \"U2FsdGVkX1/u2ITU2Z2/G8WqbKRsYoIW5DC95VgS0XTztIu81J885LbrsrAKEmDQfQvCtnPuXElWsKdK7SeRAb0XwTY5O/X2IxK9kmXP8i9p3VqDYBg+ECGQn3IeOV64\", \"7cc9cddf-15a7-4514-89ec-6f387542dd31\" : \"U2FsdGVkX18A0C8V60bd6u+i8dCPiH6poyRPfEivEqLnTXMqyP8YmaVWS3N2P3AIHzlAfznSGAte57yah9YAOSzvBJvk9+dkZ+GdpkhmwWH3+eppS5RM3m5Q3GLIu775\",\" 92e75b0b-1f8e-4851-bc90-93a56a099b40\" : \"U2FsdGVkX18u+xDk+QvfHWJlgqfKjgv0UDpTrNG3hZYL5HRb3VNz1RtnYp96mwSQMaFpyLK/qvXCS8nKqYw9z1bJY6PqkIe3ixbMk4GSLX0SubERSkCxRiBlsZW5O+Cb\",\"fd052a82-f984-45b2-ba56-390076534cef\" : \"U2FsdGVkX1/up52xhqAeE5lnfqpm6lD96kiBa0wuTGB5guSA3QxlPWQ1i/1LTd8cjGPpPlNNpqgo7cQ1PsCQmnPDjVVhbJkxpkJtH0lYt32b/MqiKfxIMcw7QScetbDI\",\"7e6936f5-a955-4ed4-9ee8-af9ceb20d8a6\" : \"U2FsdGVkX1+u+XiUSWfctLntS/kpHpfqFk9DbQrYfUdsJjvu4SuT6cLOwtSizzJO5saaZiyNwnw3UIfv0uHP6H8oSEBEMZo8RewQWIJcHCcYonRufpgZFz95vM+yAKX+\",\"bfd7d591-1894-49ed-831e-670b5ce3ee50\" : \"U2FsdGVkX19vYTPf+3xf8apsl/x3nykvXZeFjFF+qVfmatwSmFlg6YA62mzgpH8KnJknyoI42lU04tT9ypR2v/gylOAeOzETpj89YttAO1N/Q5rHW7z4xYvPllxbGQID\",\"17f647a3-e7ab-4975-9b89-64cb2723d759\" : \"U2FsdGVkX19bQ70oCV/LzMqMD99+O+16wckyFNjyyJGI9nCrDUrwPU2SO0/UeeDPUK5zUZatuEB+SYS8anpadvdT7ppLlLIB5nbFYKvd0vqeNS8DGo5v17n1nCVdUbHe\",\"7b13682a-ea04-435c-86aa-3b0b8972806b\" : \"U2FsdGVkX18SNmXNTWSCsQdi1eLyZkF9cHVj4yeSWCtiVrYQ3Yqa7ZOmctkP6z+OGQruNX2drm1eK38//FF4IseWtTLsB+KqCM5yXTOLO+7Sg1kjBvRtZS9HREfNpZmq\"}"
./concop scale --replicas execute --configuration d5c28fc4-4c58-4864-88bb-219c16f4609b --tokens   "{ \"53a061d5-1109-45b8-9938-20259aaef8c4\" : \"U2FsdGVkX1/gvoD7/gjnjuU+peYd8ycSOiZRUYJYzDBtPbxBpwxmuYZ9NclBX7GrJgWghKqychJjGCol4telgg3pCOjDrM6eS1bt+L4/5aAupPKu2bDiKT9l/2lLiTU7\", \"5abcd2cb-da6e-4d41-a77a-6a7ab3d4ef9d\" : \"U2FsdGVkX1/u2ITU2Z2/G8WqbKRsYoIW5DC95VgS0XTztIu81J885LbrsrAKEmDQfQvCtnPuXElWsKdK7SeRAb0XwTY5O/X2IxK9kmXP8i9p3VqDYBg+ECGQn3IeOV64\", \"7cc9cddf-15a7-4514-89ec-6f387542dd31\" : \"U2FsdGVkX18A0C8V60bd6u+i8dCPiH6poyRPfEivEqLnTXMqyP8YmaVWS3N2P3AIHzlAfznSGAte57yah9YAOSzvBJvk9+dkZ+GdpkhmwWH3+eppS5RM3m5Q3GLIu775\",\" 92e75b0b-1f8e-4851-bc90-93a56a099b40\" : \"U2FsdGVkX18u+xDk+QvfHWJlgqfKjgv0UDpTrNG3hZYL5HRb3VNz1RtnYp96mwSQMaFpyLK/qvXCS8nKqYw9z1bJY6PqkIe3ixbMk4GSLX0SubERSkCxRiBlsZW5O+Cb\",\"fd052a82-f984-45b2-ba56-390076534cef\" : \"U2FsdGVkX1/up52xhqAeE5lnfqpm6lD96kiBa0wuTGB5guSA3QxlPWQ1i/1LTd8cjGPpPlNNpqgo7cQ1PsCQmnPDjVVhbJkxpkJtH0lYt32b/MqiKfxIMcw7QScetbDI\",\"7e6936f5-a955-4ed4-9ee8-af9ceb20d8a6\" : \"U2FsdGVkX1+u+XiUSWfctLntS/kpHpfqFk9DbQrYfUdsJjvu4SuT6cLOwtSizzJO5saaZiyNwnw3UIfv0uHP6H8oSEBEMZo8RewQWIJcHCcYonRufpgZFz95vM+yAKX+\",\"bfd7d591-1894-49ed-831e-670b5ce3ee50\" : \"U2FsdGVkX19vYTPf+3xf8apsl/x3nykvXZeFjFF+qVfmatwSmFlg6YA62mzgpH8KnJknyoI42lU04tT9ypR2v/gylOAeOzETpj89YttAO1N/Q5rHW7z4xYvPllxbGQID\",\"17f647a3-e7ab-4975-9b89-64cb2723d759\" : \"U2FsdGVkX19bQ70oCV/LzMqMD99+O+16wckyFNjyyJGI9nCrDUrwPU2SO0/UeeDPUK5zUZatuEB+SYS8anpadvdT7ppLlLIB5nbFYKvd0vqeNS8DGo5v17n1nCVdUbHe\",\"7b13682a-ea04-435c-86aa-3b0b8972806b\" : \"U2FsdGVkX18SNmXNTWSCsQdi1eLyZkF9cHVj4yeSWCtiVrYQ3Yqa7ZOmctkP6z+OGQruNX2drm1eK38//FF4IseWtTLsB+KqCM5yXTOLO+7Sg1kjBvRtZS9HREfNpZmq\"}"

    The new reconfiguration ID is available in the reconfiguration output file.

  11. Verify the new configuration on the Replica nodes. 
    sudo docker exec operator sh -c './concop scale --replicas status'
{"10.72.95.242":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.243":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.244":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true},"10.72.95.245":{"bft":false,"configuration":"2147b337-1edd-4857-be84-67e662415370","restart":false,"wedge_status":true}}
sudo docker exec operator sh -c './concop scale --clients status'
{"10.72.95.242":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.243":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.244":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}],"10.72.95.245":[{"UUID":"25b5b049-290f-406f-8c5a-0c43d1979505","configuration":"2147b337-1edd-4857-be84-67e662415370"}]}
  12. Stop the original Replica nodes at the same state. 
    # sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastBlockID
{ "lastBlockID": "47197" }
# sudo image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastReachableBlockID
{ "lastReachableBlockID": "47197" }
  13. Repeat the stop operation on each Replica node in the Replica Network.
  14. Back up the Client nodes data.
    1. On the Client node from the initial deployment, create a /mnt/data/db/ archive.
    2. Transfer the archive to the new Client nodes and extract the data. 
      #on a client node from the original deployment:
sudo tar cvzf </path/><bckp-name> /mnt/data/db
sudo scp -r </path/><bckp-name> vmbc@<the-client-ip-to-be-added>:/config
 
#on the newly created client node:
sudo tar -zxvf </path/><bckp-name> -C /
  15. Back up and restore the RocksDB data from the original Replicanode to the new Replica nodes.

    See #GUID-F5AD904A-85F2-4FF4-A6C8-39B9189AC16A and #GUID-56975142-69F5-456C-BA7E-0D89FBE4100F.

  16. Update the agent configuration for all the newly created Replica and Client nodes. 
    sudo sed -i '/COMPONENT_NO_LAUNCH\|SKIP_CONFIG_RETRIEVAL/d' /config/agent/config.json
sudo sed -i 's/inactive/<new-configuration-id>/g' /config/agent/config.json

    The new configuration ID is available in the reconfiguration output file.

  17. Confirm the new configuration in the /config/agent/config.json file. 
    ....
"configurationSession": {
    "id": "2147b337-1edd-4857-be84-67e662415370"
},
"outboundProxyInfo": {
},
"nodeId": "c78bd97a-17d7-49b4-8590-11a85df9dd2f",
"properties": {
   "values": {
      "NEW_DATA_DISK": "True"
   }
}
....
  18. Restart all the Replica nodes in the Replica Network. 
    curl -ik -X POST -H "content-type: application/json" --data '{ "containerNames" : ["all"] }' http://localhost:8546/api/node/restart
  19. Start all the Client nodes. 
    curl -X POST 127.0.0.1:8546/api/node/management?action=start
  20. Restart the agent. 
    sudo docker restart agent
  21. Verify that all the existing and newly scaled nodes appear in Wavefront.
  22. Verify that the following metrics indicate that your blockchain network is operating properly.
    Option Description

    Metrics

    Description

    Blocks per second metrics

    All the blockchain nodes must process blocks because time blocks are constantly being added. The nodes should be a positive number to be considered in a healthy state.

    FastPaths

    All blockchain nodes must report in a fast path, and none reporting in a slow path. When the Blocks per second metrics indicate an unhealthy state, the wedge status is always false until all the nodes have stopped at the same checkpoint.