After you deploy the new VMware Blockchain nodes, you must bind these newly created nodes to the existing VMware Blockchain nodes.

Prerequisites

Verify that you have at least three new Replica nodes deployed. See Scale-Up Deployed VMware Blockchain Nodes.

Procedure

  1. Stop all the applications that invoke the connection requests to the DAML Ledger.
  2. Instantiate the operator container.

    You can only use one Client node, which is part of the original blockchain, as an operator container.

    1. SSH into a Client node.
    2. Verify that the Client node has an operator container image and identify the operator image ID.

      docker images | grep "operator"

    3. Verify that the operator container configuration file has content.
      cat /config/daml-ledger-api/concord-operator/operator.config
    4. Copy the private key content into the following location.
      vi /config/daml-ledger-api/concord-operator/operator_priv.pem
    5. Launch the Client node operator container.
      docker run -d --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public <Operator_Image_ID>
  3. Stop the Client node.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop root@localhost [ ~ ]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.2.0.0.34  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.2.0.0.34  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent root@localhost [ ~ ]#
  4. Repeat the stop operation on each Client node in the Client group.
  5. Verify that all the containers except for the agent and deployed operator container are stopped on the selected Client node.

    docker ps -a

    If the docker ps -a command shows that some containers, with the exception agent and deployed operator container, are still running, then rerun the command or use the docker stop <container_name> command to stop the containers.

  6. (Optional) If the operator container on the selected Client node was deployed, pause all the Replica nodes at the same checkpoint of the operator container and check the status periodically until all the Replica nodes' status is true.

    Any blockchain node or nodes in state transfer, or down for other reasons, cause the wedge status command to return false. The wedge status command returns true when state transfer completes, and all Replica nodes are healthy, allowing all Replica nodes to stop at the same checkpoint successfully.

    Wedge command might take some time to complete. The metrics dashboards indicate nodes that have stopped processing blocks as they have been wedged. If you notice a false report in the dashboard, contact VMware Blockchain support to diagnose the Replica nodes experiencing the problem.

    docker exec -it operator sh -c './concop wedge stop' {"succ":true} docker exec -it operator sh -c './concop wedge status' {"192.168.100.107":true,"192.168.100.108":true,"192.168.100.109":true,"192.168.100.110":true} 
  7. Verify that the following metrics indicate that your blockchain network is operating properly.
    Option Description

    Metrics

    Description

    Blocks per second metrics

    All the blockchain nodes must process blocks because time blocks are constantly being added. The nodes should be a positive number to be considered in a healthy state.

    FastPaths

    All Replica nodes must report in a fast path and none reporting in a slow path. When the Blocks per second metrics indicate an unhealthy state, the wedge status is always false until all the nodes have stopped at the same checkpoint.

  8. Stop components on the Replica nodes.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# docker ps -a
    CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                      NAMES
    3b7135c677cf        vmwaresaas.jfrog.io/vmwblockchain/agent:1.2.0.0.34    "java -jar node-agen…"   20 hours ago        Up 20 hours         127.0.0.1:8546->8546/tcp   agent
  9. Repeat the stop operation on each Replica node in the Replica Network.
  10. Verify that all the containers except for the agent are stopped.

    docker ps -a

    If the docker ps -a command shows that some containers beside the agent are running, then rerun the command or use the docker stop <container_name> command to stop the containers.

  11. Verify that the Replica nodes from the initial deployment are stopped at the same block and display the same block ID.
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastBlockID
    Note:

    If all the Replica nodes are not stopped at the same block ID, the process might have failed.

  12. Back up the Rocks DB data from a Replica node and copy it to all the new Replica nodes.
    1. Create a backup from the Rocks DB Replica node.

      tar cvzf /config/<backup_name> /mnt/data/rocksdbdata

      The <backup_name> must end in .tar.gz. For example, db-backup.tar.gz.

    2. Copy the backup file to all the new Replica nodes.

      scp -r /config/<backup_name> root@<IP_address>:/config

      The <IP_address> is the Replica node IP address.

    3. Extract the backup file content on the new Replica nodes.

      tar -zxvf /config/<backup_name> -C /

  13. Capture the metadata of the DAML database.
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "daml-ledger-api");docker run -v /config/daml-ledger-api/environment-vars:/config/daml-ledger-api/environment-vars --network blockchain-fabric $image -- --dump-index-metadata | grep WARN | cut -d "|" -f 10 | cut -d "(" -f1
  14. Remove the metadata from all the Replica nodes.
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata removeMetadata
  15. Clean up the old configuration on all the Replica nodes.
    rm /config/concord/config-generated/genSec_*
    rm /config/agent/configDownloadMarker
  16. Clean up the old configuration on all the Client nodes.
    rm /config/agent/configDownloadMarker
  17. SSH into the VMware Blockchain Orchestrator appliance.
  18. Enter the login credentials for the blockchain user account.
  19. Navigate to the /home/blockchain directory.
  20. Create a reconfigure scaleup descriptor JSON file and set the parameter values in the descriptor directory.

    The reconfigure scaleup descriptor file contains the original and new node details.

    Sample reconfigure_scaleup_deployment_descriptor.json file to bind the new Replica nodes.

    {
      "populatedReplicas": [
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.12"
          "nodeId": "ab98ab8c-6982-4341-a8c7-194fd6783c82"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.13"
          "nodeId": "335f56ee-b8d8-6557-9158-2ed635122be3"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.14"
          "nodeId": "ol5v5rws-hj01-4590-6421-f4t7136u2jg3"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.15"
          "nodeId": "0c3fba7a-f14c-45bc-8e67-5f674e3fa14c"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.20"
          "nodeId": "920699cd-ae6f-49fa-ad6e-723dccc2ec5a"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.21"
          "nodeId": "6224649c-0d39-4c50-8ea7-d22aae46091d"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.22"
          "nodeId": "8fafce9e-99b1-4239-87ea-23e756e1edaf"
        }
      ],
       "replicaNodeSpec": {
         "cpuCount": 2,
         "memoryGb": 16,
         "diskSizeGb": 64
        },  
        "blockchain": {
         "consortiumName": "finance",
         "blockchainType": "DAML"
         "blockchainId": "6d5ba56b-2ae7-4bdd-bff3-27294b64c370"
     }
    }
  21. Validate your deployment descriptor file semantics against the reconfigure-descriptor-v1.schema.
  22. Validate that vSphere credentials, network name, and folder name are correctly populated to avoid errors.
    ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  INFRA_DESC_FILENAME=infrastructure_descriptor.json DEPLOY_DESC_FILENAME=scaleup_deployment_descriptor.json ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
    ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE docker-compose -f docker-compose-orchestrator.yml up
  23. Run the VMware Blockchain Orchestrator redeployment script.
    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory> INFRA_DESC_FILENAME=<infra-desc-file-name> DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name> 
    CONFIG_SERVICE_IP=<orchestrator-ip-address>
    ORCHESTRATOR_OUTPUT_DIR=<output-directory> docker-compose -f docker-compose-orchestrator.yml up

    The <orchestrator-ip-address> is the VMware Blockchain Orchestratorappliance IP address running the configuration service container.

    The VMware Blockchain Orchestrator output file includes the reconfiguration ID for the new node configuration.

  24. Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

    Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb

  25. Locate the old configuration session ID on all the deployed VMware Blockchain nodes.
    grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"
  26. Replace the old configuration ID with the new ID on all the deployed VMware Blockchain nodes.

    For VMware Blockchain nodes created using the scale-up operation, you must remove the following, "COMPONENT_NO_LAUNCH": "True" and "SKIP_CONFIG_RETRIEVAL": "True" lines.

    sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
    rm -rf /config/agent/configDownloadMarker
  27. Verify that all the containers except for the agent are stopped.

    docker ps -a

    If the docker ps -a command shows that some containers beside the agent are running, remove these containers docker rm -f <container_name>.

  28. Restart the agents.
    1. Restart the agents on all the deployed Replica nodes.
      curl -X POST 127.0.0.1:8546/api/node/management?action=remove
      docker restart agent
    2. Restart the agents on all the deployed Client nodes.
      curl -X POST 127.0.0.1:8546/api/node/management?action=remove
      docker restart agent