With the scale-down operation, you can remove the Replica nodes from the existing Replica node Network when they are not needed.

The supported node configuration for scaling down is (n=7, f=2) to (n=4, f=1). The operator container must generate the new configuration for the selected nodes to scale down the number of nodes in a Replica Network. The new configuration contains IP addresses, public keys, and certificates for the remaining Replica and Client nodes in a new node configuration.

Prerequisites

  • Determine whether you need to scale down your node configuration. See VMware Blockchain Node Scaling Operations.

  • Verify that you have at least 7 nodes deployed. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator.

  • Note down the Blockchain ID, consortium ID, and Blockchain type details.

  • Note down the Replica node zone name, Replica node IP address, and Replica node ID details.

  • Note down the Client node zone name, Client node IP address, Client node ID, Clientnode DAML database password, and Client node group name and group ID name details.

Procedure

  1. Stop all the applications that invoke the connection requests to the DAML Ledger.
  2. Instantiate the operator container.

    You can only use one Client node, which is part of the original blockchain, as an operator container.

    1. SSH into a Client node.
    2. Verify that the Client node has an operator container image and identify the operator image ID.

      docker images | grep "operator"

    3. Verify that the operator container configuration file has content.
      cat /config/daml-ledger-api/concord-operator/operator.config
    4. Copy the private key content into the following location.
      vi /config/daml-ledger-api/concord-operator/operator_priv.pem
    5. Launch the Client node operator container.
      docker run -d --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public <Operator_Image_ID>
  3. Stop the Client node.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop root@localhost [ ~ ]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 218a1bdaddd6 vmwaresaas.jfrog.io/vmwblockchain/operator:1.2.0.0.34  "/operator/operator_…" 18 hours ago Up 18 hours operator cd476a6b3d6c vmwaresaas.jfrog.io/vmwblockchain/agent:1.2.0.0.34  "java -jar node-agen…" 18 hours ago Up 18 hours 127.0.0.1:8546->8546/tcp agent root@localhost [ ~ ]#
  4. Repeat the stop operation on each Client node in the Client group.
  5. Verify that all the containers except for the agent and deployed operator container are stopped on the Client node.

    docker ps -a

    If the docker ps -a command shows that some containers, with the exception agent and deployed operator container, are still running, then rerun the command or use the docker stop <container_name> command to stop the containers.

  6. Verify the Replica Network health.
    docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep fetchingState
    docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView|Replica ID|Primary '
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
  7. (Optional) On the selected Client node, use the operator container to pause all the Replica nodes at the same checkpoint as the operator container and check the status periodically until all the Replica nodes' status is true.

    Any blockchain node or nodes in state transfer, or down for other reasons, cause the wedge status command to return false. The wedge status command returns true when state transfer completes, and all Replica nodes are healthy, allowing all Replica nodes to stop at the same checkpoint successfully.

    Wedge command might take some time to complete. The metrics dashboards indicate nodes that have stopped processing blocks as they have been wedged. If you notice a false report in the dashboard, contact VMware Blockchain support to diagnose the Replica nodes experiencing the problem.

    docker exec -it operator sh -c './concop wedge stop' {"succ":true} docker exec -it operator sh -c './concop wedge status' {"192.168.100.107":true,"192.168.100.108":true,"192.168.100.109":true,"192.168.100.110":true} 
  8. Verify that the following metrics indicate that your blockchain network is operating properly.
    Option Description

    Metrics

    Description

    Blocks per second metrics

    All the blockchain nodes must process blocks because time blocks are constantly being added. The nodes should be a positive number to be considered in a healthy state.

    FastPaths

    All Replica nodes must report in a fast path and none reporting in a slow path. When the Blocks per second metrics indicate an unhealthy state, the wedge status is always false until all the nodes have stopped at the same checkpoint.

  9. Stop components on the Replica nodes.
    curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# curl -X POST 127.0.0.1:8546/api/node/management?action=stop
    root@localhost [ ~ ]# docker ps -a
    CONTAINER ID        IMAGE                                                                  COMMAND                  CREATED             STATUS              PORTS                      NAMES
    3b7135c677cf        vmwaresaas.jfrog.io/vmwblockchain/agent:1.2.0.0.34    "java -jar node-agen…"   20 hours ago        Up 20 hours         127.0.0.1:8546->8546/tcp   agent
  10. Repeat the stop operation on each Replica node in the Replica Network.
  11. Verify that all the containers except for the agent are stopped.

    docker ps -a

    If the docker ps -a command shows that some containers beside the agent are running, rerun the command or use the docker stop <container_name> command to stop the containers.

  12. Verify that the Replica nodes from the initial deployment are stopped at the same block and display the same block ID.
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastBlockID
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata getLastReachableBlockID
  13. Remove the metadata from all the Replica nodes.
    image=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep "concord");docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata removeMetadata
  14. Clean up the old configuration on all the Replica nodes.
    rm /config/concord/config-generated/genSec_*
    rm /config/agent/configDownloadMarker
  15. Clean up the old configuration on all the Client nodes.
    rm /config/agent/configDownloadMarker
  16. SSH into the VMware Blockchain Orchestrator appliance.
  17. Enter the login credentials for the blockchain user account.
  18. Navigate to the /home/blockchain directory.
  19. Create a reconfigure scaledown descriptor JSON file and set the parameter values in the descriptor directory.

    The reconfigure scaledown descriptor file contains the original and new node details.

    Sample reconfigure_scaledown_deployment_descriptor.json file to bind the new Replica nodes.

    {
      "populatedReplicas": [
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.12"
          "nodeId": "ab98ab8c-6982-4341-a8c7-194fd6783c82"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.13"
          "nodeId": "335f56ee-b8d8-6557-9158-2ed635122be3"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.14"
          "nodeId": "ol5v5rws-hj01-4590-6421-f4t7136u2jg3"
        },
        {
          "zoneName": "replicas"
          "providedIp": "192.168.10.15"
          "nodeId": "0c3fba7a-f14c-45bc-8e67-5f674e3fa14c"
        }
      ],
      "replicaNodeSpec": {
            "cpuCount": 2,
            "memoryGb": 16,
            "diskSizeGb": 64
        },
        "blockchain": {
         "consortiumName": "finance",
         "blockchainType": "DAML"
         "blockchainId": "6d5ba56b-2ae7-4bdd-bff3-27294b64c370"
     }
    }
  20. Validate your deployment descriptor file semantics against the reconfigure-descriptor-v1.schema.
  21. Validate that vSphere credentials, network name, and folder name are correctly populated to avoid errors.
    ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  INFRA_DESC_FILENAME=infrastructure_descriptor.json DEPLOY_DESC_FILENAME=scaleup_deployment_descriptor.json 
    CONFIG_SERVICE_IP=<orchestrator-ip-address>
    ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
    ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE docker-compose -f docker-compose-orchestrator.yml up

    The <orchestrator-ip-address> is the VMware Blockchain Orchestratorappliance IP address running the configuration service container.

  22. Run the VMware Blockchain Orchestrator redeployment script.
    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE
    ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory>
    INFRA_DESC_FILENAME=<infra-desc-file-name>
    DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name>
    ORCHESTRATOR_OUTPUT_DIR=<output-directory>
    docker-compose -f docker-compose-castor.yml up

    The VMware Blockchain Orchestrator output file includes the session ID for the new node configuration.

  23. Identify the reconfiguration ID in the /home/blockchain/output directory.

    Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb

  24. Locate the old configuration session ID on all the deployed VMware Blockchain nodes.
    grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"
  25. Replace the old configuration ID with the new ID on all the deployed VMware Blockchain nodes.
    sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
    rm -rf /config/agent/configDownloadMarker
  26. Verify that all the containers except for the agent are stopped.

    docker ps -a

    If the docker ps -a command shows that some containers beside the agent are running, remove these containers docker rm -f <container_name>.

  27. Restart the agents.
    1. Restart the agents on all the deployed Replica nodes.
      curl -X POST 127.0.0.1:8546/api/node/management?action=remove
      docker restart agent
    2. Restart the agents on all the deployed Client nodes.
      curl -X POST 127.0.0.1:8546/api/node/management?action=remove
      docker restart agent
  28. Check the Replica Network health.
    docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep fetchingState
    docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView|Replica ID|Primary '
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10