Scale Down Deployed VMware Blockchain Nodes on vSphere

With the scale-down operation, you can remove the Replica and Client nodes from the existing Replica node Network when they are not needed.

The supported node configuration for scaling down is (n=7, f=2) to (n=4, f=1). The operator container must generate the new configuration for the selected nodes to scale down the number of nodes in a Replica Network. The new configuration contains IP addresses, public keys, and certificates for the remaining Replica and Client nodes in a new node configuration.

Prerequisites

• Determine whether you need to scale down your node configuration. See VMware Blockchain Node Scaling Operations on vSphere.

• Verify that you have at least 7 nodes deployed. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator on vSphere.

• Note the blockchain ID, consortium ID, and Blockchain type details.

• Note the Replica node zone name, Replica node IP address, and Replica node ID details.

• Note down the Client node zone name, Client node IP address, Client node ID, Client node Daml database password, and Client node group name and group ID name details.

• Verify that all the deployed nodes are functioning properly. Failed nodes cannot be scaled down.

Procedure

1. Stop all the applications that invoke the connection requests to the Daml Ledger.
2. Instantiate the operator container.

   You can only use one Client node, which is part of the original blockchain, as an operator container.

   1. SSH into a Client node.
   2. Verify that the Client node has an operator container image and identify the operator image ID.

      sudo docker images | grep "operator"

   3. Verify that the operator container configuration file has content.

      sudo cat /config/daml-ledger-api/concord-operator/operator.config

   4. Copy the private key content into the following location.

      sudo vi /config/daml-ledger-api/concord-operator/operator_priv.pem

   5. Launch the Client node operator container.

      sudo docker run -d --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public 
      <Operator_Image_ID>

      A sample operator image is vmwaresaas.jfrog.io/vmwblockchain/operator:1.6.0.0.234.

3. Verify the Replica Network health.

   sudo docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
   sudo docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep fetchingState
   sudo docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView|Replica ID|Primary '
   sudo docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10

4. Verify that the following metrics indicate that your blockchain network is operating properly.

   Option	Description
   Metrics	Description
   Blocks per second metrics	All the blockchain nodes must process blocks because time blocks are constantly being added. The nodes should be a positive number to be considered in a healthy state.
   FastPaths	All Replica nodes must report in a fast path, and none reporting in a slow path. When the Blocks per second metrics indicate an unhealthy state, the wedge status is always false until all the nodes have stopped at the same checkpoint.

5. SSH into the VMware Blockchain Orchestrator appliance.
6. Enter the login credentials for the VMware Blockchain user account.
7. Navigate to the /home/blockchain directory.
8. Create a reconfigure scaledown descriptor JSON file and set the parameter values in the descriptor directory.

   The reconfigure scaledown descriptor file contains the original and new node details.

   Sample reconfigure_scaledown_deployment_descriptor.json file to bind the new Replica nodes.

   {
     "populatedReplicas": [
       {
         "zoneName": "replicas"
         "providedIp": "192.168.10.12"
         "nodeId": "ab98ab8c-6982-4341-a8c7-194fd6783c82"
       },
       {
         "zoneName": "replicas"
         "providedIp": "192.168.10.13"
         "nodeId": "335f56ee-b8d8-6557-9158-2ed635122be3"
       },
       {
         "zoneName": "replicas"
         "providedIp": "192.168.10.14"
         "nodeId": "ol5v5rws-hj01-4590-6421-f4t7136u2jg3"
       },
       {
         "zoneName": "replicas"
         "providedIp": "192.168.10.15"
         "nodeId": "0c3fba7a-f14c-45bc-8e67-5f674e3fa14c"
       }
     ],
     "replicaNodeSpec": {
           "cpuCount": 2,
           "memoryGb": 16,
           "diskSizeGb": 64
       },
       "blockchain": {
        "consortiumName": "finance",
        "blockchainType": "Daml"
        "blockchainId": "6d5ba56b-2ae7-4bdd-bff3-27294b64c370"
       }
     ],
     "populatedClients": [
       {
         "zoneName": "replica",
         "providedIp": "192.168.10.19"
         "groupName": "Group2",
         "nodeId": "<provide-value>",
         "clientGroupId": "709ca448-c6bf-4cb3-b9f3-5964a6cd9fe9",
         "damlDbPassword": "<password>"
       }
     ],
     "operatorSpecifications": {
      "operatorPublicKey": "<provide-value>"
       }
    }
   

9. Validate your deployment descriptor file semantics against the reconfigure-descriptor-v1.schema.
10. Validate the new configuration ID from the blockchain.

    ./concop scale --replicas execute --configuration NEW_CONFIGURATION_ID --restart

11. Validate that the credentials, network name, and folder name are correctly populated to avoid errors.

    ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  
    INFRA_DESC_FILENAME=infrastructure_descriptor.json 
    DEPLOY_DESC_FILENAME=scaleup_deployment_descriptor.json 
    CONFIG_SERVICE_IP=<orchestrator-ip-address>
    ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
    ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE 
    docker-compose -f docker-compose-orchestrator.yml up

    The <orchestrator-ip-address> is the VMware Blockchain Orchestrator appliance IP address running the configuration service container.

12. Run the VMware Blockchain Orchestrator reconfigure script.

    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE
    ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory>
    INFRA_DESC_FILENAME=<infra-desc-file-name>
    DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name>
    ORCHESTRATOR_OUTPUT_DIR=<output-directory>
    docker-compose -f docker-compose-castor.yml up

    The VMware Blockchain Orchestrator output file includes the session ID for the new node configuration.

13. Identify the reconfiguration ID in the /home/blockchain/output directory.

    Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb

14. Check the Replica Network health.

    sudo docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    sudo docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep fetchingState
    sudo docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView|Replica ID|Primary '
    sudo docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10