Scale Down Deployed VMware Blockchain Nodes on vSphere

With the scale-down operation, you can remove the Replica and Client nodes from the existing VMware Blockchain topology when they are not needed.

The supported node configuration for scaling down is from n=10(f=3) to n=7(f=2) or n=4 (f=1). The Concord operator container must generate the new configuration for the selected nodes to scale down the number of nodes in a Replica Network. The new reconfiguration contains IP addresses, public keys, and certificates for the remaining Replica and Client nodes.

The scale down process is not successful under the following scenarios:

The TLS keys have changed
Replica node VMs have moved to a different VM

Prerequisites

Determine whether you need to scale down your node configuration. See VMware Blockchain Node Scaling Operations on vSphere.
Verify that you have at least 7 nodes deployed. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator on vSphere.
Identify the following details from the VMware Blockchain Orchestrator output directory, /home/blockchain/output.
blockchain ID , consortium ID, and Blockchain type
Replica node zone name, Replica node IP address, and Replica node ID
Client node zone name, Client node IP address, Client node ID, Client node Daml database password, and Client node group name and group ID name
Verify that all the deployed nodes are functioning properly. Failed nodes cannot be scaled down.
Verify that you have updated the Daml index database password for the Client nodes. See Update Daml Index Database Password on the Scaled Client Node on vSphere.

Procedure

SSH into the VMware Blockchain Orchestrator appliance.
Enter the login credentials for the blockchain user account.
Navigate to the /home/blockchain directory.

Create a reconfigure scale down descriptor JSON file and set the parameter values in the descriptor directory.

Sample scaledown_deployment_descriptor.json file to scale down Replica nodes.

{
  "populatedReplicas": [
    {
      "zoneName": "replica",
      "providedIp": "10.72.216.178",
      "nodeId": "51385b16-fc84-4906-88c6-7de7df90fa4e"
    },
    {
      "zoneName": "replica",
      "providedIp": "10.72.216.180",
      "nodeId": "63f92ed8-943c-4839-b856-d30d3c194cba"
    },
    {
      "zoneName": "replica",
      "providedIp": "10.72.216.176",
      "nodeId": "75ce52ab-d039-4068-afa6-b8832a309bba"
    },
    {
      "zoneName": "replica",
      "providedIp": "10.72.216.192",
      "nodeId": "765ccae4-7db7-425b-80de-ca8373cc19ba"
    }
  ],
  "populatedClients": [
    {
      "zoneName": "client",
      "groupName": "Group2",
      "providedIp": "10.72.216.89",
      "clientGroupId" : "UUID2",
      "nodeId": "460bfef1-fc54-4148-ba4a-4c51a2fb1aea",
      "damlDbPassword": "mypass"
    },
    {
      "zoneName": "client",
      "groupName": "Group1",
      "providedIp": "10.72.216.181",
      "clientGroupId" : "UUID2",
      "nodeId": "55ff6458-8056-4ed6-9f73-4cf3f8410ddf",
      "damlDbPassword": "mypass"
    },
    {
      "zoneName": "client",
      "groupName": "Group1",
      "providedIp": "10.72.216.193",
      "clientGroupId" : "UUID2",
      "nodeId": "242113c2-89c9-4dd0-a6e4-d18d63dcf1b6",
      "damlDbPassword": "mypass"
    }
  ],
  "clientNodeSpec": {
    "cpuCount": 4,
    "memoryGb": 32,
    "diskSizeGb": 100
  },
  "blockchain": {
    "consortiumName": "consortium-1",
    "blockchainType": "DAML",
    "blockchainId": "c9106eba-ca3c-4bba-bba1-cb3e8b3e65db"
  }
}

Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.

Validate that the credentials, network name, and folder name are correctly populated to avoid errors.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  
INFRA_DESC_FILENAME=infrastructure_descriptor.json 
DEPLOY_DESC_FILENAME=scaledown_deployment_descriptor.json 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE 
docker-compose -f docker-compose-orchestrator.yml up

Run the VMware Blockchain Orchestrator reconfigure script.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
INFRA_DESC_FILENAME=infrastructure_descriptor.json
DEPLOY_DESC_FILENAME=scaledown_deployment_descriptor.json  
ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
docker-compose -f docker-compose-orchestrator.yml up

The VMware Blockchain Orchestrator output file includes the session ID for the new node configuration.

Stop all the applications that invoke the connection requests to the Daml Ledger in the original deployment.

Stop the Client nodes.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop

Identify the reconfiguration ID in the /home/blockchain/output directory.

Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb
Replace the old configuration ID with the reconfiguration ID in the /config/agent/config.json file.
Instantiate the Concord operator container.

You can only use one Client node, which is part of the original blockchain, as an Concord operator container.
1. SSH into a Client node designated to host the operator Client node.
  
  Note:
  Reboot the Concord container before using an operator container from a different Client node.
2. Verify that the Client node has a Concord operator container image and identify the operator image ID.
  
  sudo docker images | grep "operator"
3. Verify that the Concord operator container configuration file is available.
```
sudo cat /config/daml-ledger-api/concord-operator/operator.config
```
4. For deployments with unencrypted configuration, copy the private key content into the following location.
```
sudo vi /config/daml-ledger-api/concord-operator/operator_priv.pem
```
5. For deployments with encrypted configuration, deploy the encrypted operator private key content into the following location.
  The private key is encrypted and stored on disk because the deployment secure store configuration requires private keys not to be saved unprotected on disk.
```
# Replace <OP_IMAGE_ID> and paste the operator private key when prompted
sudo docker run -ti --network=blockchain-fabric --name=operator --entrypoint /operator/install_private_key.py --rm -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/clientservice/cert:/config/clientservice/cert -v /config/daml-ledger-api/config-public:/operator/config-public <Operator_IMAGE_ID>
```
6. Launch the Concord operator container Client node.
```
curl -X POST 127.0.0.1:8546/api/node/start-operator
```
7. Access the Concord operator container Client node.
```
sudo docker exec -it operator bash
```

The command stops the Replica nodes and downloads the new configuration on the Replica nodes.

./concop network-reconf addRemove-execute --configuration NEW_CONFIGURATION_ID \
                                          --auto \     #get the new configuration
                                          --restart \  #restart the replicas automatically
                                       #  --bft     \   # if you need n-f reconfiguration 
                                       #  --tokens   "{ \"node1_id\" : \"token\", \"node2_id\" : \"token\", ...}"

Stop and shut down the Replica nodes removed from the blockchain topology.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop
sudo docker stop agent
sudo shutdown -h

Shut down the agent on the Client nodes removed from the blockchain topology.
```
sudo docker stop agent
sudo shutdown -h
```

Start the remaining Client nodes and agents.

curl -X POST 127.0.0.1:8546/api/node/management?action=remove
sudo docker restart agent

Download the new configuration on the Client nodes.

curl -ik -X POST http://127.0.0.0.1:8546/api/node/reconfigure/<new-config-ID>

Start the Client node containers.
```
curl -ik -X POST -H "content-type: application/json" --data '{ "containerNames" : ["all"] }' http://localhost:8546/api/node/restart
```
If the containers do not start, use the start command curl -X POST 127.0.0.1:8546/api/node/management?action=start.