Scale Down Deployed VMware Blockchain Nodes on AWS

With the scale-down operation, you can remove the Replica and Client nodes from the existing VMware Blockchain topology when they are not needed.

The supported node configuration for scaling down is from n=10(f=3) to n=7(f=2) or n=4 (f=1). The Concord operator container must generate the new configuration for the selected nodes to scale down the number of nodes in a Replica Network. The new reconfiguration contains IP addresses, public keys, and certificates for the remaining Replica and Client nodes.

The scale down process is not successful under the following scenarios:

The TLS keys have changed
Replica node VMs have moved to a different VM

Prerequisites

Determine whether you need to scale down your node configuration. See VMware Blockchain Node Scaling Operations on AWS.
Verify that you have at least 7 nodes deployed. See Deploy VMware Blockchain Nodes Using VMware Blockchain Orchestrator on AWS.
Identify the following details from the VMware Blockchain Orchestrator output directory, /home/blockchain/output.
blockchain ID and Blockchain type
Replica node zone name, Replica node IP address, and Replica node ID
Client node zone name, Client node IP address, Client node ID, Client node Daml database password, and Client node group name and group ID name
Verify that all the deployed nodes are functioning properly. Failed nodes cannot be scaled down.
Verify that you have updated the Daml index database password for the Client nodes. See Update Daml Index Database Password on the Scaled Client Node on AWS.
If the Concord operator containers were deployed, verify that the Concord operator container is running. See Instantiate the Concord Operator Container for AWS.

Procedure

SSH into the VMware Blockchain Orchestrator appliance.
Enter the login credentials for the blockchain user account.
Navigate to the /home/blockchain directory.
Create a reconfigure scale down descriptor JSON file and set the parameter values in the descriptor directory.
Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.

Validate that the credentials, network name, and folder name are correctly populated to avoid errors.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors  
INFRA_DESC_FILENAME=infrastructure_descriptor.json 
DEPLOY_DESC_FILENAME=scaledown_deployment_descriptor.json 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=VALIDATE 
docker-compose -f docker-compose-orchestrator.yml up

Encrypt and redirect the infrastructure and the deployment descriptor files for added security.
1. Encrypt the infrastructure_descriptor.json file.
```
$HOME/descriptors > ansible-vault encrypt infrastructure_descriptor.json
New Vault password:
Confirm New Vault password:
Encryption successful
```
2. Encrypt the deployment_descriptor.json file.
```
$HOME/descriptors > ansible-vault encrypt deployment_descriptor.json
New Vault password:
Confirm New Vault password:
Encryption successful
```
3. Configure the two environment variable values.
  - ORCHESTRATOR_OUTPUT_DIR - The output directory where the output file is written.
  - ORCHESTRATOR_DEPLOYMENT_TYPE - Set deployment type to RECONFIGURE.
4. Run the secure-orchestrator.sh script from the orchestrator_runtime directory.
```
ORCHESTRATOR_OUTPUT_DIR=$HOME/output 
ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
./secure-orchestrator.sh
```
  The script creates temporary files.
  - /dev/shm/orchestrator-awsIGoa0JA/infra_descriptor
  - /dev/shm/orchestrator-awsIGoa0JA/deployment_descriptor
5. Redirect the decrypted infrastructure_descriptor.json to the infrastructure_descriptor file location.
  Use the vault password used to encrypt the infrastructure_descriptor.json file.
```
ansible-vault view $HOME/descriptors/infrastructure_descriptor.json > /dev/shm/orchestrator-awsIGoa0JA/infra_descriptor
```
6. Redirect the decrypted deployment_descriptor.json to the deployment_descriptor file location.
  Use the vault password used to encrypt the deployment_descriptor.json file.
```
ansible-vault view $HOME/descriptors/deployment_descriptor.json > /dev/shm/orchestrator-awsIGoa0JA/deployment_descriptor
```
  After the script completes running, the temporary files are deleted.
7. (Optional) If the script fails or the secure_orchestrator.sh script is terminated, delete the temporary folder under the /dev/shm/orchestrator-* directory.

Run the VMware Blockchain Orchestrator reconfigure script.

ORCHESTRATOR_DESCRIPTORS_DIR=/home/blockchain/descriptors 
ORCHESTRATOR_OUTPUT_DIR=/home/blockchain/output 
INFRA_DESC_FILENAME=infrastructure_descriptor.json
DEPLOY_DESC_FILENAME=scaledown_deployment_descriptor.json  
ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
docker-compose -f docker-compose-orchestrator.yml up

The VMware Blockchain Orchestrator output file includes the session ID for the new node configuration.

Stop all the applications that invoke the connection requests to the Daml Ledger in the original deployment.

Stop the Client nodes.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop

Identify the reconfiguration ID in the /home/blockchain/output directory.

Reconfiguration Id: 626fdcfc-9c9c-4c25-b210-9e71c471e3cb
Replace the old configuration ID with the reconfiguration ID in the /config/agent/config.json file.

The command stops the Replica nodes and downloads the new configuration on the Replica nodes.

./concop network-reconf addRemove-execute --configuration NEW_CONFIGURATION_ID \
                                          --auto \     #get the new configuration
                                          --restart \  #restart the replicas automatically
                                       #  --bft     \   # if you need n-f reconfiguration 
                                       #  --tokens   "{ \"node1_id\" : \"token\", \"node2_id\" : \"token\", ...}"

Stop and shut down the Replica nodes removed from the blockchain topology.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop
docker stop agent
sudo shutdown -h

Shut down the agent on the Client nodes removed from the blockchain topology.
```
docker stop agent
sudo shutdown -h
```

Start the remaining Client nodes and agents.

curl -X POST 127.0.0.1:8546/api/node/management?action=remove
docker restart agent

Download the new configuration on the Client nodes.

curl -ik -X POST http://127.0.0.0.1:8546/api/node/reconfigure/<new-config-ID>

Start the Client node containers.
```
curl -ik -X POST -H "content-type: application/json" --data '{ "containerNames" : ["all"] }' http://localhost:8546/api/node/restart
```
If the containers do not start, use the start command curl -X POST 127.0.0.1:8546/api/node/management?action=start.