Perform an In-Place-Based Upgrade on AWS

The in-place-based upgrade updates all the nodes within the blockchain. These nodes are backed up and upgraded consecutively.

With an in-place-based upgrade, you cannot revert to the pre-upgraded version of the product. If there is an upgrade failure, you can use the backup data captured before the upgrade to restore your blockchain nodes.

Prerequisites

Familiarize yourself with the upgrade workflow. See Considerations for Upgrading VMware Blockchain Nodes on AWS.
Verify that you have the deployed blockchain ID information.
Familiarize yourself with the backup and restore consideration for your VMware Blockchain nodes. See VMware Blockchain Node Backup and Restore Considerations on AWS.
Verify that you have access to the latest version of VMware Blockchain.
Verify that you capture the IP addresses of all the Replica and Client node VMs and Daml database passwords. You can find the information in the VMware Blockchain Orchestrator descriptor file.
Verify that you have a backup of the Replica and Client nodes. See Back-Up Replica Nodes on AWS.
Verify that your Concord containers are stopped. Complete the steps 1-12 to stop Concord containers, see Back-Up Replica Nodes on AWS.

Procedure

Update the current VMware Blockchain version to the new release version on all VMware Blockchain nodes.
```
sudo sed -i 's/:<Old version>/:<New version>/g' /config/agent/config.json
```

Upgrade jaeger-agent, Telegraf, and Wavefront-proxy from release version 1.3 to 1.4.

sudo sed -i 's/fluentd:.*/fluentd:1.1"/g' /config/agent/config.json
sudo sed -i -- 's/jaeger-agent:1.17/jaeger-agent:1.22/g' /config/agent/config.json
sudo sed -i -- 's/telegraf:1.14.0/telegraf:1.18.3/g' /config/agent/config.json
sudo sed-i -- 's/wavefront-proxy:6.1/wavefront-proxy:10.12/g'/config/agent/config.json

Upgrade Wavefront-proxy from release version 1.4 to 1.4.0.1.

sudo sed -i -- 's/wavefront-proxy:10.7/wavefront-proxy:10.12/g' /config/agent/config.json

To upgrade from 1.3 to latest versions, edit the /config/agent/config.json file.

Add the Wavefront proxy URL and token to the JSON file.

"properties": {
  "values": {
    "PROXY_WAVEFRONT_URL": "<Wavefront-url>",
    "PROXY_WAVEFRONT_TOKEN": "<Wavefront-token>"
  }
}

Edit the secure store properties.

"secureStore": {
  "storeType": "DISK",
  "url": "file:///config/agent/secrets/secret_key.json"
}

Sanitize the Replica node data.

curl -X POST 127.0.0.1:8546/api/cleanup/concord

(Optional) Update the VMware notary server.

sudo jq '.notaryServer += {"address": "https://<notary_server>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json

(Optional) Update if the self-signed notary server.

Get the TLS Certificate used for the self-signed notary server.
```
sudo mkdir -p ~/.docker/tls/<notary_server_host>:<port>
```

Replace the variable value with the TLS certificate content.

echo '<TLS_certificate_data>' > ~/.docker/tls/<notary_server_host>:<port>/ca.crt

Change the mode to the new self-signed notary server.

sudo chmod 600 ~/.docker/tls/<notary_server_host>:<port>/ca.crt
sudo cp ~/.docker/tls/127.0.0.1:4443/ca.crt /config/agent/notarySelfSignedCert.crt

(Optional) Update the container registry if it is self-signed.

Update the container registry address to the new address.

sudo jq '.containerRegistry.address = {"address": "<NEW_CONTAINER_REGISTRY_ADDRESS>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json

Get the TLS Certificate used for the self-signed container registry.
```
sudo mkdir -p /etc/docker/certs.d/<container_registry_host>:<port>
```

Replace the variable value with the TLS certificate content.

echo '<TLS_CERTIFICATE_DATA>' > /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt

Change the mode to the new self-signed notary server.

sudo chmod 600 /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt

SSH into the VMware Blockchain Orchestrator appliance.
Enter the login credentials for the blockchain user account.
Navigate to the /home/blockchain directory.
In the VMware Blockchain Orchestrator deployment descriptor file, update the following parameters.
- IP addresses of all the Replica and Client node VMs
- Node ID of the Replica and Client node
- Client group ID and group name for the Client node
- Daml database password for each Client node
- Blockchain ID

Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.

Sample reconfig_descriptor.json file.

{
    "populatedReplicas": [
      {
         "zoneName":"zone-1",
         "providedIp":"10.10.10.211",
         "nodeId":"3aa6aab4-216c-442a-aaf0-cc3268dce94a"
      },
      {
        "zoneName": "zone-1",
         "providedIp":"10.10.10.212",
         "nodeId":"3cbc8b4b-fc59-4fdd-a303-8e05e537a89e"
      },
      {
         "zoneName":"zone-1",
         "providedIp":"10.10.10.213",
         "nodeId":"7e087880-2f77-47ec-85d6-5fc6197687c3"
      },
      {
        "zoneName": "zone-1",
        "providedIp":"10.10.10.214",
        "nodeId":"bb69bd67-1593-43d7-882e-933968870402"
      },
      {
        "zoneName": "zone-1",
         "providedIp":"10.10.10.215",
         "nodeId":"beaa9a37-e50a-4b70-8fb0-c553b9db2cf1"
      }
    ],
    "replicaNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 16,
        "diskSizeGb": 64
    },
    "populatedClients": [
      {
         "zoneName":"zone-2",
         "providedIp":"10.10.10.218",
         "nodeId":"5412b1bd-2a81-4972-8845-2527bad2d482",
         "clientGroupId":"34ec2b70-7bed-4a4d-b8c7-f399af4f8e8f",
         "groupName":"Group1",
         "damlDbPassword":"v_5gN-VJu5O2L5o"
      }
    ],
    "clientNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 16,
        "diskSizeGb": 64
    },
    "populatedFullCopyClients": [
      {
          "accessKey": "secret",
          "bucketName": "bucket-1",
          "protocol": "HTTP",
          "secretKey": "secretkey",
          "url": "10.10.10.252:9001",
          "zoneName": "zone-3",
          "providedIp": "10.10.10.219",
          "nodeId": "425f23c1-8797-4666-9fd8-a3dfff033715"
      }
    ],
    "fullCopyClientNodeSpec": {
        "cpuCount": 4,
        "memoryGb": 16,
        "diskSizeGb": 64
    },
    "blockchain": {
        "consortiumName": "consortium-1",
        "blockchainType": "DAML",
        "blockchainId": "0aebe2f1-59e2-45cd-9e3b-28fa91990bd3"
    },
    "operatorSpecifications": {
        "operatorPublicKey": "-----BEGIN PUBLIC KEY-----
          \nMFkwEwYHKoZ\n
         -----END PUBLIC KEY-----\n"
    }
}

Verify that the provisioning and configuration service containers are up and running using the sudo docker ps -a command.
If the provisioning and configuration service containers are not running, run the following command:
```
CONFIG_SERVICE_IP=<orchestrator-ip-address> 
docker-compose -f docker-compose-orchestrator-prereqs.yml up
```
The <orchestrator-ip-address> is the VMware Blockchain Orchestrator appliance IP address running the configuration service container.

Run the VMware Blockchain Orchestrator reconfigure script.

ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE 
ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory> 
INFRA_DESC_FILENAME=<infra-desc-file-name> 
DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name> 
ORCHESTRATOR_OUTPUT_DIR=<output-directory> 
docker-compose -f docker-compose-orchestrator.yml up

The VMware Blockchain Orchestrator output file includes the reconfiguration ID for the new node configuration.

Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

After the command successfully completes, the reconfiguration ID is available in the output file within the output directory.

Sample reconfiguration Id is, 9018a57c-1bd9-4cd5-bbbc-ca8b69e18356.

Locate the old configuration session ID on all the deployed VMware Blockchain nodes.

grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"

Replace the old ID with the new ID on all the deployed VMware Blockchain nodes.

sudo sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
  
sudo rm -rf /config/agent/configDownloadMarker

Remove all existing components and customized configurations from the VMware Blockchain nodes.
Note:
Customized configurations are lost during this upgrade process and are not recoverable.
```
curl -X POST 127.0.0.1:8546/api/node/management?action=remove
```

Update and start the new agents on all the deployed VMware Blockchain nodes.

sudo docker rm -f agent
grep -o 'docker container.*' /var/lib/cloud/instance/user-data.txt | sudo sed 's/:<Old version>/:<New version>/' | sh
sudo docker start agent

Monitor the deployed VMware Blockchain nodes health and check whether new blocks are added to the Daml Ledger from the logs and metrics for about five minutes.

sudo docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10

Sample output:

vmbc@photon [ ~ ]# sudo docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 237, clock: 394370144936
3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 238, clock: 394478334706
3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 239, clock: 394580769869

Start the blockchain network.

Remove the stale Docker containers.

sudo docker rm$(docker ps --filter status=exited -q)

Clean up any existing older images.
```
sudo docker images | grep "<old_image_version>" | awk '{print $1 ":" $2}' | xargs docker rmi
```
The <old_image_version> is the version of the older images that have been replaced, for example, 1.4.0.37.