Perform an In-Place-Based Upgrade

The in-place-based upgrade updates all the nodes within the blockchain. These nodes are backed up and upgraded consecutively.

With an in-place-based upgrade, you cannot revert to the pre-upgraded version of the product. If there is an upgrade failure, you can use the backup data captured before the upgrade to restore your blockchain nodes.

Prerequisites

• Familiarize yourself with the upgrade workflow. See Considerations for Upgrading VMware Blockchain Nodes.

• Verify that you have the deployed blockchain ID information.
• Familiarize yourself with the backup and restore consideration for your VMware Blockchain nodes. See VMware Blockchain Node Backup and Restore Considerations on vSphere.
• Verify that you have access to the latest version of VMware Blockchain.

• Verify that you capture the IP addresses of all the Replica and Client node VMs and DAML database passwords. You can find the information in the VMware Blockchain Orchestrator descriptor file.

• Verify that you have a backup of the Replica and Client nodes. See Back-Up VMware Blockchain Nodes.

Procedure

1. Update the current VMware Blockchain version to the new release version.

   sed -i 's/:<Old version>/:<New version>/g' /config/agent/config.json

2. Sanitize the Replica node data.

   docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <ImageName> /concord/kv_blockchain_db_editor /concord/rocksdbdata removeMetadata
   rm /config/concord/config-generated/genSec_*

3. (Optional) Update the VMware notary server.

   jq '.notaryServer += {"address": "https://<notary_server>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json

4. (Optional) Update if the self-signed notary server.
   1. Get the TLS Certificate used for the self-signed notary server.

      mkdir -p ~/.docker/tls/<notary_server_host>:<port>

   2. Replace the variable value with the TLS certificate content.

      echo '<TLS_certificate_data>' > ~/.docker/tls/<notary_server_host>:<port>/ca.crt

   3. Change the mode to the new self-signed notary server.

      chmod 600 ~/.docker/tls/<notary_server_host>:<port>/ca.crt
      cp ~/.docker/tls/127.0.0.1:4443/ca.crt /config/agent/notarySelfSignedCert.crt

5. (Optional) Update the container registry if it is self-signed.
   1. Update the container registry address to the new address.

      jq '.containerRegistry.address = {"address": "<NEW_CONTAINER_REGISTRY_ADDRESS>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json

   2. Get the TLS Certificate used for the self-signed container registry.

      mkdir -p /etc/docker/certs.d/<container_registry_host>:<port>

   3. Replace the variable value with the TLS certificate content.

      echo '<TLS_CERTIFICATE_DATA>' > /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt

   4. Change the mode to the new self-signed notary server.

      chmod 600 /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt

6. SSH into the VMware Blockchain Orchestrator appliance.
7. Enter the login credentials for the blockchain user account.
8. Navigate to the /home/blockchain directory.
9. In the VMware Blockchain Orchestrator deployment descriptor file, update the following parameters.

   • IP addresses of all the Replica and Client node VMs

   • Node ID of the Replica and Client node

   • Client group ID and group name for the Client node

   • DAML database password

   • Blockchain ID

10. Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.
11. Verify that the provisioning and configuration service containers are up and running using the docker ps -a command.

    If the provisioning and configuration service containers are not running, run the following command:

    CONFIG_SERVICE_IP=<orchestrator-ip-address> docker-compose -f docker-compose-orchestrator-prereqs.yml up

    The <orchestrator-ip-address> is the VMware Blockchain Orchestrator appliance IP address running the configuration service container.

12. Run the VMware Blockchain Orchestrator redeployment script.

    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory> INFRA_DESC_FILENAME=<infra-desc-file-name> DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name> ORCHESTRATOR_OUTPUT_DIR=<output-directory> docker-compose -f docker-compose-orchestrator.yml up

    The VMware Blockchain Orchestrator output file includes the reconfiguration ID for the new node configuration.

13. Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

    Reconfiguration Id: 9018a57c-1bd9-4cd5-bbbc-ca8b69e18356

14. Locate the old configuration session ID on all the deployed VMware Blockchain nodes.

    grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"

15. Replace the old ID with the new ID on all the deployed VMware Blockchain nodes.

    sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
      
    rm -rf /config/agent/configDownloadMarker

16. Remove all existing components and customized configurations from the VMware Blockchain nodes.
    Note:

    Customized configurations are lost during this upgrade process and are not recoverable.

    curl -X POST 127.0.0.1:8546/api/node/management?action=remove

17. Update and start the new agents on all the deployed VMware Blockchain nodes.

    docker rm -f agent
    grep -o 'docker.*&' /var/lib/cloud/instance/user-data.txt | tr -d '&' | sed 's/:<Old version>/:<New version>/' | sh

18. Monitor the deployed VMware Blockchain nodes health and check whether new blocks are added to the DAML Ledger from the logs and metrics for about five minutes.

    docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep Fetching
    docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView'
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
    

19. Clean up any existing older images.

    docker images | grep "<old_image_version>" | awk '{print $1 ":" $2}' | xargs docker rmi

    The <old_image_version> is the version of the older images that have been replaced, for example, 1.0.0.37.