The in-place upgrade updates all the nodes within the blockchain. These nodes are backed up and upgraded consecutively.

With an in-place upgrade, you cannot revert to the pre-upgraded version of the product. If there is an upgrade failure, you can use the backup data that was captured before the upgrade to restore your blockchain nodes.

Prerequisites

  • Familiarize yourself with the upgrade workflow. See Considerations for Upgrading VMware Blockchain Nodes.

  • Verify that you have the deployed blockchain ID information.
  • Verify that you have access to the latest version of VMware Blockchain.
  • Verify that you capture the IP addresses of all the Replica and Client node VMs and DAML database passwords. You can find the information in the VMware Blockchain Orchestrator descriptor file.

  • Verify that you have a backup of the Replica and Client nodes. See Back-Up the Client and Replica Nodes.

Procedure

  1. Update the current VMware Blockchainversion to the new release version.
    sed -i 's/:<Old version>/:<New version>/g' /config/agent/config.json
  2. Sanitize the Replica node data.
    docker run -it --rm --entrypoint="" --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata <ImageName> /concord/sparse_merkle_db_editor /concord/rocksdbdata removeMetadata
    rm /config/concord/config-generated/genSec_*
  3. (Optional) Update the VMware notary server.
    jq '.notaryServer += {"address": "https://<notary_server>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json
  4. (Optional) Update if the self-signed notary server.
    1. Get the TLS Certificate used for the self-signed notary server.
      mkdir -p ~/.docker/tls/<notary_server_host>:<port>
    2. Replace the variable value with the TLS certificate content.
      echo '<TLS_certificate_data>' > ~/.docker/tls/<notary_server_host>:<port>/ca.crt
    3. Change the mode to the new self-signed notary server.
      chmod 600 ~/.docker/tls/<notary_server_host>:<port>/ca.crt
      cp ~/.docker/tls/127.0.0.1:4443/ca.crt /config/agent/notarySelfSignedCert.crt
  5. (Optional) Update the container registry, if it is self-signed.
    1. Update the container registry address to the new address.
      jq '.containerRegistry.address = {"address": "<NEW_CONTAINER_REGISTRY_ADDRESS>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json
    2. Get the TLS Certificate used for the self-signed container registry.
      mkdir -p /etc/docker/certs.d/<container_registry_host>:<port>
    3. Replace the variable value with the TLS certificate content.
      echo '<TLS_CERTIFICATE_DATA>' > /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt
    4. Change the mode to the new self-signed notary server.
      chmod 600 /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt
  6. SSH into the VMware Blockchain Orchestrator appliance.
  7. Enter the login credentials for the blockchain user account.
  8. Navigate to the /home/blockchain directory.
  9. In the VMware Blockchain Orchestrator deployment descriptor file, update the following parameters.
    • IP addresses of all the Replica and Client node VMs

    • Node ID of the Replica and Client node

    • Client group ID and group name for the Client node

    • DAML database password

    • Blockchain ID

  10. Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.
  11. Verify that the provisioning and configuration service containers are up and running using the docker ps command.

    If the provisioning and configuration service containers are not running, run the following command:

    CONFIG_SERVICE_IP=<host-ip-address> docker-compose -f docker-compose-orchestrator-prereqs.yml up
  12. Run the VMware Blockchain Orchestrator redeployment script.
    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory> INFRA_DESC_FILENAME=<infra-desc-file-name> DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name> ORCHESTRATOR_OUTPUT_DIR=<output-directory> docker-compose -f docker-compose-orchestrator.yml up

    The <host-ip-address> is the appliance IP address running the containers.

  13. Identify the reconfiguration ID in the /home/blockchain/output directory.

    Reconfiguration Id: 9018a57c-1bd9-4cd5-bbbc-ca8b69e18356

  14. Locate the old configuration session ID on all the deployed VMware Blockchain nodes.
    grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"
  15. Replace the old ID with the new ID on all the deployed VMware Blockchain nodes.
    sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
      
    rm -rf /config/agent/configDownloadMarker
  16. Remove all existing components and customized configurations from the VMware Blockchain nodes.
    Note:

    Customized configurations are lost during this upgrade process and are not recoverable.

    curl -X POST 127.0.0.1:8546/api/node/management?action=remove
  17. Update and start the new agents on all the deployed VMware Blockchain nodes.
    docker rm -f agent
    grep -o 'docker.*&' /var/lib/cloud/instance/user-data.txt | tr -d '&' | sed 's/:<Old version>/:<New version>/' | sh
  18. Monitor the deployed VMware Blockchain nodes health and check whether new blocks are added to the DAML Ledger from the logs and metrics for about five minutes.
    docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep Fetching
    docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView'
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
    
  19. Clean up any existing older images.
    docker images | grep "<old_image_version>" | awk '{print $1 ":" $2}' | xargs docker rmi

    The <old_image_version> is the version of the older images that have been replaced. For example, 1.0.0.37.