In-place-based upgrade updates all the nodes within the blockchain. These nodes are backed up and upgraded consecutively.

With an in-place-based upgrade, you cannot revert to the pre-upgraded version of the product. If there is an upgrade failure, you can use the backup data captured before the upgrade to restore your blockchain nodes.

Prerequisites

Procedure

  1. Update the current VMware Blockchain version to the new release version on all VMware Blockchain nodes.
    sed -i 's/:<Old_Version>/:<New_Version>/g' /config/agent/config.json
  2. Upgrade fluentd, jaeger-agent, Telegraf, and wavefront-proxy from release version 1.3 to 1.4 on all VMware Blockchain nodes.
    1. Upgrade fluent from release version 1.3 to 1.4 on all VMware Blockchain nodes.
      sed -i ‘s/fluentd:<New_Version>/fluentd:1.1/g’ /config/agent/config.json

      Use the new release version from step 1.

      For example, <New_Version> can be 1.4.0.1.137.
    2. Upgrade jaeger-agent and Telegraf from release version 1.3 to 1.4 on all VMware Blockchain nodes.
      sed -i ‘s/jaeger-agent:1.17/jaeger-agent:1.22/g’ /config/agent/config.json
      sed -i ‘s/telegraf:1.14.0/telegraf:1.18.3/g’ /config/agent/config.json
    3. Upgrade wavefront-proxy from release version 1.3 to 1.4 on all VMware Blockchain nodes.
      sed -i ‘s/wavefront-proxy:6.1/wavefront-proxy:10.12/g’ /config/agent/config.json
    4. If the VMware Blockchain Orchestrator 1.3 and 1.4 release version IP address are different, update the config service IP address from 1.3 to 1.4 on all VMware Blockchain nodes.
      sed -i ‘s/<Old_IP_Address>/<New_IP_Address>/g’ /config/agent/config.json

      <Old_IP_Address> is the VMware Blockchain Orchestrator 1.3 IP address. <New_IP_address> is the VMware Blockchain Orchestrator 1.4 IP address.

      Sample command.
      sed -i ‘s/10.0.0.252/10.0.0.251/g’ /config/agent/config.json
  3. Upgrade wavefront-proxy from release version 1.4 to 1.4.0.1.
    sed -i -- 's/wavefront-proxy:10.7/wavefront-proxy:10.12/g' /config/agent/config.json
  4. To upgrade from 1.3 to latest versions, edit the /config/agent/config.json file.
    1. Add the Wavefront proxy URL and token to the JSON file.
      "properties": {
        "values": {
          "PROXY_WAVEFRONT_URL": "<Wavefront-url>",
          "PROXY_WAVEFRONT_TOKEN": "<Wavefront-token>"
        }
      }
    2. Edit the secure store properties.
      "secureStore": {
        "storeType": "DISK",
        "url": "file:///config/agent/secrets/secret_key.json"
      }
  5. Sanitize the Replica node data.
    image=$(docker images --format “{{.Repository}}:{{.Tag}}” | grep “concord”);docker run -it --rm --entrypoint=“” --mount type=bind,source=/mnt/data/rocksdbdata,target=/concord/rocksdbdata $image /concord/kv_blockchain_db_editor /concord/rocksdbdata removeMetadata
  6. On each Replica node, where the data is sanitized, remove the gen* files from the concord config directory.

    rm /config/concord/config-generated/gen*

  7. (Optional) Update the VMware notary server.
    jq '.notaryServer += {"address": "https://<notary_server>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json
  8. (Optional) Update if the self-signed notary server.
    1. Get the TLS Certificate used for the self-signed notary server.
      mkdir -p ~/.docker/tls/<notary_server_host>:<port>
    2. Replace the variable value with the TLS certificate content.
      echo '<TLS_certificate_data>' > ~/.docker/tls/<notary_server_host>:<port>/ca.crt
    3. Change the mode to the new self-signed notary server.
      chmod 600 ~/.docker/tls/<notary_server_host>:<port>/ca.crt
      cp ~/.docker/tls/127.0.0.1:4443/ca.crt /config/agent/notarySelfSignedCert.crt
  9. (Optional) Update the container registry if it is self-signed.
    1. Update the container registry address to the new address.
      jq '.containerRegistry.address = {"address": "<NEW_CONTAINER_REGISTRY_ADDRESS>"}' /config/agent/config.json > config_tmp.json && mv config_tmp.json /config/agent/config.json
    2. Get the TLS Certificate used for the self-signed container registry.
      mkdir -p /etc/docker/certs.d/<container_registry_host>:<port>
    3. Replace the variable value with the TLS certificate content.
      echo '<TLS_CERTIFICATE_DATA>' > /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt
    4. Change the mode to the new self-signed notary server.
      chmod 600 /etc/docker/certs.d/<container_registry_host>:<port>/ca.crt
  10. SSH into the VMware Blockchain Orchestrator appliance.
  11. Enter the login credentials for the blockchain user account.
  12. Navigate to the /home/blockchain directory.
  13. In the VMware Blockchain Orchestrator deployment descriptor file, update the following parameters.
    • IP addresses of all the Replica and Client node VMs

    • Node ID of the Replica and Client node

    • Client group ID and group name for the Client node

    • Daml database password for each Client node

    • Blockchain ID

  14. Validate your deployment descriptor file semantics against the deployment-descriptor-v1.schema.

    Sample reconfig_descriptor.json file.

    {
        "populatedReplicas": [
          {
             "zoneName":"zone-1",
             "providedIp":"10.10.10.211",
             "nodeId":"3aa6aab4-216c-442a-aaf0-cc3268dce94a"
          },
          {
            "zoneName": "zone-1",
             "providedIp":"10.10.10.212",
             "nodeId":"3cbc8b4b-fc59-4fdd-a303-8e05e537a89e"
          },
          {
             "zoneName":"zone-1",
             "providedIp":"10.10.10.213",
             "nodeId":"7e087880-2f77-47ec-85d6-5fc6197687c3"
          },
          {
            "zoneName": "zone-1",
            "providedIp":"10.10.10.214",
            "nodeId":"bb69bd67-1593-43d7-882e-933968870402"
          },
          {
            "zoneName": "zone-1",
             "providedIp":"10.10.10.215",
             "nodeId":"beaa9a37-e50a-4b70-8fb0-c553b9db2cf1"
          }
        ],
        "replicaNodeSpec": {
            "cpuCount": 4,
            "memoryGb": 16,
            "diskSizeGb": 64
        },
        "populatedClients": [
          {
             "zoneName":"zone-2",
             "providedIp":"10.10.10.218",
             "nodeId":"5412b1bd-2a81-4972-8845-2527bad2d482",
             "clientGroupId":"34ec2b70-7bed-4a4d-b8c7-f399af4f8e8f",
             "groupName":"Group1",
             "damlDbPassword":"v_5gN-VJu5O2L5o"
          }
        ],
        "clientNodeSpec": {
            "cpuCount": 4,
            "memoryGb": 16,
            "diskSizeGb": 64
        },
        "populatedFullCopyClients": [
          {
              "accessKey": "secret",
              "bucketName": "bucket-1",
              "protocol": "HTTP",
              "secretKey": "secretkey",
              "url": "10.10.10.252:9001",
              "zoneName": "zone-3",
              "providedIp": "10.10.10.219",
              "nodeId": "425f23c1-8797-4666-9fd8-a3dfff033715"
          }
        ],
        "fullCopyClientNodeSpec": {
            "cpuCount": 4,
            "memoryGb": 16,
            "diskSizeGb": 64
        },
        "blockchain": {
            "consortiumName": "consortium-1",
            "blockchainType": "DAML",
            "blockchainId": "0aebe2f1-59e2-45cd-9e3b-28fa91990bd3"
        },
        "operatorSpecifications": {
            "operatorPublicKey": "-----BEGIN PUBLIC KEY-----\nMFYwEAYHKoZIN0tARealKeyCAQYFK4EEAAoDQgAEcv27VIzuoL8SiAyC\nzWx1+ASanXsjeRK5cj5se3a9KIAuUuVT9ircMAlN0nccAUbRa2Cuqw==\n-----END PUBLIC KEY-----\n"
        }
    }
  15. Verify that the provisioning and configuration service containers are up and running using the docker ps -a command.

    If the provisioning and configuration service containers are not running, run the following command:

    CONFIG_SERVICE_IP=<orchestrator-ip-address> docker-compose -f docker-compose-orchestrator-prereqs.yml up

    The <orchestrator-ip-address> is the VMware Blockchain Orchestrator appliance IP address running the configuration service container.

  16. Run the VMware Blockchain Orchestrator redeployment script.
    ORCHESTRATOR_DEPLOYMENT_TYPE=RECONFIGURE ORCHESTRATOR_DESCRIPTORS_DIR=<descriptors-directory> INFRA_DESC_FILENAME=<infra-desc-file-name> DEPLOY_DESC_FILENAME=<deployment-descriptor-file-name> ORCHESTRATOR_OUTPUT_DIR=<output-directory> docker-compose -f docker-compose-orchestrator.yml up

    The VMware Blockchain Orchestrator output file includes the reconfiguration ID for the new node configuration.

  17. Identify the reconfiguration ID in the <output-directory> that was created when you redeployed VMware Blockchain Orchestrator.

    After the command successfully completes, the reconfiguration ID is available in the output file within the output directory.

    Sample reconfiguration Id is, 9018a57c-1bd9-4cd5-bbbc-ca8b69e18356.

  18. Locate the old configuration session ID on all the deployed VMware Blockchain nodes.
    grep -A1 "configurationSession" /config/agent/config.json | grep -o ".\{8\}-.\{4\}-.\{4\}-.\{4\}-.\{12\}"
  19. Replace the old ID with the new ID on all the deployed VMware Blockchain nodes.
    sed -i 's/<old-id>/<new-id>/g' /config/agent/config.json
      
    rm -rf /config/agent/configDownloadMarker
  20. Remove all existing components and customized configurations from the VMware Blockchain nodes.
    Note:

    Customized configurations are lost during this upgrade process and are not recoverable.

    1. Verify that all the containers except the agent are stopped.
      docker ps -a

      If the docker ps -a command shows that some containers, with the exception agent, are still running, rerun the command or use the docker stop <container_name> command and stop the containers.

    2. Remove all existing components and customized configurations from all the deployed VMware Blockchain nodes.
      curl -X POST 127.0.0.1:8546/api/node/management?action=remove

      If the remove command does not work, use the docker rm -f <container_name> command and remove the containers.

  21. Update and start the new agents on all the deployed VMware Blockchain nodes.
    docker rm -f agent
    grep -o 'docker container.*' /var/lib/cloud/instance/user-data.txt | sed 's/:<Old version>/:<New version>/' | sh
    docker start agent
  22. Monitor the deployed VMware Blockchain nodes health and check whether new blocks are added to the Daml Ledger from the logs and metrics for about five minutes.
    docker exec -it telegraf curl -s http://concord:9891/metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep Fetching
    docker exec -it concord sh -c './concord-ctl status get replica' | grep -E 'lastStableSeqNum|curView'
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
    

    Sample output:

    root@photon [ ~ ]# docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|' -f 3,10
    3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 237, clock: 394370144936
    3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 238, clock: 394478334706
    3|ConcordCommandsHandler::addBlock, exit, updates: 4, duration: 0, block_id: 239, clock: 394580769869
    
  23. Clean up any existing older images.
    docker images | grep "<Old_Version>" | awk '{print $1 ":" $2}' | xargs docker rmi

    <Old_Version> is the version of the older images that have been replaced. For example, 1.3.0.3.83.