Replica pruning occurs after the blocks are safely stored in the Full Copy Client. The pruning operation involves two steps. Collecting the latest pruneable block IDs from all the Replica and the Full Copy Client nodes. Secondly, creating the prune command, which includes a list of the latest pruneable block IDs.

Upon receiving the prune command, all the Replica nodes verify the list of the latest pruneable block IDs and prune up to the minimal pruneable block ID.

The Full Copy Client, which contains the blockchain data starting from the genesis block, does not get pruned.

Note:

Data pruning on Replica nodes is irreversible. After the data is pruned from the Replica nodes, the data is no longer available to the Client nodes. As a best practice, back up the Replica nodes before you initiate a pruning process.

Pruning is enabled by default. You can configure pruning using block numbers to keep or a time parameter. The default configuration is to keep the data from the last two weeks and prune the rest.

The default Concord container pruning parameter values are as follows:

  • pruning_enabled: true

  • pruning_duration_to_keep_minutes: 20160

  • pruning_num_blocks_to_keep: 0

Prerequisites

  • Verify that you have identified and backed up the outdated data to be pruned. See Back-Up VMware Blockchain Nodes.

  • Verify that the Client node has an instantiated operator container. See Bind the Newly Deployed VMware Blockchain Nodes.

  • Verify that all the Replica and Full Copy Client nodes are caught up so that data is not lost during pruning.

  • Verify that you stop all the applications invoking requests to the Client nodes using a DAML Ledger API interface.

  • Familiarize yourself with the impact of the pruning operation. See Pruning VMware Blockchain Nodes.

Procedure

  1. (Optional) To update the default pruning values, you can edit the pruning configuration on all the Replica nodes.
    1. Verify the current pruning parameter values.
      docker exec -it concord cat /concord/config/application.config | grep -iE 'pruning_'
      pruning_duration_to_keep_minutes: 20160
      pruning_enabled: true 
      pruning_num_blocks_to_keep: 0
      Note:

      Either the default number of blocks or the duration to keep the blocks must be set to 0. Otherwise, you cannot accurately predict how many blocks are going to be pruned.

    2. Run the configuration command to change the parameter values.
      sed -i 's/<parameter-name>: <original_value>/<parameter-name>: <new_value>/g' $(find / -name application.config -type f)

      For example, you can change the duration time to keep stale key values from two weeks to 15 minutes.

      sed -i 's/pruning_duration_to_keep_minutes: 20160/pruning_duration_to_keep_minutes: 15/g' $(find / -name application.config -type f)
    3. Restart the Concord container for the configuration command changes to take effect.

      docker restart concord

  2. Verify the Replica nodes' status.
    docker exec -it telegraf curl -s http://concord:9891 metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep Fetching
    docker exec -it concord sh -c './concord-ctl status replica' | grep -E 'lastStableSeqNum|curView'
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d "|" -f 3,10
  3. Identify the latest block for pruning in each Replica or Full Copy Client node.

    If a Replica node is lagging, wait for the node to catch up using state transfer and execute the command on the Client node with the operator.

    docker exec -it operator sh -c "./concop prune latestPruneableBlock"

    The output contains the Replica or Full Copy Client node block_id and signature.

  4. Stop containers on all the Client nodes.

    docker stop daml_ledger_api daml_index_db

  5. Initiate the pruning process.

    docker exec -it operator sh -c './concop prune execute'

  6. Check whether the pruning process is successful on all the Replica nodes.

    docker exec -it operator sh -c './concop prune status'

    The output for each Replica node contains in_progress and last_pruned_block.

  7. Log in to Wavefront.
  8. Verify the Merkle keys, versioned keys, and immutable keys metrics for each block.

    For each category of keys, the metrics report shows the total number of stale and pruned keys. The metrics indicate that the number of pruned keys has increased after a successful pruning process, and the number of stale keys has decreased.

  9. Restart containers on the Client node used for operator commands.
    docker restart daml_index_db
    docker restart daml_ledger_api

Results

When the pruning process is successful, you can see the following changes in the system:

  • The genesis block changes to a larger value.

  • RocksDB disk size decreases after some time.

  • Keys related to active smart contracts are not affected regardless of their age.

What to do next

You can prune outdated data on the Client node. See Pruning Client Nodes.