Replica pruning is a two-step process. The operator container collects the latest pruneable block IDs from all Replica nodes. Then the operator container creates a includes the prune command that includes a list of all the latest pruneable block IDs, signs the prune command, and sends it to the Concord container. Upon receiving the prune command, all the Replica nodes verify the list of the latest pruneable block IDs and prune up to the minimal pruneable block ID.

Note:

Data pruning is irreversible. After the data is pruned from the Replica nodes, the data is no longer available to the Client nodes. As a best practice, back up the Replica nodes before you initiate a pruning process.

Pruning is enabled by default. The default number of blocks to keep is deactivated, and keeping the blocks is set to two weeks. The default Concord container pruning parameter values are as follows:

  • pruning_enabled: true

  • pruning_duration_to_keep_minutes: 20160

  • pruning_num_blocks_to_keep: 0

Prerequisites

Procedure

  1. (Optional) If the default pruning values need updates, you can edit the pruning configuration on all the Replica nodes.
    1. Verify the current pruning parameter values.
      docker exec -it concord cat /concord/config/application.config | grep -iE 'pruning_'
      pruning_duration_to_keep_minutes: 20160
      pruning_enabled: true 
      pruning_num_blocks_to_keep: 0
      Note:

      Either the default number of blocks or the duration to keep the blocks must be set to 0. Otherwise, you cannot accurately predict how many blocks are going to be pruned.

    2. Run the configuration command to change the parameter values.
      docker exec -it concord sed -i's/<parameter-name>: <original_value>/<parameter-name>:<new_value>/g'/concord/config/application.config

      For example, you can change the duration time to keep stale key values from two weeks to one week.

      docker exec -it concord sed -i's/pruning_duration_to_keep_minutes: 20160/pruning_duration_to_keep_minutes:10080/g' /concord/config/application.config
    3. Restart the Concord container for the configuration command changes to take effect.

      docker restart concord

  2. Verify the Replica nodes status.
    docker exec -it telegraph curl -s http://concord:9891 metrics | grep -ia last_block | tail -1
    docker exec -it concord sh -c './concord-ctl status get state-transfer' | grep Fetching
    docker exec -it concord sh -c './concord-ctl status replica' | grep -E 'lastStableSeqNum|curView'
    docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|" -f 3,10
  3. Restart the Concord containers on all the Replica nodes.

    docker restart concord

  4. Identify the latest block for pruning in each Replica node.

    If a Replica node is lagging, wait for the node to catch up using state transfer../concop prune latestPruneableBlock

    The output contains the Replica node block_id and signature.

  5. Stop containers on all the Client nodes.

    docker stop daml_ledger_api daml_index_db

  6. Initiate the pruning process.

    docker exec -it operator sh -c './concop prune execute'

  7. Check whether the pruning process is successful on all the Replica nodes.

    docker exec -it operator sh -c './concop prune status'

    The output for each Replica node contains in_progress and last_pruned_block.

  8. Log in to Wavefront.
  9. Verify the Merkle keys, versioned keys, and immutable keys metrics for each block.

    For each category of keys, the metrics report shows the total number of stale and pruned keys. The metrics indicate that after a successful pruning process, the number of pruned keys has increased, and the number of stale keys has decreased.

  10. Restart containers on the Client node used for operator commands.
    docker restart daml_index_db
    docker restart daml_ledger_api

Results

When the pruning process is successful, you can see the following changes in the system:

  • The genesis block changes to a larger value.

  • RocksDB disk size decreases after some time.

  • Keys related to active smart contracts are not affected regardless of their age.

What to do next

You can prune outdated data on the Client node. See Pruning Client Nodes.