Pruning Replica and Full Copy Client Nodes on AWS

Replica pruning occurs after the blocks are safely stored in the Full Copy Client. The pruning operation involves two steps. First, collect the latest pruneable block IDs from all the Replica and the Full Copy Client nodes. Secondly, create the prune command, which includes a list of the latest pruneable block IDs.

Upon receiving the prune command, all the Replica nodes verify the list of the latest pruneable block IDs and prune up to the minimal pruneable block ID.

The Full Copy Client, which contains the blockchain data starting from the genesis block, is not pruned.

Note:

Data pruning on Replica nodes is irreversible. After the data is pruned from the Replica nodes, the data is no longer available to the Client nodes. As a best practice, back up the Replica nodes before initiating a pruning process.

Pruning is enabled by default. You can configure pruning using block numbers to keep or a time parameter. The default configuration is to keep the data from the last two weeks and prune the rest.

The default Concord container pruning parameter values are as follows:

pruning_enabled: true
pruning_duration_to_keep_minutes: 20160
pruning_num_blocks_to_keep: 0

You can edit the pruning parameters on each Replica node using the command docker exec -it concord /bin/bash and vi /concord/config/application.config.

Prerequisites

Familiarize yourself with the impact of the pruning operation. See Pruning VMware Blockchain Nodes on AWS.
Verify that you have identified and backed up the outdated data to be pruned. See Back-Up Replica Nodes on AWS.
Verify that the Client node has an instantiated operator container. See Instantiate the Concord Operator Container for AWS.
Verify that all the Replica and Full Copy Client nodes are caught up not to lose data during pruning.
Verify that you stop all the applications invoking requests to the Client nodes using a Daml Ledger API interface.

Procedure

(Optional) Start the operator container.

image=$(docker images --format "{{.Repository}}:{{.Tag}}"| grep "operator");
docker run -d --name=operator -v /config/daml-ledger-api/concord-operator:/operator/config-local -v /config/daml-ledger-api/concord-operator:/concord/config-public -v /config/daml-ledger-api/config-local/cert:/config/daml-ledger-api/config-local/cert -v /config/daml-ledger-api/config-public:/operator/config-public$image

(Optional) Verify the current pruning parameter values.
```
docker exec-it concord cat/concord/config/application.config | grep
-iE 'pruning_'
```
The default number of blocks or the duration to keep the blocks must be set to 0.

(Optional) Run the configuration command to change the parameter values.

docker exec-it concord sed -i 's/<parameter>: <original value>/<parameter>: <new value>/g' /concord/config/application.config

For example, you can change the duration time to keep stale key values from two weeks to one week.

docker exec-it concord sed -i 's/pruning_duration_to_keep_minutes: 20160/pruning_duration_to_keep_minutes: 10080/g' /concord/config/application.config

(Optional) Start all the Replica nodes for the new pruning parameters to take effect.

curl -X POST 127.0.0.1:8546/api/node/management?action=start

Verify the Replica nodes status.

docker exec -it telegraf curl -s http://concord:9891/metrics| grep -ia last_block | tail -1
docker exec -it concord sh -c './concord-ctl status get state-transfer'| grep fetchingState
docker exec -it concord sh -c './concord-ctl status get replica'| grep -ia 'lastStableSeqNum'
docker logs --since 1m -f concord | grep -ia addBlock | cut -d '|'-f 3,10

Identify the latest block for pruning in each Replica or Full Copy Client node.
If a Replica node is lagging, it shows a smaller large pruneable block than the rest. Wait for the node to catch up using state transfer or data restore.
```
docker exec -it operator sh -c './concop prune latestPruneableBlock'
```
The output contains the Replica or Full Copy Client node block_id and signature.
Stop containers except for the agent and operator on all the Client nodes.

curl -X POST 127.0.0.1:8546/api/node/management?action=stop

Initiate the pruning process.

docker exec-it operator sh -c './concop prune start'

Check whether the pruning process is successful on all the Replica nodes.
```
docker exec-it operator sh -c './concop prune status'
```
The output for each Replica node contains in_progress and last_pruned_block.

Start all the containers on each Client node.

curl -X POST 127.0.0.1:8546/api/node/management?action=start

Results

When the pruning process is successful, you can see the following changes in the system:

The genesis block changes to a larger value.
RocksDB disk size decreases after some time.
Verify that the disk compaction on the Concord container is successful. Run the following command on all the Replica nodes.
```
sudo egrep -i 'compact' `docker inspect --format={{.LogPath}} concord
```
Keys related to active smart contracts are not affected regardless of their age.

What to do next

You can prune outdated data on the Client node. See Pruning Client Nodes on AWS.