The log parser tool simplifies debugging errors by analyzing logs and summarizing different modules.
You can peruse this summary to determine the error zone and evaluate the root cause. In some instances, you can identify potential root cause zones without knowing the exact technical root cause.
Before the availability of the log parser tool, users manually examined logs from every Replica node to find the Replica node with the error.
The log parser tool contains the log parser agent and log parser appliance. You must enable the ENABLE_SIDE_CAR_VM_DEPLOYMENT parameter in the infrastructure and deployment descriptor files before deploying the blockchain nodes. See Configuring the Infrastructure Descriptor Parameters on AWS and Configuring the Deployment Descriptor Parameters on AWS.
The log parser agent is a Docker container deployed on each blockchain node. The log parser agent sends the compressed logs to the log parser appliance at certain intervals. Log preprocessing is not performed at the agent level to avoid performance latency. The log parser agent retrieves logs from blockchain nodes and requires access to all the blockchain nodes.
The log parser appliance is deployed with the blockchain nodes. The log parser appliance container processes the logs at a certain interval and generates the log summary, which is grouped based on the time stamp. After the log summary is stored, the logs are retained in the /var/lib/docker/containers/retained_logs/ path and differentiated using a time stamp.
The log parser tool can parse up to 800MB of compressed logs or 10GB of logs.
You cannot enable or disable the log parser tool after deploying the blockchain nodes.
You cannot use the log parser tool for scaled up and down blockchain nodes.
- Disable the log parser appliance and log parser agent container.
docker stop <container Name>
- Remove all the containers that have exited
docker rm $(docker ps --filter status=exited -q)
- Remove the older log files from the sideCar VM for every 4 hours, if you are performing large transactions to avoid issues related to volume.
- Analyze the logs to identify the error and proceed to troubleshoot the problem.
The parser summary is stored in the side car node under the /var/lib/docker/containers/parser_results folder grouped by time stamp.
Log Parser Summary File Type Description
These logs show all the post-execution requests, failures, and successes.
Number of Requests processed: 3306
Number of conflicts in post-execution: 0
Number of success requests in post-execution: 3306
These logs include path change and switch sequence numbers for each Replica node. The logs indicate that all Replica nodes are in consensus with the same Slow/Fast path and sequence number.
Sample consensus_parser number - 1
path change : OPTIMISTIC_FAST to SLOW
time : 2021-03-12 12:44:57.593000
switch sequence number : 64
primary : 0
duration : 0:00:07.224000
count of request downgraded to slow: supposed to be fast : 29
The log shows the pruning operation details for each Replica node. The log is for information purposes only and not for troubleshooting.
Sample for the Prune ID: 1
Replica Name: concord2_1
Number of deleted blocks: 1302
Last deleted block id: 1302
Pruning Started at: 2021-05-11T11:14:24,539Z
Prune Finished at: 2021-05-11T11:14:24,570Z
Prune Duration 31 ms
This log indicates how many pre-execution requests succeed and fail for each Replica node. In addition, a sequence number is printed for each failed request that can be debugged later.
Number of Requests Processed: 99
Number of Requests Completed Successfully: 92
Number of Requests with Errors: 4
Number of Requests which were Retried: 2
Number of Requests which were yet to complete at the time of processing of logs: 3
Average execution time for a request: 186.01ms
Detail of every requests type
Details of Request with Errors:
Client ID: 20
Request Sequence number: 6768131009562017792
Error Message: Ignore ClientPreProcessRequestMsg as invalid reqSeqNum: 6768131009562017792, clientId: 20, invalidClient: 0, sentFromReplicaToNonPrimary:0
Replica Name: concord2_1
This log shows the required detail about View Change per Replica node. For example, there might be information on which Replica node initiated View Change and why. If a View change occurred, information on Replica nodes received New View is available.
Summary for view_change number - 1
Replica - 1, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:07.750000
Replica - 2, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.706000
Replica - 6, Previous View - 0, Complaint filed at : 2021-02-17 06:04:06.578000
Replica - 5, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000
Replica - 4, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000
Replica - 3, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000
Replica - 4, Previous View - 0, Complaint filed at : 2021-02-17 06:04:07.761000
Replica - 6, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000
Replica - 1, Previous View - 0, Complaint filed at : 2021-02-17 06:04:08.705000
New View : 1
Replica-1, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.869000
Replica-2, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.882000
Replica-4, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000
Replica-3, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000
Replica-5, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000
Replica-6, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000
This log includes details on whether State transfer occurred.
Summary for state_transfer number - 1
Start Time : 2021-02-05T05:13:21,145Z
Destination replica : concord2_1