The log parser tool simplifies debugging errors by analyzing logs and summarizing different modules.

You can peruse this summary to determine the error zone and evaluate the root cause. In some instances, you can identify potential root cause zones without knowing the exact technical root cause.

Before the availability of the log parser tool, users manually examined logs from every Replica node to find the Replica node with the error.

The log parser tool contains the log parser agent and log parser appliance. You must enable the ENABLE_SIDE_CAR_VM_DEPLOYMENT parameter in the infrastructure and deployment descriptor files before deploying the blockchain nodes. See Configuring the Infrastructure Descriptor Parameters on AWS and Configuring the Deployment Descriptor Parameters on AWS.

The log parser agent is a Docker container deployed on each blockchain node. The log parser agent sends the compressed logs to the log parser appliance at certain intervals. Log preprocessing is not performed at the agent level to avoid performance latency. The log parser agent retrieves logs from blockchain nodes and requires access to all the blockchain nodes.

The log parser appliance is deployed with the blockchain nodes. The log parser appliance container processes the logs at a certain interval and generates the log summary, which is grouped based on the time stamp. After the log summary is stored, the logs are retained in the /var/lib/docker/containers/retained_logs/ path and differentiated using a time stamp.

The log parser tool can parse up to 800MB of compressed logs or 10GB of logs.

Note:

You cannot enable or disable the log parser tool after deploying the blockchain nodes.

You cannot use the log parser tool for scaled up and down blockchain nodes.

Procedure

  • Disable the log parser appliance and log parser agent container. 
    docker stop <container Name>
  • Remove all the containers that have exited 
    docker rm $(docker ps --filter status=exited -q)
  • Remove the older log files from the sideCar VM for every 4 hours, if you are performing large transactions to avoid issues related to volume.
  • Analyze the logs to identify the error and proceed to troubleshoot the problem.

    The parser summary is stored in the side car node under the /var/lib/docker/containers/parser_results folder grouped by time stamp.

    Log Parser Summary File Type Description

    post_execution_parser_summary.log

    These logs show all the post-execution requests, failures, and successes.

    Sample post_execution_parser

    Number of Requests processed: 3306

    Number of conflicts in post-execution: 0

    Number of success requests in post-execution: 3306

    consensus_parser_summary.log

    These logs include path change and switch sequence numbers for each Replica node. The logs indicate that all Replica nodes are in consensus with the same Slow/Fast path and sequence number.

    Sample consensus_parser number - 1

    path change : OPTIMISTIC_FAST to SLOW

    time : 2021-03-12 12:44:57.593000

    switch sequence number : 64

    primary : 0

    duration : 0:00:07.224000

    count of request downgraded to slow: supposed to be fast : 29
    replica_pruning_parser_summary.log

    The log shows the pruning operation details for each Replica node. The log is for information purposes only and not for troubleshooting.

    Sample for the Prune ID: 1

    Replica Name: concord2_1

    Number of deleted blocks: 1302

    Last deleted block id: 1302

    Pruning Started at: 2021-05-11T11:14:24,539Z

    Prune Finished at: 2021-05-11T11:14:24,570Z

    Prune Duration 31 ms
    pre_execution_parser_summary.log

    This log indicates how many pre-execution requests succeed and fail for each Replica node. In addition, a sequence number is printed for each failed request that can be debugged later.

    Sample pre_execution_parser

    Number of Requests Processed: 99

    Number of Requests Completed Successfully: 92

    Number of Requests with Errors: 4

    Number of Requests which were Retried: 2

    Number of Requests which were yet to complete at the time of processing of logs: 3

    Average execution time for a request: 186.01ms

    Detail of every requests type

    Details of Request with Errors:

    Client ID: 20

    Request Sequence number: 6768131009562017792

    Error Message: Ignore ClientPreProcessRequestMsg as invalid reqSeqNum: 6768131009562017792, clientId: 20, invalidClient: 0, sentFromReplicaToNonPrimary:0

    Replica Name: concord2_1
    vc_parser_summary.log

    This log shows the required detail about View Change per Replica node. For example, there might be information on which Replica node initiated View Change and why. If a View change occurred, information on Replica nodes received New View is available.

    Summary example

    Summary for view_change number - 1

    Replica - 1, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:07.750000

    Replica - 2, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.706000

    Replica - 6, Previous View - 0, Complaint filed at : 2021-02-17 06:04:06.578000

    Replica - 5, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000

    Replica - 4, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000

    Replica - 3, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000

    Replica - 4, Previous View - 0, Complaint filed at : 2021-02-17 06:04:07.761000

    Replica - 6, Previous View - 0, Timer Complaint filed at : 2021-02-17 06:04:08.719000

    Replica - 1, Previous View - 0, Complaint filed at : 2021-02-17 06:04:08.705000

    New View : 1

    Replica-1, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.869000

    Replica-2, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.882000

    Replica-4, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000

    Replica-3, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000

    Replica-5, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000

    Replica-6, Previous view - 0, New View - 1, agreed at : 2021-02-17 06:04:08.895000
    st_parser_summary.log

    This log includes details on whether State transfer occurred.

    Summary example

    Summary for state_transfer number - 1

    Start Time : 2021-02-05T05:13:21,145Z

    Destination replica : concord2_1