Monitor Packet Processing Queue

The packet processing engine on the Gateway involves multiple stages, and each stage has a packet processing queue in between. Due to the bursty nature of traffic through a Gateway, occasional packet builds up in the packet forwarding queues are expected. However, consistent high queue length in certain queues indicate a capacity problem.

The following example shows output of the debug.py command to view handoff queue output.

The output has been truncated to display only the first and last entries, for conciseness. You can exclude the -v option in the command to view the output in tabular format.

vcadmin@vcg1-example:~$ /opt/vc/bin/debug.py -v --handoff
{
  "handoffq": [
    {
      "deq": 12126489784,
      "drops": 0,
      "enq": 12126482089,
      "name": "vc_queue_net_sch",
      "qlength": 0,
      "qlimit": 4096,
      "sleeping": 1475174572,
      "tid": 1502,
      "wmark": 1280,
      "wmark_1min": 385,
      "wmark_5min": 450,
      "wokenup": 1164664965
    }, 
    … 
    {
      "deq": 767292,
      "drops": 0,
      "enq": 767272,
      "name": "vc_queue_ip_common_bh_1",
      "qlength": 0,
      "qlimit": 16384,
      "sleeping": 596612,
      "tid": 1512,
      "wmark": 53,
      "wmark_1min": 3,
      "wmark_5min": 3,
      "wokenup": 596209
    }
  ]
}
vcadmin@vcg1-example:~$ 

You need to note the values of qlength and wmark.

The qlength column indicates the number of packets that are currently buffered in the queue. The wmark column indicates the maximum depth a queue has ever reached, which indicates how close a Gateway has come to dropping packets. The impact and remediation for these depends largely on the queue being monitored.

You should monitor both the critical and non-critical queues.