Display Firewall Packet Drop Statistics

Starting with NSX for vSphere 6.2.3, you can use the command show packet drops to display packet drop statistics for the firewall.

To run the command, log in to the NSX Edge CLI and enter basic mode. For more information, see the NSX Command Line Interface Reference. For example:

show packet drops

vShield Edge Packet Drop Stats:


Firewall Drop Counters
======================

Ipv4 Rules
==========
Chain - INPUT
rid pkts bytes target prot opt in out source    destination
0    119 30517 DROP   all  --   *   * 0.0.0.0/0 0.0.0.0/0    state INVALID
0      0     0 DROP   all  --   *   * 0.0.0.0/0 0.0.0.0/0
Chain - POSTROUTING
rid pkts bytes target prot opt in out source    destination
0    101 4040  DROP   all   --  *   * 0.0.0.0/0 0.0.0.0/0    state INVALID
0      0    0  DROP   all   --  *   * 0.0.0.0/0 0.0.0.0/0

Ipv6 Rules
==========
Chain - INPUT
rid pkts bytes target prot opt in out source destination
0      0     0   DROP  all      *   * ::/0   ::/0            state INVALID
0      0     0   DROP  all      *   * ::/0   ::/0
Chain - POSTROUTING
rid pkts bytes target prot opt in out source destination
0      0     0   DROP  all       *   * ::/0   ::/0           state INVALID
0      0     0   DROP  all       *   * ::/0   ::/0

Edge Packet Firewall Issues

To run a command, log in to the NSX Edge CLI and enter basic mode. For more information, see the NSX Command Line Interface Reference.

  1. Check the firewall rules table with the show firewall command. The usr_rules table displays the configured rules.

    nsxedge> show firewall
    Chain PREROUTING (policy ACCEPT 3146M packets, 4098G bytes)
    rid    pkts bytes target     prot opt in     out     source               destination
    
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    rid    pkts bytes target     prot opt in     out     source               destination
    0     78903   16M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    0         0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
    0      140K 9558K block_in   all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     23789 1184K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0      116K 8374K usr_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0         0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0
    
    Chain FORWARD (policy ACCEPT 3146M packets, 4098G bytes)
    rid    pkts bytes target     prot opt in     out     source               destination
    
    Chain OUTPUT (policy ACCEPT 173K packets, 22M bytes)
    rid    pkts bytes target     prot opt in     out     source               destination
    
    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
    rid    pkts bytes target     prot opt in     out     source               destination
    0     78903   16M ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
    0      679K   41M DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
    0     3146M 4098G block_out  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0         0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in tap0 --physdev-out vNic_+
    0         0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vNic_+ --physdev-out tap0
    0         0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in na+ --physdev-out vNic_+
    0         0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vNic_+ --physdev-out na+
    0     3145M 4098G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0      221K   13M usr_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0         0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0
    
    Chain block_in (1 references)
    rid    pkts bytes target     prot opt in     out     source               destination
    
    Chain block_out (1 references)
    rid    pkts bytes target     prot opt in     out     source               destination
    
    Chain usr_rules (2 references)
    rid    pkts bytes target     prot opt in     out     source               destination
    131074 70104 5086K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set 0_131074-os-v4-1 src
    131075  116K 8370K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set 1_131075-ov-v4-1 dst
    131073  151K 7844K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0
    

    Check for an incrementing value of a DROP invalid rule in the POST_ROUTING section of the show firewall command. Typical reasons include:

    • Asymmetric routing issues

    • TCP-based applications that have been inactive for more than one hour. If there are inactivity time-out issues and applications are idle for an unusually long time, increase inactivity-timeout settings using the REST API. See https://kb.vmware.com/kb/2101275

  2. Collect the show ipset command output.

    nsxedge> show ipset
    Name: 0_131074-os-v4-1
    Type: bitmap:if (Interface Match)
    Revision: 3
    Header: range 0-64000
    Size in memory: 8116
    References: 1
    Number of entries: 1
    Members:
    vse (vShield Edge Device)
    
    Name: 0_131074-os-v6-1
    Type: bitmap:if (Interface Match)
    Revision: 3
    Header: range 0-64000
    Size in memory: 8116
    References: 1
    Number of entries: 1
    Members:
    vse (vShield Edge Device)
    
    Name: 1_131075-ov-v4-1
    Type: hash:oservice (Match un-translated Ports)
    Revision: 2
    Header: family inet hashsize 64 maxelem 65536
    Size in memory: 704
    References: 1
    Number of entries: 2
    Members:
    Proto=6, DestPort=179, SrcPort=Any    (encoded: 0.6.0.179,0.6.0.0/16)
    Proto=89, DestPort=Any, SrcPort=Any    (encoded: 0.89.0.0/16,0.89.0.0/16)
    
    Name: 1_131075-ov-v6-1
    Type: hash:oservice (Match un-translated Ports)
    Revision: 2
    Header: family inet hashsize 64 maxelem 65536
    Size in memory: 704
    References: 1
    Number of entries: 2
    Members:
    Proto=89, DestPort=Any, SrcPort=Any    (encoded: 0.89.0.0/16,0.89.0.0/16)
    Proto=6, DestPort=179, SrcPort=Any    (encoded: 0.6.0.179,0.6.0.0/16)
    
  3. Enable logging on a particular firewall rule using the REST API or the Edge user interface, and monitor the logs with the show log follow command.

    If logs are not seen, enable logging on the DROP Invalid rule using the following REST API.

    URL : https://NSX_Manager_IP/api/4.0/edges/{edgeId}/firewall/config/global
    
    PUT Method 
    Input representation 
    <globalConfig>   <!-- Optional -->
    <tcpPickOngoingConnections>false</tcpPickOngoingConnections>   <!-- Optional. Defaults to false -->
    <tcpAllowOutOfWindowPackets>false</tcpAllowOutOfWindowPackets>    <!-- Optional. Defaults to false -->
    <tcpSendResetForClosedVsePorts>true</tcpSendResetForClosedVsePorts>    <!-- Optional. Defaults to true -->
    <dropInvalidTraffic>true</dropInvalidTraffic>    <!-- Optional. Defaults to true -->
    <logInvalidTraffic>true</logInvalidTraffic>     <!-- Optional. Defaults to false -->
    <tcpTimeoutOpen>30</tcpTimeoutOpen>       <!-- Optional. Defaults to 30 -->
    <tcpTimeoutEstablished>3600</tcpTimeoutEstablished>   <!-- Optional. Defaults to 3600 -->
    <tcpTimeoutClose>30</tcpTimeoutClose>   <!-- Optional. Defaults to 30 -->
    <udpTimeout>60</udpTimeout>             <!-- Optional. Defaults to 60 -->
    <icmpTimeout>10</icmpTimeout>           <!-- Optional. Defaults to 10 -->
    <icmp6Timeout>10</icmp6Timeout>           <!-- Optional. Defaults to 10 -->
    <ipGenericTimeout>120</ipGenericTimeout>    <!-- Optional. Defaults to 120 -->
    </globalConfig>
    Output representation 
    No payload
    

    Use the show log follow command to look for logs similar to:

    2016-04-18T20:53:31+00:00 edge-0 kernel: nf_ct_tcp: invalid TCP flag combination IN= OUT= 
    SRC=172.16.1.4 DST=192.168.1.4 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=43343 PROTO=TCP 
    SPT=5050 DPT=80 SEQ=0 ACK=1572141176 WINDOW=512 RES=0x00 URG PSH FIN URGP=0
    2016-04-18T20:53:31+00:00 edge-0 kernel: INVALID IN= OUT=vNic_1 SRC=172.16.1.4 
    DST=192.168.1.4 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=43343 PROTO=TCP SPT=5050 DPT=80 
    WINDOW=512 RES=0x00 URG PSH FIN URGP=0
    

  4. Check for matching connections in the Edge firewall state table with the show flowtable rule_id command:

    nsxedge> show flowtable
    1: tcp  6 21554 ESTABLISHED src=192.168.110.10 dst=192.168.5.3 sport=25981 
    d port=22 pkts=52 bytes=5432 src=192.168.5.3 dst=192.168.110.10 sport=22 dport=259 
    81 pkts=44 bytes=7201 [ASSURED] mark=0 rid=131073 use=1
    2: tcp  6 21595 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=53194 
    dport=10 001 pkts=33334 bytes=11284650 src=127.0.0.1 dst=127.0.0.1 sport=10001 dport=5319 
    4 pkts=33324 bytes=1394146 [ASSURED] mark=0 rid=0 use=1
    

    Compare the active connection count and the maximum allowed count with the show flowstats command:

    nsxedge> show flowstats
    Total Flow Capacity: 65536
    Current Statistics :
    cpu=0 searched=3280373 found=3034890571 new=52678 invalid=659946 ignore=77605 
    delete=52667 delete_list=49778 insert=49789 insert_failed=0 drop=0 early_drop=0 
    error=0 search_restart=0
    

  5. Check the Edge logs with the show log follow command, and look for any ALG drops. Search for strings similar to tftp_alg, msrpc_alg, or oracle_tns. For additional information, see: