The NSX CLI can be used to troubleshoot firewall packet drop issues.
Display Firewall Packet Drop Statistics
Starting with NSX Data Center for vSphere 6.2.3, you can use the command show packet drops to display packet drop statistics for the firewall.
To run the command, log in to the NSX Edge CLI and enter basic mode. For more information, see the
NSX Command Line Interface Reference. For example:
show packet drops vShield Edge Packet Drop Stats: Firewall Drop Counters ====================== Ipv4 Rules ========== Chain - INPUT rid pkts bytes target prot opt in out source destination 0 119 30517 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain - POSTROUTING rid pkts bytes target prot opt in out source destination 0 101 4040 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Ipv6 Rules ========== Chain - INPUT rid pkts bytes target prot opt in out source destination 0 0 0 DROP all * * ::/0 ::/0 state INVALID 0 0 0 DROP all * * ::/0 ::/0 Chain - POSTROUTING rid pkts bytes target prot opt in out source destination 0 0 0 DROP all * * ::/0 ::/0 state INVALID 0 0 0 DROP all * * ::/0 ::/0
Edge Packet Firewall Issues
To run a command, log in to the NSX Edge CLI and enter basic mode. For more information, see the NSX Command Line Interface Reference.
- Check the firewall rules table with the show firewall command. The usr_rules table displays the configured rules.
nsxedge> show firewall Chain PREROUTING (policy ACCEPT 3146M packets, 4098G bytes) rid pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 0 packets, 0 bytes) rid pkts bytes target prot opt in out source destination 0 78903 16M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 0 140K 9558K block_in all -- * * 0.0.0.0/0 0.0.0.0/0 0 23789 1184K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 116K 8374K usr_rules all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain FORWARD (policy ACCEPT 3146M packets, 4098G bytes) rid pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 173K packets, 22M bytes) rid pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) rid pkts bytes target prot opt in out source destination 0 78903 16M ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 0 679K 41M DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 0 3146M 4098G block_out all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap0 --physdev-out vNic_+ 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vNic_+ --physdev-out tap0 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in na+ --physdev-out vNic_+ 0 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in vNic_+ --physdev-out na+ 0 3145M 4098G ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 0 221K 13M usr_rules all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 Chain block_in (1 references) rid pkts bytes target prot opt in out source destination Chain block_out (1 references) rid pkts bytes target prot opt in out source destination Chain usr_rules (2 references) rid pkts bytes target prot opt in out source destination 131074 70104 5086K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 match-set 0_131074-os-v4-1 src 131075 116K 8370K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 match-set 1_131075-ov-v4-1 dst 131073 151K 7844K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Check for an incrementing value of a DROP invalid rule in the POST_ROUTING section of the show firewall command. Typical reasons include:- Asymmetric routing issues
- TCP-based applications that have been inactive for more than one hour. If there are inactivity time-out issues and applications are idle for an unusually long time, increase inactivity-timeout settings using the REST API. See https://kb.vmware.com/kb/2101275
- Collect the show ipset command output.
nsxedge> show ipset Name: 0_131074-os-v4-1 Type: bitmap:if (Interface Match) Revision: 3 Header: range 0-64000 Size in memory: 8116 References: 1 Number of entries: 1 Members: vse (vShield Edge Device) Name: 0_131074-os-v6-1 Type: bitmap:if (Interface Match) Revision: 3 Header: range 0-64000 Size in memory: 8116 References: 1 Number of entries: 1 Members: vse (vShield Edge Device) Name: 1_131075-ov-v4-1 Type: hash:oservice (Match un-translated Ports) Revision: 2 Header: family inet hashsize 64 maxelem 65536 Size in memory: 704 References: 1 Number of entries: 2 Members: Proto=6, DestPort=179, SrcPort=Any (encoded: 0.6.0.179,0.6.0.0/16) Proto=89, DestPort=Any, SrcPort=Any (encoded: 0.89.0.0/16,0.89.0.0/16) Name: 1_131075-ov-v6-1 Type: hash:oservice (Match un-translated Ports) Revision: 2 Header: family inet hashsize 64 maxelem 65536 Size in memory: 704 References: 1 Number of entries: 2 Members: Proto=89, DestPort=Any, SrcPort=Any (encoded: 0.89.0.0/16,0.89.0.0/16) Proto=6, DestPort=179, SrcPort=Any (encoded: 0.6.0.179,0.6.0.0/16) - Enable logging on a particular firewall rule using the REST API or the Edge user interface, and monitor the logs with the show log follow command.
If logs are not seen, enable logging on the DROP Invalid rule using the following REST API.
URL : https://NSX_Manager_IP/api/4.0/edges/{edgeId}/firewall/config/global PUT Method Input representation <globalConfig> <!-- Optional --> <tcpPickOngoingConnections>false</tcpPickOngoingConnections> <!-- Optional. Defaults to false --> <tcpAllowOutOfWindowPackets>false</tcpAllowOutOfWindowPackets> <!-- Optional. Defaults to false --> <tcpSendResetForClosedVsePorts>true</tcpSendResetForClosedVsePorts> <!-- Optional. Defaults to true --> <dropInvalidTraffic>true</dropInvalidTraffic> <!-- Optional. Defaults to true --> <logInvalidTraffic>true</logInvalidTraffic> <!-- Optional. Defaults to false --> <tcpTimeoutOpen>30</tcpTimeoutOpen> <!-- Optional. Defaults to 30 --> <tcpTimeoutEstablished>3600</tcpTimeoutEstablished> <!-- Optional. Defaults to 3600 --> <tcpTimeoutClose>30</tcpTimeoutClose> <!-- Optional. Defaults to 30 --> <udpTimeout>60</udpTimeout> <!-- Optional. Defaults to 60 --> <icmpTimeout>10</icmpTimeout> <!-- Optional. Defaults to 10 --> <icmp6Timeout>10</icmp6Timeout> <!-- Optional. Defaults to 10 --> <ipGenericTimeout>120</ipGenericTimeout> <!-- Optional. Defaults to 120 --> </globalConfig> Output representation No payloadUse the show log follow command to look for logs similar to:2016-04-18T20:53:31+00:00 edge-0 kernel: nf_ct_tcp: invalid TCP flag combination IN= OUT= SRC=172.16.1.4 DST=192.168.1.4 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=43343 PROTO=TCP SPT=5050 DPT=80 SEQ=0 ACK=1572141176 WINDOW=512 RES=0x00 URG PSH FIN URGP=0 2016-04-18T20:53:31+00:00 edge-0 kernel: INVALID IN= OUT=vNic_1 SRC=172.16.1.4 DST=192.168.1.4 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=43343 PROTO=TCP SPT=5050 DPT=80 WINDOW=512 RES=0x00 URG PSH FIN URGP=0
- Check for matching connections in the Edge firewall state table with the show flowtable rule_id command:
nsxedge> show flowtable 1: tcp 6 21554 ESTABLISHED src=192.168.110.10 dst=192.168.5.3 sport=25981 d port=22 pkts=52 bytes=5432 src=192.168.5.3 dst=192.168.110.10 sport=22 dport=259 81 pkts=44 bytes=7201 [ASSURED] mark=0 rid=131073 use=1 2: tcp 6 21595 ESTABLISHED src=127.0.0.1 dst=127.0.0.1 sport=53194 dport=10 001 pkts=33334 bytes=11284650 src=127.0.0.1 dst=127.0.0.1 sport=10001 dport=5319 4 pkts=33324 bytes=1394146 [ASSURED] mark=0 rid=0 use=1Compare the active connection count and the maximum allowed count with the show flowstats command:nsxedge> show flowstats Total Flow Capacity: 65536 Current Statistics : cpu=0 searched=3280373 found=3034890571 new=52678 invalid=659946 ignore=77605 delete=52667 delete_list=49778 insert=49789 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 -
Check the Edge logs with the show log follow command, and look for any ALG drops. Search for strings similar to tftp_alg, msrpc_alg, or oracle_tns. For additional information, see:
