DNS slow client attack is similar to HTTP slowloris attack, in which client sends DNS request over TCP byte by byte slowly to keep the connection alive, occupying the packet buffer. When done simultaneously over multiple TCP connections, this can result in a lack of memory resources to process legitimate requests as Service Engine buffers the data and waits for a complete DNS request on each connection.
Symptoms
The most common symptoms include too many opened TCP connections on the DNS virtual services and legitimate DNS Traffic stops working.
Events on Service Engines with code PKT_DROP_NO_PKT_BUFF are observed and connections drops are observed due to a low packet buffer, as shown below.
A high number of TCP connections is observed in the connection_stats in tcpstats of the attacked DNS virtual service.
To check the number of connections, monitor the values of connections_accepted and connection_established using the show virtualservice <dns-vs-name> tcpstat filter type frontend | grep -vw 0 command. The dropped connections because of time exceeding the client_dns_tcp_request_timeout can be checked in tcps_dns_connection_closed_slow_client of the show virtualservice <dns-vs-name> tcpstat filter type frontend | grep -vw 0 command.
[admin:cntrlr]: > show virtualservice vs-try tcpstat filter type frontend | grep -vw 0 +----------------------------------------------+------------------------------------+ | Field | Value | +----------------------------------------------+------------------------------------+ | se_uuid | Avi-Service-Engine:se-0000006f0ba7 | | proc_id | PROC_Aggregate | | connection_stats | | | connections_accepted | 101 | | connections_established | 101 | | connections_closed | 101 | | rx_stats | | | total_packets_received | 24911 | | packets_received_in_sequence | 24709 | | bytes_received_in_sequence | 1669642 | | ack_packets_received | 101 | | ack_byte_received | 101 | | tx_stats | | | total_packets_sent | 12803 | | data_bytes_sent | 843786 | | ack_only_packet | 12702 | | retransmit_stats | | | dup_ack_packets_received | 1 | | syncache_stats | | | entry_added_to_syncache | 101 | | entry_completed | 101 | | connections_dropped | | | connections_dropped_after_established | 101 | | num_resets_sent | 101 | | packets_dropped | | | timeout | | | sack_stats | | | ecn_stats | | | misc_stats | | | rtt_updated | 202 | | times_hdr_predict_ok_for_data_pkts | 24708 | | connections_using_auto_gateway | 101 | | dns_stats | | | query_others | 100 | | local_responses | 100 | | tcps_dns_connection_closed_slow_client | 100 | +----------------------------------------------+------------------------------------+ [admin:cntrlr]: >
Mitigating DNS Slow Client Attack
There is a client_dns_tcp_request_timeout parameter associated with each open TCP (DNS) connection. The functionality of this timeout is to expire after the set limit to ensure that the connection is closed within the time limit. So, in case the attacker sends a few bytes and opens many connections, the connection will be terminated after the timeout expires. The timeout is configurable and can be set in the application profile of a virtual service. Connections are closed aggressively when the timeout value is set to lower values, and the connections last for a longer time when the large timeout values are configured.
Log in to NSX Advanced Load Balancer CLI and use the configure applicationprofile <application profile name> command to set the desired timeout values for the DNS service profile, as shown below.
[admin:cntrlr]: > configure applicationprofile System-DNS Updating an existing object. Currently, the object is: +--------------------------------------+---------------------------------------------------------+ | Field | Value | +--------------------------------------+---------------------------------------------------------+ | uuid | applicationprofile-ea8c65d4-ad02-4b68-b8a9-d1b1186f42be | | name | System-DNS | | type | APPLICATION_PROFILE_TYPE_DNS | | dns_service_profile | | | num_dns_ip | 1 | | ttl | 30 sec | | error_response | DNS_ERROR_RESPONSE_NONE | | edns | True | | dns_over_tcp_enabled | True | | aaaa_empty_response | True | | negative_caching_ttl | 30 sec | | admin_email | hostmaster | | close_tcp_connection_post_response | False | | client_dns_tcp_request_timeout | 10000 milliseconds | | ecs_stripping_enabled | True | | preserve_client_ip | False | | preserve_client_port | False | | preserve_dest_ip_port | False | | tenant_ref | admin | +--------------------------------------+---------------------------------------------------------+ [admin:cntrlr]: applicationprofile> dns_service_profile [admin:cntrlr]: applicationprofile:dns_service_profile> client_dns_tcp_request_timeout 100 Overwriting the previously entered value for client_dns_tcp_request_timeout [admin:cntrlr]: applicationprofile:dns_service_profile> save [admin:cntrlr]: applicationprofile> save +--------------------------------------+---------------------------------------------------------+ | Field | Value | +--------------------------------------+---------------------------------------------------------+ | uuid | applicationprofile-ea8c65d4-ad02-4b68-b8a9-d1b1186f42be | | name | System-DNS | | type | APPLICATION_PROFILE_TYPE_DNS | | dns_service_profile | | | num_dns_ip | 1 | | ttl | 30 sec | | error_response | DNS_ERROR_RESPONSE_NONE | | edns | True | | dns_over_tcp_enabled | True | | aaaa_empty_response | True | | negative_caching_ttl | 30 sec | | admin_email | hostmaster | | close_tcp_connection_post_response | False | | client_dns_tcp_request_timeout | 100 milliseconds | | ecs_stripping_enabled | True | | preserve_client_ip | False | | preserve_client_port | False | | preserve_dest_ip_port | False | | tenant_ref | admin | +--------------------------------------+---------------------------------------------------------+ [admin:cntrlr]: >
