In August 2019, Netflix discovered several resource exhaustion vectors that can be used to launch denial of service (DoS) attacks against servers supporting HTTP/2 communication. Netflix worked with Google and CERT/CC to coordinate disclosure to the Internet community. Four new parameters have been incorporated in the NSX Advanced Load Balancer to cover these security issues.
For more information on the DoS Advisory, see HTTP/2 Denial of Service Advisory.
Common Vulnerabilities and Exposures
The following 8 Common Vulnerabilities and Exposures (CVEs) are associated with the HTTP/2 Denial of Service Advisory:
CVE-2019-9511: HTTP/2 Data Dribble
CVE-2019-9512: HTTP/2 Ping Flood
CVE-2019-9513: HTTP/2 Resource Loop
CVE-2019-9514: HTTP/2 Reset Flood
CVE-2019-9515: HTTP/2 Settings Flood
CVE-2019-9516: HTTP/2 0-Length Headers Leak
CVE-2019-9517: HTTP/2 Internal Data Buffering
CVE-2019-9518: HTTP/2 Empty Frame Flooding
Addressing the CVEs and Exposures
The above CVEs and other issues are addressed through new HTTP-application-profile parameters and other code. The four parameters are accessible through the NSX Advanced Load Balancer REST API and NSX Advanced Load Balancer CLI.
max_http2_control_frames_per_connection
Maximum number of control frames that a client can send over an HTTP/2 connection
Used to detect ping flood (CVE-2019-9512), reset flood (CVE-2019-9514), setting flood (CVE-2019-9515) and resource loop issue (CVE-2019-9513)
Range: 0-10,000; default value= 1000
0 is interpreted as unlimited control frames on a client-side HTTP/2 connection
max_http2_queued_frames_to_client_per_connection
Maximum number of frames that can be queued waiting to be sent over a client-side HTTP/2 connection at any given time
Used to detect data dribble (CVE-2019-9511 ) and internal data buffering (CVE-2019-9517)
Range: 0 -10,000; default value= 10000 is interpreted as unlimited queued frames
Also incorporated is a fix from NGINX, nginx/nginx@a987f81
max_http2_empty_data_frames_per_connection
Maximum number of empty data frames over a client-side HTTP/2 connection
Used to detect empty data frame flooding (CVE-2019-9518)
-
Note:
For an empty header frame, our current implementation closes the connection and reports an error like: HTTP2 Frame Size Error: Client sent HEADERS frame with empty header block.
Range: 0 - 10,000; default = 10000 is interpreted as unlimited empty frames
max_http2_concurrent_streams_per_connection
Maximum number of concurrent streams over a client-side HTTP/2 connection
Used to control the concurrent streams one HTTP/2 connection can handle at any given time
Range 1 - 256; default value= 128
-
Note:
This parameter is not related to any of the CVEs issues mentioned above.
The 0-length headers leak (CVE-2019-9516) is detected by current code. Upon detection, the stream will be closed and NSX Advanced Load Balancer reports an error: HTTP2 Protocol Error: Client sent zero length header name.
max_http2_requests_per_connection
This value controls the maximum number of that can be sent over a client-side HTTP/2 connection. If the value is set to 0, this means an unlimited number of requests can be sent over an HTTP/2 client-side connection.
max_http2_header_field
This field controls the maximum size (in BYTES) of the compressed request header field. The limit applies equally to both name and value of the request header. The range varies from 1-8192 BYTES. The default value is 4096 BYTES.
http2_initial_window_size
This field controls the window size of the initial flow control in HTTP/2 streams. The value for this field ranges from 64-32768 KB. The default value for this field is 64kb.
CLI Configuration Example
[admin:th-controller-3]: > show applicationprofile applicationprofile-22 +----------------------------------------------------+---------------------------------------------------------+ | Field | Value | +----------------------------------------------------+---------------------------------------------------------+ | uuid | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c | | name | applicationprofile-22 | | type | APPLICATION_PROFILE_TYPE_HTTP | | http_profile | | | connection_multiplexing_enabled | True | | xff_enabled | False | | xff_alternate_name | X-Forwarded-For | | ssl_everywhere_enabled | False | | hsts_enabled | False | | hsts_max_age | 365 | | secure_cookie_enabled | False | | httponly_enabled | False | | http_to_https | False | | server_side_redirect_to_https | False | | x_forwarded_proto_enabled | False | | spdy_enabled | False | | spdy_fwd_proxy_mode | False | | post_accept_timeout | 30000 milliseconds | | client_header_timeout | 10000 milliseconds | | client_body_timeout | 30000 milliseconds | | keepalive_timeout | 30000 milliseconds | | client_max_header_size | 12 kb | | client_max_request_size | 48 kb | | client_max_body_size | 0 kb | | max_rps_unknown_uri | 0 | | max_rps_cip | 0 | | max_rps_uri | 0 | | max_rps_cip_uri | 0 | | ssl_client_certificate_mode | SSL_CLIENT_CERTIFICATE_NONE | | websockets_enabled | True | | max_rps_unknown_cip | 0 | | max_bad_rps_cip | 0 | | max_bad_rps_uri | 0 | | max_bad_rps_cip_uri | 0 | | keepalive_header | False | | use_app_keepalive_timeout | False | | allow_dots_in_header_name | False | | disable_keepalive_posts_msie6 | True | | enable_request_body_buffering | False | | enable_fire_and_forget | False | | max_response_headers_size | 48 kb | | http2_enabled | True | | respond_with_100_continue | True | | hsts_subdomains_enabled | True | | enable_request_body_metrics | False | | fwd_close_hdr_for_bound_connections | True | | max_keepalive_requests | 100 | | disable_sni_hostname_check | False | | max_http2_control_frames_per_connection | 1000 | | max_http2_queued_frames_to_client_per_connection | 1000 | | max_http2_empty_data_frames_per_connection | 1000 | | max_http2_concurrent_streams_per_connection | 128 | | reset_conn_http_on_ssl_port | False | | preserve_client_ip | False | | preserve_client_port | False | | tenant_ref | admin | +----------------------------------------------------+---------------------------------------------------------+ [admin:th-controller-3]: > [admin:th-controller-3]: applicationprofile> configure applicationprofile applicationprofile-22 [admin:th-controller-3]: applicationprofile> http_profile [admin:th-controller-3]: applicationprofile:http_profile> max_http2_control_frames_per_connection 2000 Overwriting the previously entered value for max_http2_control_frames_per_connection [admin:th-controller-3]: applicationprofile:http_profile> max_http2_queued_frames_to_client_per_connection 2000 Overwriting the previously entered value for max_http2_queued_frames_to_client_per_connection [admin:th-controller-3]: applicationprofile:http_profile> max_http2_empty_data_frames_per_connection 2000 Overwriting the previously entered value for max_http2_empty_data_frames_per_connection [admin:th-controller-3]: applicationprofile:http_profile> max_http2_concurrent_streams_per_connection 256 Overwriting the previously entered value for max_http2_concurrent_streams_per_connection [admin:th-controller-3]: applicationprofile:http_profile> save [admin:th-controller-3]: applicationprofile> save [admin:th-controller-3]: > show applicationprofile applicationprofile-22 +----------------------------------------------------+---------------------------------------------------------+ | Field | Value | +----------------------------------------------------+---------------------------------------------------------+ | uuid | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c | | name | applicationprofile-22 | | type | APPLICATION_PROFILE_TYPE_HTTP | | http_profile | | | connection_multiplexing_enabled | True | | xff_enabled | False | | xff_alternate_name | X-Forwarded-For | | ssl_everywhere_enabled | False | | hsts_enabled | False | | hsts_max_age | 365 | | secure_cookie_enabled | False | | httponly_enabled | False | | http_to_https | False | | server_side_redirect_to_https | False | | x_forwarded_proto_enabled | False | | spdy_enabled | False | | spdy_fwd_proxy_mode | False | | post_accept_timeout | 30000 milliseconds | | client_header_timeout | 10000 milliseconds | | client_body_timeout | 30000 milliseconds | | keepalive_timeout | 30000 milliseconds | | client_max_header_size | 12 kb | | client_max_request_size | 48 kb | | client_max_body_size | 0 kb | | max_rps_unknown_uri | 0 | | max_rps_cip | 0 | | max_rps_uri | 0 | | max_rps_cip_uri | 0 | | ssl_client_certificate_mode | SSL_CLIENT_CERTIFICATE_NONE | | websockets_enabled | True | | max_rps_unknown_cip | 0 | | max_bad_rps_cip | 0 | | max_bad_rps_uri | 0 | | max_bad_rps_cip_uri | 0 | | keepalive_header | False | | use_app_keepalive_timeout | False | | allow_dots_in_header_name | False | | disable_keepalive_posts_msie6 | True | | enable_request_body_buffering | False | | enable_fire_and_forget | False | | max_response_headers_size | 48 kb | | http2_enabled | True | | respond_with_100_continue | True | | hsts_subdomains_enabled | True | | enable_request_body_metrics | False | | fwd_close_hdr_for_bound_connections | True | | max_keepalive_requests | 100 | | disable_sni_hostname_check | False | | max_http2_control_frames_per_connection | 2000 | | max_http2_queued_frames_to_client_per_connection | 2000 | | max_http2_empty_data_frames_per_connection | 2000 | | max_http2_concurrent_streams_per_connection | 256 | | reset_conn_http_on_ssl_port | False | | preserve_client_ip | False | | preserve_client_port | False | | tenant_ref | admin | +----------------------------------------------------+---------------------------------------------------------+ [admin:th-controller-3]: >
