In August 2019, Netflix discovered several resource exhaustion vectors that can be used to launch denial of service (DoS) attacks against servers supporting HTTP/2 communication. Netflix worked with Google and CERT/CC to coordinate disclosure to the Internet community. Four new parameters have been incorporated in the Avi Load Balancer to cover these security issues.

For more information on the DoS Advisory, see HTTP/2 Denial of Service Advisory.

Common Vulnerabilities and Exposures

The following 8 Common Vulnerabilities and Exposures (CVEs) are associated with the HTTP/2 Denial of Service Advisory:

  • CVE-2019-9511: HTTP/2 Data Dribble

  • CVE-2019-9512: HTTP/2 Ping Flood

  • CVE-2019-9513: HTTP/2 Resource Loop

  • CVE-2019-9514: HTTP/2 Reset Flood

  • CVE-2019-9515: HTTP/2 Settings Flood

  • CVE-2019-9516: HTTP/2 0-Length Headers Leak

  • CVE-2019-9517: HTTP/2 Internal Data Buffering

  • CVE-2019-9518: HTTP/2 Empty Frame Flooding

Addressing the CVEs and Exposures

The above CVEs and other issues are addressed through new HTTP-application-profile parameters and other code. The four parameters are accessible through the Avi Load Balancer REST API and Avi Load Balancer CLI.

  • max_http2_control_frames_per_connection

    • Maximum number of control frames that a client can send over an HTTP/2 connection

    • Used to detect ping flood (CVE-2019-9512), reset flood (CVE-2019-9514), setting flood (CVE-2019-9515) and resource loop issue (CVE-2019-9513)

    • Range: 0-10,000; default value= 1000

    • 0 is interpreted as unlimited control frames on a client-side HTTP/2 connection

  • max_http2_queued_frames_to_client_per_connection

    • Maximum number of frames that can be queued waiting to be sent over a client-side HTTP/2 connection at any given time

    • Used to detect data dribble (CVE-2019-9511 ) and internal data buffering (CVE-2019-9517)

    • Range: 0 -10,000; default value= 10000 is interpreted as unlimited queued frames

    • Also incorporated is a fix from NGINX, nginx/nginx@a987f81

  • max_http2_empty_data_frames_per_connection

    • Maximum number of empty data frames over a client-side HTTP/2 connection

    • Used to detect empty data frame flooding (CVE-2019-9518)

    • Note:

      For an empty header frame, our current implementation closes the connection and reports an error like: HTTP2 Frame Size Error: Client sent HEADERS frame with empty header block.

    • Range: 0 - 10,000; default = 10000 is interpreted as unlimited empty frames

  • max_http2_concurrent_streams_per_connection

    • Maximum number of concurrent streams over a client-side HTTP/2 connection

    • Used to control the concurrent streams one HTTP/2 connection can handle at any given time

    • Range 1 - 256; default value= 128

    • Note:

      This parameter is not related to any of the CVEs issues mentioned above.

The 0-length headers leak (CVE-2019-9516) is detected by current code. Upon detection, the stream will be closed and NSX Advanced Load Balancer reports an error: HTTP2 Protocol Error: Client sent zero length header name.

  • max_http2_requests_per_connection

    • This value controls the maximum number of that can be sent over a client-side HTTP/2 connection. If the value is set to 0, this means an unlimited number of requests can be sent over an HTTP/2 client-side connection.

  • max_http2_header_field

    • This field controls the maximum size (in BYTES) of the compressed request header field. The limit applies equally to both name and value of the request header. The range varies from 1-8192 BYTES. The default value is 4096 BYTES.

  • http2_initial_window_size

    • This field controls the window size of the initial flow control in HTTP/2 streams. The value for this field ranges from 64-32768 KB. The default value for this field is 64kb.

CLI Configuration Example

[admin:th-controller-3]: > show applicationprofile  applicationprofile-22
+----------------------------------------------------+---------------------------------------------------------+
| Field                                              | Value                                                   |
+----------------------------------------------------+---------------------------------------------------------+
| uuid                                               | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c |
| name                                               | applicationprofile-22                                   |
| type                                               | APPLICATION_PROFILE_TYPE_HTTP                           |
| http_profile                                       |                                                         |
|   connection_multiplexing_enabled                  | True                                                    |
|   xff_enabled                                      | False                                                   |
|   xff_alternate_name                               | X-Forwarded-For                                         |
|   ssl_everywhere_enabled                           | False                                                   |
|   hsts_enabled                                     | False                                                   |
|   hsts_max_age                                     | 365                                                     |
|   secure_cookie_enabled                            | False                                                   |
|   httponly_enabled                                 | False                                                   |
|   http_to_https                                    | False                                                   |
|   server_side_redirect_to_https                    | False                                                   |
|   x_forwarded_proto_enabled                        | False                                                   |
|   spdy_enabled                                     | False                                                   |
|   spdy_fwd_proxy_mode                              | False                                                   |
|   post_accept_timeout                              | 30000 milliseconds                                      |
|   client_header_timeout                            | 10000 milliseconds                                      |
|   client_body_timeout                              | 30000 milliseconds                                      |
|   keepalive_timeout                                | 30000 milliseconds                                      |
|   client_max_header_size                           | 12 kb                                                   |
|   client_max_request_size                          | 48 kb                                                   |
|   client_max_body_size                             | 0 kb                                                    |
|   max_rps_unknown_uri                              | 0                                                       |
|   max_rps_cip                                      | 0                                                       |
|   max_rps_uri                                      | 0                                                       |
|   max_rps_cip_uri                                  | 0                                                       |
|   ssl_client_certificate_mode                      | SSL_CLIENT_CERTIFICATE_NONE                             |
|   websockets_enabled                               | True                                                    |
|   max_rps_unknown_cip                              | 0                                                       |
|   max_bad_rps_cip                                  | 0                                                       |
|   max_bad_rps_uri                                  | 0                                                       |
|   max_bad_rps_cip_uri                              | 0                                                       |
|   keepalive_header                                 | False                                                   |
|   use_app_keepalive_timeout                        | False                                                   |
|   allow_dots_in_header_name                        | False                                                   |
|   disable_keepalive_posts_msie6                    | True                                                    |
|   enable_request_body_buffering                    | False                                                   |
|   enable_fire_and_forget                           | False                                                   |
|   max_response_headers_size                        | 48 kb                                                   |
|   http2_enabled                                    | True                                                    |
|   respond_with_100_continue                        | True                                                    |
|   hsts_subdomains_enabled                          | True                                                    |
|   enable_request_body_metrics                      | False                                                   |
|   fwd_close_hdr_for_bound_connections              | True                                                    |
|   max_keepalive_requests                           | 100                                                     |
|   disable_sni_hostname_check                       | False                                                   |
|   max_http2_control_frames_per_connection          | 1000                                                    |
|   max_http2_queued_frames_to_client_per_connection | 1000                                                    |
|   max_http2_empty_data_frames_per_connection       | 1000                                                    |
|   max_http2_concurrent_streams_per_connection      | 128                                                     |
|   reset_conn_http_on_ssl_port                      | False                                                   |
| preserve_client_ip                                 | False                                                   |
| preserve_client_port                               | False                                                   |
| tenant_ref                                         | admin                                                   |
+----------------------------------------------------+---------------------------------------------------------+
[admin:th-controller-3]: >
[admin:th-controller-3]: applicationprofile> configure applicationprofile applicationprofile-22
[admin:th-controller-3]: applicationprofile> http_profile
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_control_frames_per_connection 2000
Overwriting the previously entered value for max_http2_control_frames_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_queued_frames_to_client_per_connection 2000
Overwriting the previously entered value for max_http2_queued_frames_to_client_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_empty_data_frames_per_connection 2000
Overwriting the previously entered value for max_http2_empty_data_frames_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> max_http2_concurrent_streams_per_connection 256
Overwriting the previously entered value for max_http2_concurrent_streams_per_connection
[admin:th-controller-3]: applicationprofile:http_profile> save
[admin:th-controller-3]: applicationprofile> save
[admin:th-controller-3]: > show applicationprofile applicationprofile-22
+----------------------------------------------------+---------------------------------------------------------+
| Field                                              | Value                                                   |
+----------------------------------------------------+---------------------------------------------------------+
| uuid                                               | applicationprofile-8cd83b63-9ef8-4ce3-b498-2d173a09ed7c |
| name                                               | applicationprofile-22                                   |
| type                                               | APPLICATION_PROFILE_TYPE_HTTP                           |
| http_profile                                       |                                                         |
|   connection_multiplexing_enabled                  | True                                                    |
|   xff_enabled                                      | False                                                   |
|   xff_alternate_name                               | X-Forwarded-For                                         |
|   ssl_everywhere_enabled                           | False                                                   |
|   hsts_enabled                                     | False                                                   |
|   hsts_max_age                                     | 365                                                     |
|   secure_cookie_enabled                            | False                                                   |
|   httponly_enabled                                 | False                                                   |
|   http_to_https                                    | False                                                   |
|   server_side_redirect_to_https                    | False                                                   |
|   x_forwarded_proto_enabled                        | False                                                   |
|   spdy_enabled                                     | False                                                   |
|   spdy_fwd_proxy_mode                              | False                                                   |
|   post_accept_timeout                              | 30000 milliseconds                                      |
|   client_header_timeout                            | 10000 milliseconds                                      |
|   client_body_timeout                              | 30000 milliseconds                                      |
|   keepalive_timeout                                | 30000 milliseconds                                      |
|   client_max_header_size                           | 12 kb                                                   |
|   client_max_request_size                          | 48 kb                                                   |
|   client_max_body_size                             | 0 kb                                                    |
|   max_rps_unknown_uri                              | 0                                                       |
|   max_rps_cip                                      | 0                                                       |
|   max_rps_uri                                      | 0                                                       |
|   max_rps_cip_uri                                  | 0                                                       |
|   ssl_client_certificate_mode                      | SSL_CLIENT_CERTIFICATE_NONE                             |
|   websockets_enabled                               | True                                                    |
|   max_rps_unknown_cip                              | 0                                                       |
|   max_bad_rps_cip                                  | 0                                                       |
|   max_bad_rps_uri                                  | 0                                                       |
|   max_bad_rps_cip_uri                              | 0                                                       |
|   keepalive_header                                 | False                                                   |
|   use_app_keepalive_timeout                        | False                                                   |
|   allow_dots_in_header_name                        | False                                                   |
|   disable_keepalive_posts_msie6                    | True                                                    |
|   enable_request_body_buffering                    | False                                                   |
|   enable_fire_and_forget                           | False                                                   |
|   max_response_headers_size                        | 48 kb                                                   |
|   http2_enabled                                    | True                                                    |
|   respond_with_100_continue                        | True                                                    |
|   hsts_subdomains_enabled                          | True                                                    |
|   enable_request_body_metrics                      | False                                                   |
|   fwd_close_hdr_for_bound_connections              | True                                                    |
|   max_keepalive_requests                           | 100                                                     |
|   disable_sni_hostname_check                       | False                                                   |
|   max_http2_control_frames_per_connection          | 2000                                                    |
|   max_http2_queued_frames_to_client_per_connection | 2000                                                    |
|   max_http2_empty_data_frames_per_connection       | 2000                                                    |
|   max_http2_concurrent_streams_per_connection      | 256                                                     |
|   reset_conn_http_on_ssl_port                      | False                                                   |
| preserve_client_ip                                 | False                                                   |
| preserve_client_port                               | False                                                   |
| tenant_ref                                         | admin                                                   |
+----------------------------------------------------+---------------------------------------------------------+
[admin:th-controller-3]: >