Configure the TCP Layer 4 Load Balancer

This topic describes how to define network profile to configure the NSX-T Load Balancer for VMware Tanzu Kubernetes Grid Integrated Edition.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the CRD, creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

One TCP layer 4 load balancer virtual server for the Kubernetes API server.
One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring TCP layer 4 ingress controller see Configure the TCP Ingress Controller Network Profile, below.

For information about configuring layer 7 ingress routing load balancers see Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD. For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller.

For more information about the NSX-T Load Balancer, see Create an IP Pool in Manager Mode or Add an IP Address Pool in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Configure the TCP Ingress Controller Network Profile

The TCP layer 4 virtual server provisioned for each Kubernetes service is controlled by the parameters exposed in a network profile.

Note: The TCP layer 4 virtual server that fronts the Kubernetes API server is always created, and it is not controlled by the parameters exposed in the network profile.

NSX-T TCP Ingress Controller Network Profile Configuration

The NSX Ingress Controller is configured using the ncp.ini network profile configuration file.

The TCP Ingress Controller network profile has the following format:

{
  "name": "network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": NSX-LB,        
           "x_forwarded_for": "FORWARD-TYPE",
           "max_l4_lb_service": MAX-SERVERS,
           "l4_persistence_type": "source_ip",
           "l4_lb_algorithm": "LB-BEHAVIOR"
           }
       }
    }
}

Where:

DESCRIP is your description for this network profile configuration.
NSX-LB is your preference for whether the NSX-T Load Balancer is used for your Kubernetes clusters. For more information see Configure Which NSX Load Balancer to Use, below.
FORWARD-TYPE (Optional) is your request header original client source IP. For more information see the Configure the Client Source IP Address, below.
MAX-SERVERS is your maximum number of layer 4 virtual servers per cluster. For more information see the Configure the Maximum Number of Layer 4 Load Balancer Virtual Servers, below.
LB-BEHAVIOR (Optional) is your load balancer behavior. For more information see the Configure the Layer 4 Load Balancer Algorithm, below.

For example:

{
  "name": "network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": true,        
           "x_forwarded_for": "replace",
           "max_l4_lb_service": 10,
           "l4_persistence_type": "source_ip",
           "l4_lb_algorithm": "weighted_round_robin"
           }
       }
    }
}

The following table describes the Ingress Controller configuration parameters:

Parameter	Type	Description
`name`	String	User-defined name of the network profile.
`description`	String	User-defined description for the network profile.
`parameters`	Map	One or more name-value pairs.
`cni_configurations`	Map	Map containing two key-value pairs: `type` and `parameters`.
`type`	Constant String	Only `nsxt` is accepted.
`parameters`	Map	Map containing one or more key-value pairs for NCP settings.
`nsx_lb`	Boolean	Flag to control if the NSX-T Load Balancer is used for the Kubernetes cluster.
`x_forwarded_for`	String	Sets the original client source IP in the request header. Default is none. Accepts `“insert”` and `“replace”`. Enabling the network profile `x_forwarded_for` parameter automatically enables the `x_forwarded_port` and `x_forward_protocol` parameters.
`max_l4_lb_service`	Integer	Limit the maximum number of layer 4 virtual servers per cluster. Minimum is `1`.
`l4_persistence_type`	String	Connection stickiness based on `source_ip` (only accepted value).
`l4_lb_algorithm`	String	Specify the layer 4 load balancer behavior. Accepts `“round_robin”` (default), `“least_connection”`, `“ip_hash”`, `“weighted_round_robin”`.

The nsx_lb parameter is used to control the TCP layer 4 virtual server that is provisioned for each Kubernetes service of type: LoadBalancer.

The nsx_ingress_controller parameter is used to control if NCP is used as the Ingress Controller for the Kubernetes cluster. By default when you define an ingress resource for a Kubernetes cluster, NCP instructs the NSX-T load balancer to provision 2 layer 7 virtual services (HTTP and HTTPS) as the Ingress Controller.

The nsx_ingress_controller parameter is subject to the nsx_lb parameter as described in the following table:

nsx_lb setting	nsx_ingress_controller: `true`	nsx_ingress_controller: `false`
nsx_lb: `true`	Use the NSX-T Layer 4 LoadBalancer and the NCP-provisioned Layer 7 Ingress Controller.	Use the NSX-T Layer 4 LoadBalancer and a third-party Ingress Controller, such as NGINX.
nsx_lb: `false`	Invalid configuration. You cannot disable the NSX-T Load Balancer and use NCP as the Ingress Controller. The network profile will fail validation.	Use a third-party load balancer and a third-party ingress controller.

Configure Which NSX Load Balancer to Use

The nsx_lb flag controls whether to deploy either the NSX-T Load Balancer or a third-party load balancer, such as Nginx.
The nsx_lb parameter accepts true or false. The default setting is true and the NSX-T Load Balancer is deployed. To use a third party load balancer, set this parameter to false.

For example:

{
  "name": "example_network_profile",
  "description": "nsx_lb and nsx_ingress_controller are enabled",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": false,
         "nsx_ingress_controller": false
      }
    }
  }
}

Note: The nsx_lb parameter maps to the use_native_loadbalancer parameter in NCP.ini.

Since the nsx_ingress_controller is a component of the NSX-T Load Balancer, if you choose to use a third-party load balancer (nsx_lb: false), you must also explicitly disable the nsx_ingress_controller. See Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller.

Configure the Client Source IP Address

The X-Forwarded-For HTTP header field is used to identify the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.

In network profile, the x_forwarded_for parameter can be enabled to ensure that the client IP address will be set in the HTTP request header. The x_forwarded_for parameter is useful in situations where it is important to know the source IP address of the client request, such as for auditing purposes.

Note: Enabling the network profile x_forwarded_for parameter automatically enables the x_forwarded_port and x_forward_protocol parameters. This is supported with NSX-T v3.0.x and later.

The x_forwarded_for parameter type is String that accepts "insert" and "replace". Any other type will be rejected. Missing entry is accepted.

If x_forwarded_for is set to “insert”, the client source IP will be appended (comma separated) to the existing set of client source IP addresses. If x_forwarded_for is set to “replace”, any existing client source IP address will be replaced with the current client source IP address.

For example, with the following network profile the source IP address of the client will be appended to the existing set of client source IP addresses:

{
  "name": "example-network-profile",
  "description": "x_forwarded_for insert",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "x_forwarded_for": "insert"
      }
    }
  }
}

For example, with the following network profile the existing source IP address of the client will replace all other client source IP entries:

{
  "name": "example-network-profile",
  "description": "x_forwarded_for replace",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "x_forwarded_for": "replace"
      }
    }
  }
}

Configure the Maximum Number of Layer 4 Load Balancer Virtual Servers

The default NSX-T Load Balancer behavior is that auto-scaling is unlimited. This means that the number of layer 4 virtual servers that can be deployed is governed only by the capacity of the Edge Cluster where the load balancer service is deployed. As a result, in theory it is possible for a single Kubernetes cluster to use up all of the layer 4 virtual servers that the Edge Cluster can support.

The max_l4_service parameter sets the upper limit for the number of virtual servers that can be used by a Kubernetes cluster. You can use this parameter to limit the number of virtual servers that can be created per Kubernetes cluster.

The max_l4_lb_service data type is an integer. The value must be larger or equal to 1. Missing entry is accepted.

For example, the following network profile uses the max_l4_lb_service parameter to limit the number of layer 4 virtual servers to 100 per cluster:

{
  "name": "example_network_profile",
  "description": "max_l4_lb_service",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "max_l4_lb_service": 100
         }
     }
  }
}

Configure the Layer 4 Persistence Type

The l4_persistence_type is used to set connection stickiness based on source_ip.

The l4_persistence_type data type is string. The only accepted value is source_ip.

{
  "name": "example_network_profile",
  "description": "l4_persistence_type",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
        "l4_persistence_type": "source_ip"
      }
    }
  }
}

Configure the Layer 4 Load Balancer Algorithm

The l4_lb_algorithm is used to set the algorithm type for the layer 4 NSX-T Load Balancer service.

The l4_lb_algorithm data type is string enumeration that accepts one of the following values:

"round_robin" (default)
"least_connection"
"ip_hash"
"weighted_round_robin"

For example, the following network profile specifies the weighted_round_robin as the load balancer algorithm:

{
  "name": "example_network_profile",
  "description": "l4_lb_algorithm",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
        "l4_lb_algorithm": "weighted_round_robin"
      }
    }
  }
}