Configure the HTTP/S Layer 7 Ingress Controller

This topic describes how to define network profiles for VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) provisioned Kubernetes clusters on vSphere with NSX.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the Kubernetes CustomResourceDefinition (CRD), creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

One TCP layer 4 load balancer virtual server for the Kubernetes API server.
One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller, below. For information about configuring layer 7 ingress routing load balancers see Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD.

For information about configuring TCP layer 4 ingress routing load balancers see Defining Network Profiles for the TCP Layer 4 Load Balancer.

For more information about the NSX-T Load Balancer, see Create an IP Pool in Manager Mode or Add an IP Address Pool in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Configure the HTTP/HTTPS Ingress Controller Network Profile

The HTTP/HTTPS layer 7 virtual servers provisioned for each Kubernetes service are controlled by the parameters exposed in a network profile.

NSX-T HTTP/HTTPS Ingress Controller Network Profile Configuration

The NSX Ingress Controller is configured using the ncp.ini network profile configuration file.

The HTTP/HTTPS Ingress Controller network profile has the following format:

{
  "name": "ncp_network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": NSX-LB,   
           "ingress_ip": "IP-ADDRESS",
           "ingress_persistence_settings": {
             "persistence_type": "PERS-TYPE",
             "persistence_timeout": TIMEOUT 
             }
           }
      }
    }
}

Where:

DESCRIP is your description for this network profile configuration.
NSX-LB is your preference for whether the NSX-T Load Balancer is used for your Kubernetes clusters. For more information, see Configure the NSX Ingress Controller below.
IP-ADDRESS is IP address to use for ingress controller load balancer. For more information, see Configure the Ingress IP below.
PERS-TYPE is the persistence type to use for ingress controller load balancer. For more information, see Configure the Ingress Persistence Settings below.
TIMEOUT is the persistence timeout to use for ingress controller load balancer.
For more information, see Configure the Ingress Persistence Settings below.

For example:

{
  "name": "ncp_network_profile",
  "description": "Example network profile for ingress controller",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": true,  
           "ingress_ip": "192.168.160.212",
           "ingress_persistence_settings": {
             "persistence_type": "cookie",
             "persistence_timeout": 1 
             }
           }
      }
    }
}

The following table describes the Ingress Controller configuration parameters:

Parameter	Type	Description
`name`	String	User-defined name of the network profile.
`description`	String	User-defined description for the network profile.
`parameters`	Map	Map containing one or more key-value pairs.
`cni_configurations`	Map	Map containing `type` and `parameters` key-value pairs for configuring NCP.
`type`	Constant String	Values: `“nsxt”`.
`parameters`	Map	Map containing one or more key-value pairs for NCP settings.
`nsx_lb`	Boolean Updatable	Use NSX-T layer 4 virtual server for each Kubernetes service of type LoadBalancer. Values: `true`, `false`. Default: `true`.
`nsx_ingress_controller`	Boolean	Use NSX-T layer 7 virtual server as the ingress controller for the Kubernetes cluster. Values: `true`, `false`. Default: `true`.
`ingress_ip`	String	IP address to use for ingress controller load balancer.
`ingress_persistence_settings`	String Updatable	Map containing one or more key-value pairs for customizing Layer 7 persistence. See also: `persistence_timeout` and `persistence_type`
`persistence_type`	String Updatable	An `ingress_persistence_settings` parameter. Specify the ingress persistence type. Values: `“none”`, `“cookie”`, `“source_ip”`.
`persistence_timeout`	Integer Updatable	An `ingress_persistence_settings` parameter. Persistence timeout interval in seconds. See also: `connect_retry_timeout` and `lb_http_response_timeout`.

The nsx_lb parameter is used to control the TCP layer 4 virtual server that is provisioned for each Kubernetes service of type: LoadBalancer.

When you configure an NSX Load Balancer as your Kubernetes cluster ingress resource, NCP instructs the NSX-T Load Balancer to provision two layer 7 virtual services (HTTP and HTTPS) as the cluster Ingress Controller:

nsx_lb setting	Description
nsx_lb: `true`	Use an NSX-T Layer 4 LoadBalancer and NCP-provisioned Layer 7 Ingress Controller.
nsx_lb: `false`	Use a third-party load balancer and a third-party ingress controller, such as NGINX.

Configure the NSX Ingress Controller

NCP depends on the NSX-T Load Balancer to fulfill its role as an Ingress Controller. To use a third-party ingress controller, such as the NGINX Ingress Controller, set nsx_lb to false.

For example:

The following network profile uses the NSX-T Load Balancer and NSX Ingress Controller:

{
  "name": "example_network_profile",
  "description": "Use the NSX-T Load Balancer and NSX Ingress Controller",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": true
      }
    }
  }
}

The following network profile uses a third party load balancer and a third-party ingress controller:

{
  "name": "example_network_profile",
  "description": "Use a 3rd party load balancer and ingress controller",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": false
      }
    }
  }
}

Configure the Ingress IP

The ingress_ip parameter instructs NCP to create an ingress virtual server with the given IP address.

The ingress_ip parameter type is a string that accepts any valid IP address. Missing entry is accepted.

Example network profile for ingress_ip:

{
  "name": "example-network-profile",
  "description": "ingress_ip",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"
         }
     }
  }
}

An invalid IP address is rejected with an invalid parameter value error.

For example:

The following network profile parameters cannot be parsed because the "ingress_ip" configuration specifies an invalid IP address:

{
  "name": "example-network-profile",
  "description": "ingress_ip-ERROR",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.460.212"
         }
     }
  }
}

The following network profile cannot be parsed because the "ingress_ip" configuration is not a string and the JSON input is invalid:

{
  "name": "example-network-profile",
  "description": "ingress_ip-ERROR",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": 192.168.160.212
         }
     }
  }
}

Configure the Ingress Persistence Settings

The ingress_persistence parameter lets you customize layer 7 persistence for Kubernetes services.

The ingress_persistence_settings parameter is a map that supports two keys:

persistence_type
persistence_timeout

These two keys are correlated and must be set/unset at the same time. If persistence_type and persistence_timeout are not both specified, the network profile fails validation.

Parameter	Data Type	Description
`persistence_type`	String	Valid values are `“none”`, `“cookie”` or `“source_ip”`. An empty value is not accepted.
`persistence_timeout`	Integer	Value that is equal to `1` or larger. Empty value is not accepted.

For example:

Network profile for ingress_persistence_settings:

{
  "name": "example_network_profile",
  "description": "ingress_persistence_settings",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"       
           "ingress_persistence_settings": {
             "persistence_type": "cookie",
             "persistence_timeout": 1
            }
         }
     }
  }
}

Network profile for ingress_persistence_settings:

{
  "name": "example_network_profile",
  "description": "ingress_persistence_settings",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"         
           "ingress_persistence_settings": {
             "persistence_type": "source_ip",
             "persistence_timeout": 100
            }
         }
     }
  }
}