Configure the HTTP/S Layer 7 Ingress Controller

This topic describes how to define network profiles for Kubernetes clusters provisioned with VMware Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the CRD, creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

One TCP layer 4 load balancer virtual server for the Kubernetes API server.
One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller, below. For information about configuring layer 7 ingress routing load balancers see Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD.

For information about configuring TCP layer 4 ingress routing load balancers see Defining Network Profiles for the TCP Layer 4 Load Balancer.

For more information about the NSX-T Load Balancer, see Create an IP Pool in Manager Mode or Add an IP Address Pool in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Configure the HTTP/HTTPS Ingress Controller Network Profile

The HTTP/HTTPS layer 7 virtual servers provisioned for each Kubernetes service are controlled by the parameters exposed in a network profile.

NSX-T HTTP/HTTPS Ingress Controller Network Profile Configuration

The NSX Ingress Controller is configured using the ncp.ini network profile configuration file.

The HTTP/HTTPS Ingress Controller network profile has the following format:

{
  "name": "ncp_network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": NSX-LB,   
           "nsx_ingress_controller": NCP-IC,        
           "ingress_ip": "IP-ADDRESS",
           "ingress_persistence_settings": {
             "persistence_type": "PERS-TYPE",
             "persistence_timeout": TIMEOUT 
             }
           }
      }
    }
}

Where:

DESCRIP is your description for this network profile configuration.
NSX-LB is your preference for whether the NSX-T Load Balancer is used for your Kubernetes clusters. For more information see Configure the NSX Ingress Controller, below.
NCP-IC is your preference for whether the NCP is used as the Ingress Controller for your Kubernetes clusters.
IP-ADDRESS is IP address to use for ingress controller load balancer. For more information see Configure the Ingress IP, below.
PERS-TYPE is the persistence type to use for ingress controller load balancer. For more information see Configure the Ingress Persistence Settings, below.
TIMEOUT is the persistence timeout to use for ingress controller load balancer.
For more information see Configure the Ingress Persistence Settings, below.

For example:

{
  "name": "ncp_network_profile",
  "description": "Example network profile for ingress controller",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": true,  
           "nsx_ingress_controller": false,        
           "ingress_ip": "192.168.160.212",
           "ingress_persistence_settings": {
             "persistence_type": "cookie",
             "persistence_timeout": 1 
             }
           }
      }
    }
}

The following table describes the Ingress Controller configuration parameters:

Parameter	Type	Description
`name`	String	User-defined name of the network profile.
`description`	String	User-defined description for the network profile.
`parameters`	Map	One or more name-value pairs.
`cni_configurations`	Map	Map containing two key-value pairs: `type` and `parameters`.
`type`	Constant String	Only `nsxt` is accepted.
`parameters`	Map	Map containing one or more key-value pairs for NCP settings.
`nsx_lb`	Boolean	Flag to control if the NSX-T Load Balancer is used for the Kubernetes cluster.
`nsx_ingress_controller`	Boolean	Flag to control if NCP is used as the Ingress Controller for the Kubernetes cluster.
`ingress_ip`	String	IP address to use for ingress controller load balancer.
`ingress_persistence_settings`	Map	Holds parameters for customizing Layer 7 persistence.
`persistence_type`	String	Valid values are `cookie` or `source_ip`. An empty value is not accepted.
`persistence_timeout`	Integer	Value that is equal to `1` or larger. Empty value is not accepted.

The nsx_lb parameter is used to control the TCP layer 4 virtual server that is provisioned for each Kubernetes service of type: LoadBalancer.

The nsx_ingress_controller parameter is used to control if NCP is used as the Ingress Controller for the Kubernetes cluster. By default when you define an ingress resource for a Kubernetes cluster, NCP instructs the NSX-T load balancer to provision 2 layer 7 virtual services (HTTP and HTTPS) as the Ingress Controller.

The nsx_ingress_controller parameter is subject to the nsx_lb parameter as described in the following table:

nsx_lb setting	nsx_ingress_controller: `true`	nsx_ingress_controller: `false`
nsx_lb: `true`	Use the NSX-T Layer 4 LoadBalancer and the NCP-provisioned Layer 7 Ingress Controller.	Use the NSX-T Layer 4 LoadBalancer and a third-party Ingress Controller, such as NGINX.
nsx_lb: `false`	Invalid configuration. You cannot deactivate the NSX-T Load Balancer and use NCP as the Ingress Controller. The network profile will fail validation.	Use a third-party load balancer and a third-party ingress controller.

Configure the NSX Ingress Controller

NCP depends on the NSX-T Load Balancer to fulfill its role as an Ingress Controller. To use a third-party ingress controller, such as the NGINX Ingress Controller, set the nsx_ingress_controller to false.

For example:

The following network profile uses the NSX-T Load Balancer and a third-party ingress controller:

{
  "name": "example_network_profile",
  "description": "Use the nsx_lb with a 3rd party ingress controller",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": true,
         "nsx_ingress_controller": false
      }
    }
  }
}

The following network profile uses a third party load balancer and a third-party ingress controller:

{
  "name": "example_network_profile",
  "description": "Use the nsx_lb with a 3rd party ingress controller",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": false,
         "nsx_ingress_controller": false
      }
    }
  }
}

You should not deactivate nsx_lb and use the NCP Ingress Controller. Using the NCP Ingress Controller with nsx_lb deactivated is invalid.

For example, the following is invalid:

{
  "name": "example_network_profile",
  "description": "Use the nsx_lb with a 3rd party ingress controller",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": false,
         "nsx_ingress_controller": true
      }
    }
  }
}

Configure the Ingress IP

The ingress_ip parameter instructs NCP to create an ingress virtual server with the given IP address.

The ingress_ip parameter type is a string that accepts any valid IP address. Missing entry is accepted.

Example network profile for ingress_ip:

{
  "name": "example-network-profile",
  "description": "ingress_ip",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"
         }
     }
  }
}

An invalid IP address is rejected with an invalid parameter value error.

For example:

The following network profile parameters cannot be parsed because the "ingress_ip" configuration specifies an invalid IP address:

{
  "name": "example-network-profile",
  "description": "ingress_ip-ERROR",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.460.212"
         }
     }
  }
}

The following network profile cannot be parsed because the "ingress_ip" configuration is not a string and the JSON input is invalid:

{
  "name": "example-network-profile",
  "description": "ingress_ip-ERROR",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": 192.168.160.212
         }
     }
  }
}

Configure the Ingress Persistence Settings

The ingress_persistence parameter lets you customize layer 7 persistence for Kubernetes services.

The ingress_persistence_settings parameter is a map that supports two keys:

persistence_type
persistence_timeout

These two keys are correlated and must be set/unset at the same time. If persistence_type and persistence_timeout are not both specified, the network profile fails validation.

Parameter	Data Type	Description
`persistence_type`	String	Valid values are `cookie` or `source_ip`. An empty value is not accepted.
`persistence_timeout`	Integer	Value that is equal to `1` or larger. Empty value is not accepted.

For example:

Network profile for ingress_persistence_settings:

{
  "name": "example_network_profile",
  "description": "ingress_persistence_settings",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"       
           "ingress_persistence_settings": {
             "persistence_type": "cookie",
             "persistence_timeout": 1
            }
         }
     }
  }
}

Network profile for ingress_persistence_settings:

{
  "name": "example_network_profile",
  "description": "ingress_persistence_settings",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"         
           "ingress_persistence_settings": {
             "persistence_type": "source_ip",
             "persistence_timeout": 100
            }
         }
     }
  }
}