This topic describes how to define network profiles for VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) provisioned Kubernetes clusters on vSphere with NSX.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the Kubernetes CustomResourceDefinition (CRD), creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

  • One TCP layer 4 load balancer virtual server for the Kubernetes API server.
  • One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
  • Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller, below. For information about configuring layer 7 ingress routing load balancers see Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD.

For information about configuring TCP layer 4 ingress routing load balancers see Defining Network Profiles for the TCP Layer 4 Load Balancer.

For more information about the NSX-T Load Balancer, see Create an IP Pool in Manager Mode or Add an IP Address Pool in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Configure the HTTP/HTTPS Ingress Controller Network Profile

The HTTP/HTTPS layer 7 virtual servers provisioned for each Kubernetes service are controlled by the parameters exposed in a network profile.

NSX-T HTTP/HTTPS Ingress Controller Network Profile Configuration

The NSX Ingress Controller is configured using the ncp.ini network profile configuration file.

The HTTP/HTTPS Ingress Controller network profile has the following format:

{
  "name": "ncp_network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": NSX-LB,   
           "ingress_ip": "IP-ADDRESS",
           "ingress_persistence_settings": {
             "persistence_type": "PERS-TYPE",
             "persistence_timeout": TIMEOUT 
             }
           }
      }
    }
}

Where:

For example:

{
  "name": "ncp_network_profile",
  "description": "Example network profile for ingress controller",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": true,  
           "ingress_ip": "192.168.160.212",
           "ingress_persistence_settings": {
             "persistence_type": "cookie",
             "persistence_timeout": 1 
             }
           }
      }
    }
}

The following table describes the Ingress Controller configuration parameters:

Parameter Type Description
name String User-defined name of the network profile.
description String User-defined description for the network profile.
parameters Map Map containing one or more key-value pairs.
cni_configurations Map Map containing type and parameters key-value pairs for configuring NCP.
type Constant
String
Values: “nsxt”.
parameters Map Map containing one or more key-value pairs for NCP settings.
nsx_lb Boolean
Updatable
Use NSX-T layer 4 virtual server for each Kubernetes service of type LoadBalancer.
Values: true, false.
Default: true.
nsx_ingress_controller Boolean Use NSX-T layer 7 virtual server as the ingress controller for the Kubernetes cluster.
Values: true, false.
Default: true.
ingress_ip String IP address to use for ingress controller load balancer.
ingress_persistence_settings String
Updatable
Map containing one or more key-value pairs for customizing Layer 7 persistence.
See also: persistence_timeout and persistence_type
persistence_type String
Updatable
An ingress_persistence_settings parameter. Specify the ingress persistence type.
Values: “none”, “cookie”, “source_ip”.
persistence_timeout Integer
Updatable
An ingress_persistence_settings parameter. Persistence timeout interval in seconds.
See also: connect_retry_timeout and lb_http_response_timeout.

The nsx_lb parameter is used to control the TCP layer 4 virtual server that is provisioned for each Kubernetes service of type: LoadBalancer.

When you configure an NSX Load Balancer as your Kubernetes cluster ingress resource, NCP instructs the NSX-T Load Balancer to provision two layer 7 virtual services (HTTP and HTTPS) as the cluster Ingress Controller:

nsx_lb setting Description
nsx_lb: true Use an NSX-T Layer 4 LoadBalancer and NCP-provisioned Layer 7 Ingress Controller.
nsx_lb: false Use a third-party load balancer and a third-party ingress controller, such as NGINX.

Configure the NSX Ingress Controller

NCP depends on the NSX-T Load Balancer to fulfill its role as an Ingress Controller. To use a third-party ingress controller, such as the NGINX Ingress Controller, set nsx_lb to false.

For example:

  • The following network profile uses the NSX-T Load Balancer and NSX Ingress Controller:

    {
      "name": "example_network_profile",
      "description": "Use the NSX-T Load Balancer and NSX Ingress Controller",
      "parameters": {
        "cni_configurations": {
          "type": "nsxt",
          "parameters": {
             "nsx_lb": true
          }
        }
      }
    }
    
  • The following network profile uses a third party load balancer and a third-party ingress controller:

    {
      "name": "example_network_profile",
      "description": "Use a 3rd party load balancer and ingress controller",
      "parameters": {
        "cni_configurations": {
          "type": "nsxt",
          "parameters": {
             "nsx_lb": false
          }
        }
      }
    }
    

Configure the Ingress IP

The ingress_ip parameter instructs NCP to create an ingress virtual server with the given IP address.

The ingress_ip parameter type is a string that accepts any valid IP address. Missing entry is accepted.

Example network profile for ingress_ip:

{
  "name": "example-network-profile",
  "description": "ingress_ip",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "ingress_ip": "192.168.160.212"
         }
     }
  }
}

An invalid IP address is rejected with an invalid parameter value error.

For example:

  • The following network profile parameters cannot be parsed because the "ingress_ip" configuration specifies an invalid IP address:

    {
      "name": "example-network-profile",
      "description": "ingress_ip-ERROR",
      "parameters" : {
         "cni_configurations": {
             "type": "nsxt",
             "parameters": {
               "ingress_ip": "192.168.460.212"
             }
         }
      }
    }
    
  • The following network profile cannot be parsed because the "ingress_ip" configuration is not a string and the JSON input is invalid:

    {
      "name": "example-network-profile",
      "description": "ingress_ip-ERROR",
      "parameters" : {
         "cni_configurations": {
             "type": "nsxt",
             "parameters": {
               "ingress_ip": 192.168.160.212
             }
         }
      }
    }
    

Configure the Ingress Persistence Settings

The ingress_persistence parameter lets you customize layer 7 persistence for Kubernetes services.

The ingress_persistence_settings parameter is a map that supports two keys:

  • persistence_type
  • persistence_timeout

These two keys are correlated and must be set/unset at the same time. If persistence_type and persistence_timeout are not both specified, the network profile fails validation.

Parameter Data Type Description
persistence_type String Valid values are “none”, “cookie” or “source_ip”. An empty value is not accepted.
persistence_timeout Integer Value that is equal to 1 or larger. Empty value is not accepted.

For example:

  • Network profile for ingress_persistence_settings:

    {
      "name": "example_network_profile",
      "description": "ingress_persistence_settings",
      "parameters" : {
         "cni_configurations": {
             "type": "nsxt",
             "parameters": {
               "ingress_ip": "192.168.160.212"       
               "ingress_persistence_settings": {
                 "persistence_type": "cookie",
                 "persistence_timeout": 1
                }
             }
         }
      }
    }
    
  • Network profile for ingress_persistence_settings:

    {
      "name": "example_network_profile",
      "description": "ingress_persistence_settings",
      "parameters" : {
         "cni_configurations": {
             "type": "nsxt",
             "parameters": {
               "ingress_ip": "192.168.160.212"         
               "ingress_persistence_settings": {
                 "persistence_type": "source_ip",
                 "persistence_timeout": 100
                }
             }
         }
      }
    }
    
check-circle-line exclamation-circle-line close-line
Scroll to top icon