Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD

This topic describes how to scale ingress resources for VMware Tanzu Kubernetes Grid Integrated Edition (TKGI).

Note: This feature requires NCP v2.5.1 or later.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the Kubernetes CustomResourceDefinition (CRD), creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

• One TCP layer 4 load balancer virtual server for the Kubernetes API server.
• One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
• Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring layer 7 ingress routing load balancers see Determine Your Load Balancer’s Status, below. For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller.

For information about configuring TCP layer 4 ingress controller see Defining Network Profiles for the TCP Layer 4 Load Balancer.

For more information about the NSX-T Load Balancer, see Create an IP Pool in Manager Mode or Add an IP Address Pool in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Prerequisites

Before scaling your ingress load balancers, understand your load balancer’s status: Use the NSXLoadBalancerMonitor CRD to monitor your NSX-T load balancer service, including traffic, usage, and health score information. The NSXLoadBalancerMonitor CRD provides information for the health of the NSX-T load balancer service, and the NSX-T Edge Node running the load balancer.

For more information about monitoring using the NSXLoadBalancerMonitor CRD see Monitoring Ingress Resources.

Scale Ingress Load Balancer Resources

The LoadBalancer CRD provides you with an interactive method to scale the load balancer for ingress routing.

Create a New Ingress Load Balancer

Use the LoadBalancer CRD to create a new ingress load balancer.

1. To configure a new ingress load balancer, configure a new YAML file as follows:

   apiVersion: vmware.com/v1alpha1
   kind: LoadBalancer
   metadata:
     name: LB-NAME
   spec:
     httpConfig: HTTP-CONFIG
       virtualIP: IP-ADDRESS  
       port: PORT
       tls:
         port: TLS-PORT
         secretName: SECRET-NAME
         secretNamespace: SECRET-NAMESPACE
       xForwardedFor: FORWARD-TYPE
       affinity:
         type: IP-SOURCE
         timeout: TIMEOUT
     size: SIZE
     virtualNetwork: NETWORK-NAME
   status:
     httpVirtualIP: V-IP-ADDRESS
   

   Where:

   • LB-NAME is the display name of the loadBalancer.
   • HTTP-CONFIG (Optional) is the configuration to support http/https route on the loadBalancer. Set as httpConfig: {} to apply default settings.
   • IP-ADDRESS (Optional) is the virtual IP address. Defaults to auto_allocate.
   • PORT (Optional) is the port. Defaults to 80.
   • TLS-PORT (Optional) is the TLS port. Defaults to 443.
   • SECRET-NAME (Optional) is the TLS secret name. Defaults to nil.
   • SECRET-NAMESPACE (Optional) is the TLS secret namespace. Defaults to nil. You must deploy the new ingress load balancer in the same namespace where you deploy the ingress resource.
   • FORWARD-TYPE (Optional) is the forward type. Supported values are: INSERT and REPLACE. Defaults to nil.
   • IP-SOURCE (Optional) is the source IP. Supported values are: sourceIP and cookie.
   • TIMEOUT (Optional) is the connection timeout. Defaults to 10800.
   • SIZE (Optional) is the ingress load balancer size. Supported values are: SMALL and MEDIUM. Defaults to SMALL.
   • NETWORK-NAME (Optional) is the virtual network name. Defaults to nil.
   • V-IP-ADDRESS is the external IP address for http/https virtual server. The external IP address can be auto-allocated or user specified.

2. To create a new ingress load balancer run the following command:

   kubectl apply –f YAML-FILE
   

   Where YAML-FILE is the filename of a the load balancer configuration YAML file.

   For example:

   # kubectl apply –f lb.yaml  
   apiVersion: vmware.com/v1alpha1  
   kind: LoadBalancer  
   metadata:  
     name: cluster1_lbs0  
   spec:  
     httpConfig:  
       virtualIP:  
       port: 233  
       tls:  
         port: 2333  
         secretName: default_secret  
         secretNamespace: default  
       xForwardedFor: INSERT  
       affinity:  
         type: source_ip  
         timeout: 100  
     size: MEDIUM  
     virtualNetwork: virtualnetwork1  
   status:  
     httpVirtualIP: <realized external ip>  
   

Configure Your Kubernetes Ingress Resource to Use the New Ingress Load Balancer

Annotate the Kubernetes ingress resource with the newly created ingress load balancer. NCP will attach the ingress rules to the scaled out load balancer.

1. To configure a Kubernetes ingress resource with the new ingress load balancer, configure a new YAML file as follows:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ING-NAME
     annotations:
       kubernetes.io/ingress.class: "nsx"
       nsx/loadbalancer: LB-NAME
   spec:
     rules:
     - host: HOST-NAME
       http:
         paths:
         - path: HTTP-PATH
           pathType: PATH-TYPE
           backend:
             service:
               name: SERVICE-NAME
               port:
                 number: SERVICE-PORT
   

   Where:

   • ING-NAME is the name of the ingress resource.
   • LB-NAME is the display name of the loadBalancer.
   • HOST-NAME is the host name.
   • HTTP-PATH is the HTTP path.
   • PATH-TYPE refers to Ingress Path types in the Kubernetes documentation.
   • SERVICE-NAME is the http backend service name.
   • SERVICE-PORT is the http backend service port.

2. To annotate the Kubernetes ingress resource with the newly created ingress load balancer, run the following command:

   kubectl apply –f YAML-FILE
   

   Where YAML-FILE is the filename of a the Kubernetes ingress resource configuration YAML file.
   
   For example:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: svc-ingress1
     annotations:
       kubernetes.io/ingress.class: "nsx"
       nsx/loadbalancer: cluster1_lbs0
   spec:
     rules:
     - host: test.com
       http:
         paths:
         - path: /testpath
           pathType: Prefix
           backend:
             service:
               name: svc1
               port:
                 number: 80