Configuring load balancing involves configuring a Kubernetes LoadBalancer service or Ingress resource, and the NCP replication controller.

You can create a layer 4 load balancer by configuring a Kubernetes service of type LoadBalancer, and a layer 7 load balancer by configuring a Kubernetes Ingress resource.

To configure load balancing in NCP, in the ncp-rc.yml file:

  1. Set use_native_loadbalancer = True.

  2. (Optional) Set pool_algorithm to ROUND_ROBIN or LEAST_CONNECTION/IP_HASH. The default is ROUND_ROBIN.

  3. (Optional) Set service_size = SMALL, MEDIUM, or LARGE. The default is SMALL.

The LEAST_CONNECTION/IP_HASH algorithm means that traffic from the same source IP address will be sent to the same backend pod.

The small NSX-T Data Center load balancer supports the following:

  • 10 NSX-T virtual servers.

  • 10 NSX-T pools.

  • 30 NSX-T pool members.

  • 8 ports for LoadBalancer services.

  • A total of 10 ports defined by the LoadBalancer services and Ingress resources.

  • A total of 30 endpoints referenced by the LoadBalancer services and Ingress resources.

The medium NSX-T Data Center load balancer supports the following:

  • 100 NSX-T virtual servers.

  • 100 NSX-T pools.

  • 300 NSX-T pool members.

  • 98 ports for LoadBalancer services.

  • A total of 100 ports defined by the LoadBalancer services and Ingress resources.

  • A total of 300 endpoints referenced by the LoadBalancer services and Ingress resources.

The large NSX-T Data Center load balancer supports the following:

  • 1000 NSX-T virtual servers.

  • 1000 NSX-T pools.

  • 3000 NSX-T pool members.

  • 998 ports for LoadBalancer services.

  • A total of 1000 ports defined by the LoadBalancer services and Ingress resources.

  • A total of 3000 endpoints referenced by the LoadBalancer services and Ingress resources.

After the load balancer is created, the load balancer size cannot be changed by updating the configuration file. It can be changed through the UI or API.

Setting Persistence for Layer 4 and Layer 7 Load Balancing

You can specify a persistence setting with the parameters l4_persistence and l7_persistence in the NCP ConfigMap. The available option for layer 4 persistence is source IP. The available options for layer 7 persistence are cookie and source IP. The default is <None>. For example,

   # Choice of persistence type for ingress traffic through L7 Loadbalancer.
   # Accepted values:
   # 'cookie'
   # 'source_ip'
   l7_persistence = cookie

   # Choice of persistence type for ingress traffic through L4 Loadbalancer.
   # Accepted values:
   # 'source_ip'
   l4_persistence = source_ip

Note:

If you configure both a layer 4 and a layer 7 load balancer, you can set either l4_persistence or l7_persistence, or both to source_ip, but you cannot set l4_persistence to source_ip and l7_persistence to cookie. If by mistake you set l4_persistence to source_ip and l7_persistence to cookie, the LoadBalancer service will not work. To resolve the issue, you must delete the Ingress resource and the LoadBalancer service, change the persistence settings, restart NCP, and recreate the Ingress resource and the LoadBalancer service.

Ingress

  • NSX-T Data Center will create one layer 7 load balancer for Ingresses with TLS specification, and one layer 7 load balancer for Ingresses without TLS specification.

  • All Ingresses will get a single IP address.

  • The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools option in the [nsx_v3] section in ncp.ini. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).

  • Starting with NCP 2.3.2, the Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).

  • Starting with NCP 2.3.2, you can change to a different IP pool by changing the configuration and restarting NCP.

  • You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.

  • Ingresses without TLS specification will be hosted on HTTP Virtual Server (port 80).

  • Ingresses with TLS specification will be hosted on HTTPS Virtual Server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.

  • The order of creation of secrets and Ingress does not matter. If the secret object is present and there is an Ingress referencing it, the certificate will be imported in NSX-T Data Center. If the secret is deleted or the last Ingress referencing the secret is deleted, the certificate corresponding to the secret will be deleted.

  • Modification of Ingress by adding or removing the TLS section is supported. When the tls key is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS Virtual Server (port 443) to the HTTP Virtual Server (port 80). Similarly, when the tls key is added to Ingress specification, the Ingress rules are transferred from the HTTP Virtual Server (port 80) to the HTTPS Virtual Server (port 443).

  • If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied.

  • Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.

  • If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error.

  • Wildcard URI matching is supported using the regular expression characters "." and "*". For example, the path "/coffee/.*" matches "/coffee/" followed by zero, one or more characters, such as "/coffee/", "/coffee/a", "/coffee/b", but not "/coffee", "/coffeecup" or "/coffeecup/a". Note that if the path contains "/*", for example "/tea/*", it will match "/tea" followed by zero, one or more characters, such as "/tea", "/tea/", "/teacup", "/teacup/", "/tea/a" or "/teacup/b". In this case, the regular expression special character "*" is acting as a wildcard character as well.

    An Ingress specification example:

    kind: Ingress
    metadata:
      name: cafe-ingress
    spec:
      rules:
      - http:
          paths:
          - path: /tea/*        #Matches /tea, /tea/, /teacup, /teacup/, /tea/a, /teacup/b, etc.
            backend:
              serviceName: tea-svc
              servicePort: 80
          - path: /coffee/.*    #Matches /coffee/, /coffee/a but NOT /coffee, /coffeecup, etc.
            backend:
              serviceName: coffee-svc
              servicePort: 80

  • You can configure URL request rewrite by adding an annotation to the Ingress resource. For example,

    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: cafe.example.com
        http:
          paths:
          - path: /tea
            backend:
              serviceName: tea-svc
              servicePort: 80
          - path: /coffee
            backend:
              serviceName: coffee-svc
              servicePort: 80

    The paths /tea and /coffee will be rewritten to / before the URL is sent to the backend service.

  • Starting with NCP 2.3.2, the Ingress annotation kubernetes.io/ingress.allow-http is supported.

    • If the annotation is set to false, only HTTPS rules will be created.

    • If the annotation is set to true or missing, HTTP rules will be created. Additionally, HTTPS rules will be created if the TLS section is present in the Ingress specification.

  • Error are annotated to the Ingress resource. The error key is ncp/error.loadbalancer and the warning key is ncp/warning.loadbalancer. The possible error and warning are:

    • ncp/error.loadbalancer: DEFAULT_BACKEND_IN_USE

      This error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. There can be only one default backend for a group of Ingresses with and without TLS. To fix the error, delete and recreate the Ingress with a correct specification.

    • ncp/warning.loadbalancer: SECRET_NOT_FOUND

      This error indicates that the secret specified in the Ingress specification does not exist. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.

Layer 7 Load Balancer and Network Policy

When traffic is forwarded to the pods from the NSX load balancer virtual server, the source IP is the tier-1 router's uplink port's IP address. This address is on the private tier-1 transit network, and can cause the CIDR-based network policies to disallow traffic that should be allowed. To avoid this issue, the network policy must be configured such that the tier-1 router's uplink port's IP address is part of the allowed CIDR block. This internal IP address will be visible to the user as part of the Ingress specification in the status.loadbalancer.ingress.ip field and as an annotation (ncp/internal_ip_for_policy) on the Ingress resource.

For example, if the external IP address of the virtual server is 4.4.0.5 and the IP address of the internal tier-1 router's uplink port is 100.64.224.11, the Ingress specification will be:

    kind: Ingress
    ...
    status:
      loadBalancer:
      ingress:
      - ip: 4.4.0.5
      - ip: 100.64.224.11

The annotation on the Ingress resource will be:

    ncp/internal_ip_for_policy: 100.64.224.11

The IP address 100.64.224.11 must belong to the allowed CIDR in the ipBlock selector of the network policy. For example,

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    ...
    ingress:
    - from:
      - ipBlock:
         cidr: 100.64.224.11/32

Kubernetes LoadBalancer Service and Layer 4 Load Balancer

  • NSX-T Data Center will create a layer 4 load balancer virtual server and pool for each service port.

  • Both TCP and UDP are supported.

  • Each service will have a unique IP address.

  • The service is allocated an IP address from an external IP pool. You can specify the name or ID of an IP pool with the loadBalancerIP spec in the LoadBalancer definition. The Loadbalancer service's IP will be allocated from this IP pool. If the loadBalancerIP spec is empty, the IP will be allocated from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. (Note: This option is available in NCP 2.3.2 only.) If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The LoadBalancer service is exposed on this IP address and the service port.

  • Starting with NCP 2.3.2, you can change to a different IP pool by changing the configuration and restarting NCP.

  • Starting with NCP 2.3.1, the IP pool specified by loadBalancerIP must have the tag {"ncp/owner": cluster:<cluster>}.

  • Error are annotated to a service. The error key is ncp/error.loadbalancer. The possible errors are:

    • ncp/error.loadbalancer: IP_POOL_NOT_FOUND

      This error indicates that you specify loadBalancerIP: <nsx-ip-pool> but <nsx-ip-pool> does not exist. The service will be inactive. To fix the error, specify a valid IP pool, delete and recreate the service.

    • ncp/error.loadbalancer: IP_POOL_EXHAUSTED

      This error indicates that you specify loadBalancerIP: <nsx-ip-pool> but the IP pool has exhausted its IP addresses. The service will be inactive. To fix the error, specify an IP pool that has available IP addresses, delete and recreate the service.

    • ncp/error.loadbalancer: IP_POOL_NOT_UNIQUE

      This error indicates that multiple IP pools have the name that is specified by loadBalancerIP: <nsx-ip-pool>. The service will be inactive.

    • ncp/error.loadbalancer: POOL_ACCESS_DENIED

      (For NCP 2.3.1 and later releases) This error indicates that the IP pool specified by loadBalancerIP does not have the tag {"ncp/owner": cluster:<cluster>} or the cluster specified in the tag does not match the name of the Kubernetes cluster.

  • Starting with NCP 2.3.1, automatic scaling of the layer 4 load balancer is supported. If a Kubernetes LoadBalancer service is created or modified so that it requires additional virtual servers and the existing layer 4 load balancer does not have the capacity, a new layer 4 load balancer will be created. NCP will also delete a layer 4 load balancer that no longer has virtual servers attached. This feature is enabled by default. You can disable it by setting l4_lb_auto_scaling to false in the NCP ConfigMap. This feature requires NSX-T Data Center 2.3 or later releases.

Note:

You cannot assign a specific IP address for the LoadBalancer service or the Ingress resource. Any address you specify when creating the LoadBalancer service or Ingress resource will be ignored.

Sample Script to Generate a CA-Signed Certificate

The script below generates a CA-signed certificate and a private key stored in the files <filename>.crt and <finename>.key, respectively. The genrsa command generates a CA key. The CA key should be encrypted. You can specify an encryption method with the command such as aes256.

#!/bin/bash
host="www.example.com"
filename=server

openssl genrsa -out ca.key 4096
openssl req -key ca.key -new -x509 -days 365 -sha256 -extensions v3_ca -out ca.crt -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl req -out ${filename}.csr -new -newkey rsa:2048 -nodes -keyout ${filename}.key -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl x509 -req -days 360 -in ${filename}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${filename}.crt -sha256

Mount the Default Certificate and Key into the NCP Pod

After the certificate and private key have been generated, place them in the directory /etc/nsx-ujo on the host VM. Assuming the certificate and key files are named lb-default.crt and lb-default.key, respectively, edit ncp-rc.yaml so that these files on the host are mounted into the pod. For example,

spec:
  ...
  containers:
  - name: nsx-ncp
    ...
    volumeMounts:
    ...
    - name: lb-default-cert
      # Mount path must match nsx_v3 option "lb_default_cert_path"
      mountPath: /etc/nsx-ujo/lb-default.crt
    - name: lb-priv-key
      # Mount path must match nsx_v3 option "lb_priv_key_path"
      mountPath: /etc/nsx-ujo/lb-default.key
  volumes:
  ...
  - name: lb-default-cert
    hostPath:
      path: /etc/nsx-ujo/lb-default.crt
  - name: lb-priv-key
    hostPath:
      path: /etc/nsx-ujo/lb-default.key