Configuring load balancing involves configuring a Kubernetes LoadBalancer service or Ingress resource, and the NCP replication controller.

You can create a layer 4 load balancer by configuring a Kubernetes service of type LoadBalancer, and a layer 7 load balancer by configuring a Kubernetes Ingress resource.

To configure load balancing in NCP, in the ncp-rc.yml file:

  1. Set use_native_loadbalancer = True.
  2. (Optional) Set pool_algorithm to ROUND_ROBIN or LEAST_CONNECTION/IP_HASH. The default is ROUND_ROBIN.
  3. (Optional) Set service_size = SMALL, MEDIUM, or LARGE. The default is SMALL.

The LEAST_CONNECTION/IP_HASH algorithm means that traffic from the same source IP address will be sent to the same backend pod.

For details about what NSX-T load balancers of different sizes support, see the NSX-T Data Center Administration Guide.

After the load balancer is created, the load balancer size cannot be changed by updating the configuration file. It can be changed through the NSX Manager UI or API.

Setting Persistence for Layer 4 and Layer 7 Load Balancing

You can specify a persistence setting with the parameters l4_persistence and l7_persistence in the NCP ConfigMap. The available option for layer 4 persistence is source IP. The available options for layer 7 persistence are cookie and source IP. The default is <None>. For example,
   # Choice of persistence type for ingress traffic through L7 Loadbalancer.
   # Accepted values:
   # 'cookie'
   # 'source_ip'
   l7_persistence = cookie

   # Choice of persistence type for ingress traffic through L4 Loadbalancer.
   # Accepted values:
   # 'source_ip'
   l4_persistence = source_ip

For a Kubernetes LoadBalancer service, you can also specify sessionAffinity on the service spec to configure persistence behavior for the service if the global layer 4 persistence is turned off, that is, l4_persistence is set to <None>. If l4_persistence is set to source_ip, the sessionAffinity on the service spec can be used to customize the persistence timeout for the service. The default layer 4 persistence timeout is 10800 seconds (same as that specified in the Kubernetes documentation for services (https://kubernetes.io/docs/concepts/services-networking/service). All services with default persistence timeout will share the same NSX-T load balancer persistence profile. A dedicated profile will be created for each service with a non-default persistence timeout.

Note: If the backend service of an Ingress is a service of type LoadBalancer, then the layer 4 virtual server for the service and the layer 7 virtual server for the Ingress cannot have different persistence settings, for example, source_ip for layer 4 and cookie for layer 7. In such a scenario, the persistence settings for both virtual servers must be the same ( source_ip, cookie, or None), or one of them is None (then the other setting can be source_ip or cookie). An example of such a scenario:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
-----
apiVersion: v1
kind: Service
metadata:
  name: tea-svc <==== same as the Ingress backend above
  labels:
    app: tea
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: tcp
  selector:
    app: tea
  type: LoadBalancer

Ingress

  • NSX-T Data Center will create one layer 7 load balancer for Ingresses with TLS specification, and one layer 7 load balancer for Ingresses without TLS specification.
  • All Ingresses will get a single IP address.
  • The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools option in the [nsx_v3] section in ncp.ini. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
  • The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
  • You can change to a different IP pool by changing the configuration and restarting NCP.
  • You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.
  • Ingresses without TLS specification will be hosted on HTTP virtual server (port 80).
  • Ingresses with TLS specification will be hosted on HTTPS virtual server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.
  • The order of creation of secrets and Ingress does not matter. If the secret object is present and there is an Ingress referencing it, the certificate will be imported in NSX-T Data Center. If the secret is deleted or the last Ingress referencing the secret is deleted, the certificate corresponding to the secret will be deleted.
  • Modification of Ingress by adding or removing the TLS section is supported. When the tls key is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS virtual server (port 443) to the HTTP virtual server (port 80). Similarly, when the tls key is added to Ingress specification, the Ingress rules are transferred from the HTTP virtual server (port 80) to the HTTPS virtual server (port 443).
  • If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied.
  • Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.
  • If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error.
  • Wildcard URI matching is supported using the regular expression characters "." and "*". For example, the path "/coffee/.*" matches "/coffee/" followed by zero, one or more characters, such as "/coffee/", "/coffee/a", "/coffee/b", but not "/coffee", "/coffeecup" or "/coffeecup/a".
    An Ingress specification example:
    kind: Ingress
    metadata:
      name: cafe-ingress
    spec:
      rules:
      - http:
          paths:
          - path: /coffee/.*    #Matches /coffee/, /coffee/a but NOT /coffee, /coffeecup, etc.
            backend:
              serviceName: coffee-svc
              servicePort: 80
  • You can configure URL request rewrite by adding an annotation to the Ingress resource. For example,
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: cafe.example.com
        http:
          paths:
          - path: /tea
            backend:
              serviceName: tea-svc
              servicePort: 80
          - path: /coffee
            backend:
              serviceName: coffee-svc
              servicePort: 80

    The paths /tea and /coffee will be rewritten to / before the URL is sent to the backend service.

  • The Ingress annotation kubernetes.io/ingress.allow-http is supported.
    • If the annotation is set to false, only HTTPS rules will be created.
    • If the annotation is set to true or missing, HTTP rules will be created. Additionally, HTTPS rules will be created if the TLS section is present in the Ingress specification.
  • Errors are annotated to the Ingress resource. The error key is ncp/error.loadbalancer and the warning key is ncp/warning.loadbalancer. The possible error and warning are:
    • ncp/error.loadbalancer: DEFAULT_BACKEND_IN_USE

      This error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. There can be only one default backend for a group of Ingresses with and without TLS. To fix the error, delete and recreate the Ingress with a correct specification.

    • ncp/warning.loadbalancer: SECRET_NOT_FOUND

      This error indicates that the secret specified in the Ingress specification does not exist. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.

    • ncp/warning.loadbalancer: INVALID_INGRESS
      This error indicates that one of the following conditions is true. The Ingress will be inactive. To fix the error, delete and recreate the Ingress with a correct specification.
      • An Ingress rule conflicts with another Ingress rule in the same Kubernetes cluster.
      • The allow-http annotation is set to False and the Ingress does not have a TLS section.
      • An Ingress rule does not have host and path specified. Such an Ingress rule has the same functionality as the Ingress default backend. Use the Ingress default backend instead.

Service of Type LoadBalancer

  • NSX-T Data Center will create a layer 4 load balancer virtual server and pool for each service port.
  • Both TCP and UDP are supported.
  • Each service will have a unique IP address.
  • The service is allocated an IP address from an external IP pool based on the loadBalancerIP field in the LoadBalancer definition. The loadBalancerIP field can be empty, have an IP address or the name or ID of an IP pool. If the loadBalancerIP field is empty, the IP will be allocated from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The LoadBalancer service is exposed on this IP address and the service port.
  • You can change to a different IP pool by changing the configuration and restarting NCP.
  • The IP pool specified by loadBalancerIP must have the tag scope: ncp/owner, tag: cluster:<cluster_name>.

  • Error are annotated to a service. The error key is ncp/error.loadbalancer. The possible errors are:
    • ncp/error.loadbalancer: IP_POOL_NOT_FOUND

      This error indicates that you specify loadBalancerIP: <nsx-ip-pool> but <nsx-ip-pool> does not exist. The service will be inactive. To fix the error, specify a valid IP pool, delete and recreate the service.

    • ncp/error.loadbalancer: IP_POOL_EXHAUSTED

      This error indicates that you specify loadBalancerIP: <nsx-ip-pool> but the IP pool has exhausted its IP addresses. The service will be inactive. To fix the error, specify an IP pool that has available IP addresses, delete and recreate the service.

    • ncp/error.loadbalancer: IP_POOL_NOT_UNIQUE

      This error indicates that multiple IP pools have the name that is specified by loadBalancerIP: <nsx-ip-pool>. The service will be inactive.

    • ncp/error.loadbalancer: POOL_ACCESS_DENIED

      This error indicates that the IP pool specified by loadBalancerIP does not have the tag scope: ncp/owner, tag: cluster:<cluster_name> or the cluster specified in the tag does not match the name of the Kubernetes cluster.

    • ncp/error.loadbalancer: LB_VIP_CONFLICT

      This error indicates that the IP in the loadBalancerIP field is the same as the IP of an active service. The service will be inactive.

  • Automatic scaling of the layer 4 load balancer is supported. If a Kubernetes LoadBalancer service is created or modified so that it requires additional virtual servers and the existing layer 4 load balancer does not have the capacity, a new layer 4 load balancer will be created. NCP will also delete a layer 4 load balancer that no longer has virtual servers attached. This feature is enabled by default. You can disable it by setting l4_lb_auto_scaling to false in the NCP ConfigMap.

Load Balancer and Network Policy

When traffic is forwarded to the pods from the NSX load balancer virtual server, the source IP is the tier-1 router's uplink port's IP address. This address is on the private tier-1 transit network, and can cause the CIDR-based network policies to disallow traffic that should be allowed. To avoid this issue, the network policy must be configured such that the tier-1 router's uplink port's IP address is part of the allowed CIDR block. This internal IP address will be visible in the status.loadbalancer.ingress.ip field and as an annotation (ncp/internal_ip_for_policy) on the Ingress resource.

For example, if the external IP address of the virtual server is 4.4.0.5 and the IP address of the internal tier-1 router's uplink port is 100.64.224.11, the status will be:
    status:
      loadBalancer:
      ingress:
      - ip: 4.4.0.5
      - ip: 100.64.224.11
The annotation on the Ingress and service of type LoadBalancer resource will be:
    ncp/internal_ip_for_policy: 100.64.224.11
The IP address 100.64.224.11 must belong to the allowed CIDR in the ipBlock selector of the network policy. For example,
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    ...
    ingress:
    - from:
      - ipBlock:
         cidr: 100.64.224.11/32

Sample Script to Generate a CA-Signed Certificate

The script below generates a CA-signed certificate and a private key stored in the files <filename>.crt and <finename>.key, respectively. The genrsa command generates a CA key. The CA key should be encrypted. You can specify an encryption method with the command such as aes256.
#!/bin/bash
host="www.example.com"
filename=server

openssl genrsa -out ca.key 4096
openssl req -key ca.key -new -x509 -days 365 -sha256 -extensions v3_ca -out ca.crt -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl req -out ${filename}.csr -new -newkey rsa:2048 -nodes -keyout ${filename}.key -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl x509 -req -days 360 -in ${filename}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${filename}.crt -sha256

Mount the Default Certificate and Key into the NCP Pod

After the certificate and private key have been generated, place them in the directory /etc/nsx-ujo on the host VM. Assuming the certificate and key files are named lb-default.crt and lb-default.key, respectively, edit ncp-rc.yaml so that these files on the host are mounted into the pod. For example,
spec:
  ...
  containers:
  - name: nsx-ncp
    ...
    volumeMounts:
    ...
    - name: lb-default-cert
      # Mount path must match nsx_v3 option "lb_default_cert_path"
      mountPath: /etc/nsx-ujo/lb-default.crt
    - name: lb-priv-key
      # Mount path must match nsx_v3 option "lb_priv_key_path"
      mountPath: /etc/nsx-ujo/lb-default.key
  volumes:
  ...
  - name: lb-default-cert
    hostPath:
      path: /etc/nsx-ujo/lb-default.crt
  - name: lb-priv-key
    hostPath:
      path: /etc/nsx-ujo/lb-default.key