Configuring load balancing involves configuring a Kubernetes LoadBalancer service or an OpenShift route. You also need to configure the NCP replication controller. The LoadBalancer service is for layer 4 traffic and the OpenShift route is for layer 7 traffic.

When you configure a Kubernetes LoadBalancer service, it is allocated an IP address from the external IP block that you configure. The load balancer is exposed on this IP address and the service port. You can specify the name or ID of an IP pool using the loadBalancerIP spec in the LoadBalancer definition. The Loadbalancer service's IP will be allocated from this IP pool. If the loadBalancerIP spec is empty, the IP will be allocated from the external IP block that you configure.

The IP pool specified by loadBalancerIP must have the tag scope: ncp/owner, tag: cluster:<cluster_name>.

To use the NSX-T Data Center load balancer, you must configure load balancing in NCP. In the ncp_rc.yml file, do the following:

  1. Set use_native_loadbalancer = True.
  2. Set pool_algorithm to WEIGHTED_ROUND_ROBIN.
  3. Set lb_default_cert_path and lb_priv_key_path to be the full path names of the CA-signed certificate file and the private key file, respectively. See below for a sample script to generate a CA-signed certificate. In addition, mount the default certificate and key into the NCP pod. See below for instructions.
  4. (Optional) Specify a persistence setting with the parameters l4_persistence and l7_persistence. The available option for layer 4 persistence is source IP. The available options for layer 7 persistence are cookie and source IP. The default is <None>. For example, 
       # Choice of persistence type for ingress traffic through L7 Loadbalancer.
   # Accepted values:
   # 'cookie'
   # 'source_ip'
   l7_persistence = cookie

   # Choice of persistence type for ingress traffic through L4 Loadbalancer.
   # Accepted values:
   # 'source_ip'
   l4_persistence = source_ip
  5. (Optional) Set service_size = SMALL, MEDIUM, or LARGE. The default is SMALL.
  6. If you are running OpenShift 3.11, you must perform the following configuration so that OpenShift will not assign an IP to the LoadBalancer service.
    • Set ingressIPNetworkCIDR to 0.0.0.0/32 under networkConfig in the /etc/origin/master/master-config.yaml file.
    • Restart the API server and controllers with the following commands: 
         master-restart api
   master-restart controllers

For a Kubernetes LoadBalancer service, you can also specify sessionAffinity on the service spec to configure persistence behavior for the service if the global layer 4 persistence is turned off, that is, l4_persistence is set to <None>. If l4_persistence is set to source_ip, the sessionAffinity on the service spec can be used to customize the persistence timeout for the service. The default layer 4 persistence timeout is 10800 seconds (same as that specified in the Kubernetes documentation for services (https://kubernetes.io/docs/concepts/services-networking/service). All services with default persistence timeout will share the same NSX-T load balancer persistence profile. A dedicated profile will be created for each service with a non-default persistence timeout.

Note: If the backend service of an Ingress is a service of type LoadBalancer, then the layer 4 virtual server for the service and the layer 7 virtual server for the Ingress cannot have different persistence settings, for example, source_ip for layer 4 and cookie for layer 7. In such a scenario, the persistence settings for both virtual servers must be the same ( source_ip, cookie, or None), or one of them is None (then the other setting can be source_ip or cookie). An example of such a scenario: 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
-----
apiVersion: v1
kind: Service
metadata:
  name: tea-svc <==== same as the Ingress backend above
  labels:
    app: tea
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: tcp
  selector:
    app: tea
  type: LoadBalancer

Router Sharding

NCP always handles TLS edge termination and HTTP routes, and skips TLS passthrough routes and TLS re-encrypt routes regardless of their namespaces or namespace labels. To restrict a OpenShift router to handling only TLS re-encrypt and passthrough routes, you must perform the following steps:
  • Add a namespace label selector to the Openshift router.
  • Add a namespace label to the target namespace.
  • Create TLS re-encrypt/passthrough routes in the target namespace.

For example, to configure a router with a namespace label selector, run the following command (assuming the router's service account name is router):

oc set env dc/router NAMESPACE_LABELS="router=r1"

The router will now handle routes from the selected namespaces. To make this selector match a namespace, run the following command (assuming the namespace is called ns1):

oc label namespace ns1 "router=r1"

Layer 7 Load Balancer Example

The following YAML file configures two replication controllers (tea-rc and coffee-rc), two services (tea-svc and coffee-svc), and two routes (cafe-route-multi and cafe-route) to provide layer 7 load balancing. 
# RC
apiVersion: v1
kind: ReplicationController
metadata:
  name: tea-rc
spec:
  replicas: 2
  template:
    metadata:
       labels:
         app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/hello
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: coffee-rc
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/hello
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
# Services
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
    app: tea
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: tea
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
  labels:
    app: coffee
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: coffee
---
# Routes
apiVersion: v1
kind: Route
metadata:
  name: cafe-route-multi
spec:
  host: www.cafe.com
  path: /drinks
  to:
    kind: Service
    name: tea-svc
    weight: 1
  alternateBackends:
  - kind: Service
    name: coffee-svc
    weight: 2
---
apiVersion: v1
kind: Route
metadata:
  name: cafe-route
spec:
  host: www.cafe.com
  path: /tea-svc
  to:
    kind: Service
    name: tea-svc
    weight: 1

Additional Notes

  • Only edge termination is supported for HTTPS traffic.
  • Wildcard subdomain is supported. For example, if wildcardPolicy is set to Subdomain, and the host name is set to wildcard.example.com, any request to *.example.com will be serviced.
  • If NCP throws an error during the processing of a Route event due to misconfiguration, you need to correct the Route YAML file, delete and recreate the Route resource.
  • NCP does not enforce hostname ownership by namespaces.
  • One Loadbalancer service is supported per Kubernetes cluster.
  • NSX-T Data Center will create a layer 4 load balancer virtual server and pool for each LoadBalancer service port. Both TCP and UDP are supported.
  • The NSX-T Data Center load balancer comes in different sizes. For information about configuring an NSX-T Data Center load balancer, see the NSX-T Data Center Administration Guide.

    After the load balancer is created, the load balancer size cannot be changed by updating the configuration file. It can be changed through the UI or API.

  • Automatic scaling of the layer 4 load balancer is supported. If a Kubernetes LoadBalancer service is created or modified so that it requires additional virtual servers and the existing layer 4 load balancer does not have the capacity, a new layer 4 load balancer will be created. NCP will also delete a layer 4 load balancer that no longer has virtual servers attached. This feature is enabled by default. You can disable it by setting l4_lb_auto_scaling to false in the NCP ConfigMap.

Sample Script to Generate a CA-Signed Certificate

The script below generates a CA-signed certificate and a private key stored in the files <filename>.crt and <finename>.key, respectively. The genrsa command generates a CA key. The CA key should be encrypted. You can specify an encryption method with the command such as aes256. 
#!/bin/bash
host="www.example.com"
filename=server

openssl genrsa -out ca.key 4096
openssl req -key ca.key -new -x509 -days 365 -sha256 -extensions v3_ca -out ca.crt -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl req -out ${filename}.csr -new -newkey rsa:2048 -nodes -keyout ${filename}.key -subj "/C=US/ST=CA/L=Palo Alto/O=OS3/OU=Eng/CN=${host}"
openssl x509 -req -days 360 -in ${filename}.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ${filename}.crt -sha256

Mount the Default Certificate and Key into the NCP Pod

After the certificate and private key have been generated, place them in the directory /etc/nsx-ujo on the host VM. Assuming the certificate and key files are named lb-default.crt and lb-default.key, respectively, edit ncp-rc.yaml so that these files on the host are mounted into the pod. For example, 
spec:
  ...
  containers:
  - name: nsx-ncp
    ...
    volumeMounts:
    ...
    - name: lb-default-cert
      # Mount path must match nsx_v3 option "lb_default_cert_path"
      mountPath: /etc/nsx-ujo/lb-default.crt
    - name: lb-priv-key
      # Mount path must match nsx_v3 option "lb_priv_key_path"
      mountPath: /etc/nsx-ujo/lb-default.key
  volumes:
  ...
  - name: lb-default-cert
    hostPath:
      path: /etc/nsx-ujo/lb-default.crt
  - name: lb-priv-key
    hostPath:
      path: /etc/nsx-ujo/lb-default.key