Starting with NCP 2.5.1, you can create CRDs (CustomResourceDefinitions) to monitor the usage of NSX load balancers and to create additional NSX layer 7 load balancers to handle Ingress workloads that the default load balancer cannot handle. These CRDs are not for scaling layer 4 load balancers that are created for Kubernetes LoadBalancer services.

The CRDs are:
  • NSXLoadBalancerMonitor - This CRD is used to report usage statistics of the NSX load balancers.
  • LoadBalancer - This CRD is used to create new NSX load balancers. The definition of this resource is in the NCP YAML file. This is a clusterwide resource and is not bound to a particular namespace.

To enable this feature, in the NCP YAML file, the enable_lb_crd option in the k8s section must be set to True, and the policy_nsxapi option must be set to False. Therefore, you must use the manager UI and API to configure NSX-T resources.

To create a new NSX load balancer, apply a YAML file that defines a LoadBalancer CRD. For example, 
apiVersion: vmware.com/v1alpha1
kind: LoadBalancer
metadata:
    name: cluster1-lbs0
spec:
    httpConfig: {}

This YAML file will create an NSX load balancer of small size, and a pair of layer 7 virtual servers without persistence, SSL or X-forward settings. The IP of the virtual server is allocated from the configured default external pool for load balancers. The ports by default are 80 and 443.

To check the creation status of the LoadBalancer CRD, run the following command: 
kubectl get lb <name of the LoadBalancer> -o yaml
The result looks something like the following: 
status:
 conditions:
 - status: "True"
   type: Ready
 httpVirtualIP: <realized virtual IP>

This result indicates that the creation was successful. If the creation failed, status will be "False" and there will not be a virtual IP.

You can also customize settings for the NSX load balancer and virtual servers. To configure the IP and port for the virtual server: 
spec:
    httpConfig:
        virtualIP: <ip address, default to auto-allocate>
        port: <port number, default to 80>
To specify the session affinity and X-forwarded-mode: 
spec:
    httpConfig:
         xForwardedFor: <INSERT or REPLACE, default to None>
         affinity:
             type: <source_ip or cookie, default to None>
             timeout: <timeout number, default to 10800>
To configure TLS settings: 
spec:
    httpConfig:
        tls:
            port: <tls port number, default to 443>
            secretName: <name of secret, default to None>
            secretNamespace: <namespace of secret, default to None>
Note that even if you set the HTTP and HTTPS ports to non-default values, the Ingress printer will always display the default port values (80 and 443) when showing the Ingress status because of a Kubernetes limitation. You should still use the configured ports to access Ingress. For example, 
curl -I -HHost:tea.example.com http://$INGRESS_IP:$CRD_LB_HTTP_PORT/tea

You can create the secret before or after the creation of LoadBalancer. To update the certificate, remove the secretName and secretNamespace from the LoadBalancer spec first, update the data of the secret, then re-attach the same secret using the above configuration. Creating a new secret and updating the secretName and secretNamespace will also work. Note that sharing the same secret data between different CRD load balancers is not supported. You must configure CRD load balancers with different certificates.

To view the status and statistics of NSX load balancers, run the following command: 
kubectl get lbm
This will list all the NSXLoadBlancerMonitors, one for each NSX load balancer. The following information is displayed:
  • Usage - The number of workloads on the NSX load balancer.
  • Traffic - The aggregated statistics of each Virtual Server.
  • Health - This field has two dimensions:
    • servicePressureIndex - This indicates the performance of the load balancer. Two values are provided: score and severity.
    • infraPressureIndex - This indicates the performance of the underlying infrastructure components. In NCP 2.5.1, this value is not always accurate.
    • The field metrics gives an idea of the parameters that are considered when the health score is calculated.

When the servicePressureIndex of a load balancer is HIGH, you can migrate the Ingress workload to other load balancers, which must be the default load balancer or load balancers created using the LoadBalancer CRD.

To place an Ingress on a dedicated load balancer, add an annotation to the Ingress specification. For example, 
 annotations:
   nsx/loadbalancer: <name of the LoadBalancer CRD>

If the annotation is missing or set to null, the Ingress is placed on the default NSX load balancer.