You can create CRDs (CustomResourceDefinitions) to monitor the usage of NSX load balancers and to create additional NSX layer 7 load balancers to handle Ingress workloads that the default load balancer cannot handle. These CRDs are not for scaling layer 4 load balancers that are created for Kubernetes LoadBalancer services.

If you have a shared tier-1 topology, you must configure tier0_gateway under [nsx_v3] in the nsx-ncp-config ConfigMap to use this feature.

The CRDs are:
  • NSXLoadBalancerMonitor - This CRD is used to report usage statistics of the NSX load balancers. In Policy mode, this CRD will only monitor namespace load balancers created using the LoadBalancer CRD.
  • LoadBalancer - This CRD is used to create new NSX load balancers. The definition of this resource is in the NCP YAML file. In Policy mode and TKGI deployments, this is a namespace resource. In Manager mode deployments, this is a clusterwide resource.

The procedure to enable this feature is the same for Manager mode and Policy mode.

Perform the following steps to enable this feature:
  • Set the enable_lb_crd option in the [k8s] section to True.
  • Apply the NCP YAML file with the following command:

    kubectl apply -f ncp-<platform>.yaml

To create a new NSX load balancer, apply a YAML file that defines a LoadBalancer CRD. For example, 
apiVersion: vmware.com/v1alpha1
kind: LoadBalancer
metadata:
    name: cluster1-lbs0
spec:
    httpConfig: {}
    size: SMALL

This YAML file will create an NSX load balancer of the specified size, and a pair of layer 7 virtual servers without persistence, SSL or X-forward settings. The size parameter can be SMALL, MEDIUM, or LARGE. The default value is SMALL. After the NSX load balancer is created, the size cannot be changed and any update to the size parameter will be ignored. The IP of the virtual server is allocated from the configured default external pool for load balancers. The ports by default are 80 and 443. Non-standard ports are supported if the custom port is included in the HTTP HOST header. Note that custom ports are only supported in non-TLS Ingresses. Also, the virtual servers created by the LoadBalancer CRD do not support the "enable access log" parameter.

To check the creation status of the LoadBalancer CRD, run the following command: 
kubectl get lb <name of the LoadBalancer> -o yaml
The result looks something like the following: 
status:
 conditions:
 - status: "True"
   type: Ready
 httpVirtualIP: <realized virtual IP>

This result indicates that the creation was successful. If the creation failed, status will be False and there will not be a virtual IP.

You can also customize settings for the NSX load balancer and virtual servers. To configure the IP and port for the virtual server: 
spec:
    httpConfig:
        virtualIP: <ip address, default to auto-allocate>
        port: <port number, default to 80>

Note: You must not configure the same virtualIP for different LoadBalancer CRDs.

To specify the session affinity and X-forwarded-mode: 
spec:
    httpConfig:
         xForwardedFor: <INSERT or REPLACE, default to None>
         affinity:
             type: <source_ip or cookie, default to None>
             timeout: <timeout number, default to 10800>
To configure TLS settings: 
spec:
    httpConfig:
        tls:
            port: <tls port number, default to 443>
            secretName: <name of secret, default to None>
Note that even if you set the HTTP and HTTPS ports to non-default values, the Ingress printer will always display the default port values (80 and 443) when showing the Ingress status because of a Kubernetes limitation. You should still use the configured ports to access Ingress. For example, 
curl -I -HHost:tea.example.com http://$INGRESS_IP:$CRD_LB_HTTP_PORT/tea

You can create the secret before or after the creation of LoadBalancer. To update the certificate, remove the secretName from the LoadBalancer spec first, update the data of the secret, then re-attach the same secret using the above configuration. Creating a new secret and updating the secretName will also work. Note that sharing the same secret data between different CRD load balancers is not supported. You must configure CRD load balancers with different certificates.

To view the status and statistics of NSX load balancers, run the following command: 
kubectl get lbm
This will list all the NSXLoadBlancerMonitors, one for each NSX load balancer. The following information is displayed:
  • Usage - The number of workloads on the NSX load balancer.
  • Traffic - The aggregated statistics of each Virtual Server.
  • Health - This field has two dimensions:
    • servicePressureIndex - This indicates the performance of the load balancer. Two values are provided: score and severity.
    • infraPressureIndex - This indicates the performance of the underlying infrastructure components. In NCP 2.5.1, this value is not always accurate.
    • The field metrics gives an idea of the parameters that are considered when the health score is calculated.

When the servicePressureIndex of a load balancer is HIGH, you can migrate the Ingress workload to other load balancers, which must be the default load balancer or load balancers created using the LoadBalancer CRD.

To place an Ingress on a dedicated load balancer, add an annotation to the Ingress specification. For example, 
 annotations:
   nsx/loadbalancer: <name of the LoadBalancer CRD>

If the annotation is missing or set to null, the Ingress is placed on the default NSX load balancer.