You can create CRDs (CustomResourceDefinitions) to monitor the usage of NSX load balancers and to create additional NSX layer 7 load balancers to handle Ingress workloads that the default load balancer cannot handle. These CRDs are not for scaling layer 4 load balancers that are created for Kubernetes LoadBalancer services.

In Manager mode, this feature is supported starting with NCP 2.5.1. In Policy mode, this feature is supported starting with NCP 3.0.1.

The CRDs are:
  • NSXLoadBalancerMonitor - This CRD is used to report usage statistics of the NSX load balancers. In Policy mode, this CRD will only monitor namespace load balancers created using the LoadBalancer CRD.
  • LoadBalancer - This CRD is used to create new NSX load balancers. The definition of this resource is in the NCP YAML file. In Policy mode and PKS deployments, this is a namespace resource. In Manager mode deployments, this is a clusterwide resource.
In Manager mode, perform the following steps to enable this feature:
  • Set the enable_lb_crd option in the k8s section to True.
  • Apply the NCP YAML file with the following command:

    kubectl apply -f ncp-<platform>.yaml

In Policy mode, perform the following steps to enable this feature:
  • If you upgraded NCP from a previous version to 3.0.1, delete the old CRD definitions with the following commands:
    kubectl delete crd nsxlbmonitors.vmware.com
kubectl delete crd loadbalancers.vmware.com
  • Set the enable_lb_crd and enable_vnet_crd options in the k8s section to True.
  • Apply the Policy version of the NCP YAML file with the following command:

    kubectl apply -f ncp-<platform>-policy.yaml

In Manager mode, to create a new NSX load balancer, apply a YAML file that defines a LoadBalancer CRD. For example, 
apiVersion: vmware.com/v1alpha1
kind: LoadBalancer
metadata:
    name: cluster1-lbs0
spec:
    httpConfig: {}
In Policy mode, LoadBalancer needs to be created with VirtualNetwork. For example, 
apiVersion: vmware.com/v1alpha1
kind: VirtualNetwork
metadata:
    name: vnet
---
apiVersion: vmware.com/v1alpha1
kind: LoadBalancer
metadata:
    name: cluster1-lbs0
spec:
    virtualNetworkName: vnet # Set to match VirtualNetwork object name
    httpConfig: {}

This YAML file will create an NSX load balancer of small size, and a pair of layer 7 virtual servers without persistence, SSL or X-forward settings. The IP of the virtual server is allocated from the configured default external pool for load balancers. The ports by default are 80 and 443. Non-standard ports are supported if the custom port is included in the HTTP HOST header. Note that custom ports are only supported in non-TLS Ingresses.

To check the creation status of the LoadBalancer CRD, run the following command: 
kubectl get lb <name of the LoadBalancer> -o yaml
The result looks something like the following: 
status:
 conditions:
 - status: "True"
   type: Ready
 httpVirtualIP: <realized virtual IP>

This result indicates that the creation was successful. If the creation failed, status will be "False" and there will not be a virtual IP.

You can also customize settings for the NSX load balancer and virtual servers. To configure the IP and port for the virtual server: 
spec:
    httpConfig:
        virtualIP: <ip address, default to auto-allocate>
        port: <port number, default to 80>
To specify the session affinity and X-forwarded-mode: 
spec:
    httpConfig:
         xForwardedFor: <INSERT or REPLACE, default to None>
         affinity:
             type: <source_ip or cookie, default to None>
             timeout: <timeout number, default to 10800>
To configure TLS settings: 
spec:
    httpConfig:
        tls:
            port: <tls port number, default to 443>
            secretName: <name of secret, default to None>
            secretNamespace: <namespace of secret, default to None>
Note that even if you set the HTTP and HTTPS ports to non-default values, the Ingress printer will always display the default port values (80 and 443) when showing the Ingress status because of a Kubernetes limitation. You should still use the configured ports to access Ingress. For example, 
curl -I -HHost:tea.example.com http://$INGRESS_IP:$CRD_LB_HTTP_PORT/tea

You can create the secret before or after the creation of LoadBalancer. To update the certificate, remove the secretName and secretNamespace from the LoadBalancer spec first, update the data of the secret, then re-attach the same secret using the above configuration. Creating a new secret and updating the secretName and secretNamespace will also work. Note that sharing the same secret data between different CRD load balancers is not supported. You must configure CRD load balancers with different certificates.

To view the status and statistics of NSX load balancers, run the following command: 
kubectl get lbm
This will list all the NSXLoadBlancerMonitors, one for each NSX load balancer. The following information is displayed:
  • Usage - The number of workloads on the NSX load balancer.
  • Traffic - The aggregated statistics of each Virtual Server.
  • Health - This field has two dimensions:
    • servicePressureIndex - This indicates the performance of the load balancer. Two values are provided: score and severity.
    • infraPressureIndex - This indicates the performance of the underlying infrastructure components. In NCP 2.5.1, this value is not always accurate.
    • The field metrics gives an idea of the parameters that are considered when the health score is calculated.

When the servicePressureIndex of a load balancer is HIGH, you can migrate the Ingress workload to other load balancers, which must be the default load balancer or load balancers created using the LoadBalancer CRD.

To place an Ingress on a dedicated load balancer, add an annotation to the Ingress specification. For example, 
 annotations:
   nsx/loadbalancer: <name of the LoadBalancer CRD>

If the annotation is missing or set to null, the Ingress is placed on the default NSX load balancer.