L4Rule CRD is created in a given namespace where the operator desires to have more control. The topic discusses the details and associated rules of using each field of the L4Rule CRD.

L4 Rule CRD can be used to modify the default properties of the L4 virtual service and the pools created from a service of Type LoadBalancer. Service of type LoadBalancer has to be annotated with the name of the CRD to attach the CRD to the service. A sample L4 Rule CRD is as shown below:

apiVersion: ako.vmware.com/v1alpha2
kind: L4Rule
metadata:
  name: my-l4-rule
  namespace: green
spec:
  analyticsProfileRef: Custom-Analytics-Profile
  analyticsPolicy:
    fullClientLogs:
      enabled: true
      duration: 0
      throttle: 30
  applicationProfileRef: Custom-L4-Application-Profile
  loadBalancerIP: 49.20.193.207
  performanceLimits:
    maxConcurrentConnections: 105
    maxThroughput: 100
  networkProfileRef: Custom-Network-Profile
  networkSecurityPolicyRef: Custom-Network-Security-Policy
  securityPolicyRef: Custom-Security-Policy
  vsDatascriptRefs:
    - Custom-DS-01
    - Custom-DS-02
  backendProperties:
    - port: 80
      protocol: TCP
      enabled: true
      applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
      healthMonitorRefs:
        - Custom-HM-01
        - Custom-HM-02
      lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
      lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
      lbAlgorithmConsistentHashHdr: custom-string
      sslProfileRef: Custom-SSL-Profile
      sslKeyAndCertificateRef: Custom-Key-And-Certificate
      pkiProfileRef: Custom-PKI-Profile
      analyticsPolicy:
        enableRealtimeMetrics: true
      minServersUp: 1
  listenerProperties:
    - port: 80
      protocol: TCP
      enableSsl: true
  sslKeyAndCertificateRefs:
    - Custom-L4-SSL-Key-Cert
  sslProfileRef: Custom-L4-SSL-Profile
Note:

The L4 Rule CRD must be configured in the same namespace as the service of type LoadBalancer.

Attaching L4Rule to LoadBalancer type of Services

An L4 Rule is applied to a virtual service and pool created from the LoadBalancer type of Services when the l4 rule is attached to the service. An L4Rule can be attached by annotating the service with the name of the L4Rule CRD with ako.vmware.com/l4rule as the key and name of the l4 rule crd as the value.

metadata:
  annotations:
    ako.vmware.com/l4rule: <name-of-the-l4-rule-crd>

Consider the following example showing a service my-service of type LoadBalancer annotated with an L4Rule my-l4-rule.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotation:
    ako.vmware.com/l4rule: my-l4-rule
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  clusterIP: 10.0.171.239
  type: LoadBalancer

Express Custom Analytics Profiles

L4Rule CRD can be used to express analytics profile references. The analytics profile reference must have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

analyticsProfile: Custom-Analytics-Profile

The analytics profiles can be used for various Network/Health Score analytics settings, log processing, and so on.

Configure Analytics Policy

The L4Rule CRD can be used to configure analytics policies such as enable/disable non-significant logs, throttle the number of non-significant logs per second on each SE, and the duration for which the system must capture the logs.

analyticsPolicy:
  fullClientLogs:
    enabled: true
    duration: 0
    throttle: 30

Here, the field throttle is configured as shown below:

  • For throttle to take effect, the value of enabled has to be set to True.

  • Range: 0 to 65535

  • Default value: 10

  • To deactivate throttling, set the value of throttle to 0

By default, the AKO sets the duration of logging the non-significant logs to 30 minutes. Configure the duration as Zero (0) to capture the logs indefinitely.

Express Custom Application Profiles

L4 Rule CRD can be used to express application profile references. The application profile can be used to enable PROXY Protocol, rate limit the connections from a client, and so on. The application profile must be created in the NSX Advanced Load Balancer Controller before referring to it. The application profile must be of TYPE of APPLICATION_PROFILE_TYPE_L4.

applicationProfile: Custom-L4-Application-Profile

Note:

  • The application profile should be of type L4 or L4 SSL/TLS.

  • If SSL is enabled for any port in listenerProperties, then application profile should be of type L4 SSL/TLS.

  • Starting with AKO version 1.11.1, L4 SSL/TLS is supported.

Express Custom Load Balancer IP

The loadBalancerIP field can be used to provide a valid preferred IPv4 address for L4 virtual services. The preferred IP must be part of the IPAM configured for the cloud, and must not overlap with any other IP addresses already in use. In case of any misconfigurations whatsoever, AKO would fail to configure the virtual service appropriately throwing an ERROR log for the same.

loadBalancerIP: "49.20.193.207"

Note:

The L4Rule CRD is not aware of any misconfigurations during its creation process, and as a result, the L4Rule will still be marked as Accepted.

Configure Performance Limits

The L4 Rule CRD can be used to configure the performance limit settings such as maximum concurrent client connections allowed, and maximum throughput per second for all clients allowed through the client side.

performanceLimits:
  maxConcurrentConnections: 105
  maxThroughput: 100

The maxConcurrentConnections and maxThroughput supports values from 0 to 65535.

Express Custom Network Profile

The L4 Rule CRD can be used to express a custom network profile. The network profile can be used to configure either TCP/UDP proxy settings or TCP/UDP fast path settings. The network profile must be created in the NSX Advanced Load Balancer Controller before referring to it.

networkProfileRef: Custom-Network-Profile

The AKO defaults the network profile to System-TCP-Proxy.

Note:

  • The network profile settings are dependent on the license configured in the NSX Advanced Load Balancer Controller. Before configuring the profile in the CRD, see Licensing topic in the VMware NSX Advanced Load BalancerAdministration Guide.

  • Starting with AKO version 1.11.1, SSL for L4 virtual service is supported. If SSL is enabled for any port in listenerProperties, then network profile should be of type TCP proxy, since only a single TCP port definition is allowed in the LoadBalancer service for L4 SSL.

Express Custom Network Security Policy

The L4 Rule CRD can be used to express a custom network security policy. The Network security policy can be configured with rules to allow/deny/rate limit connections from a single or group of IP addresses, and so on.

networkSecurityPolicyRef: Custom-Network-Security-Policy

The Network Security Policy must be created in the NSX Advanced Load Balancer Controller before referring to it.

Express Custom Security Policy

The L4Rule CRD can be used to express a custom Security Policy. Security Policy is applied to the traffic of the virtual service, and it is used to specify various configuration information used to perform Distributed Denial of Service (DDoS) attacks detection and mitigation.

securityPolicyRef: Custom-Security-Policy

The Security Policy must be created in the NSX Advanced Load Balancer Controller before referring to it.

Express Custom DataScripts

The L4Rule CRD can be used to express DataScripts references. The DataScripts references must have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

vsDatascriptRefs:
  - Custom-DS-01
  - Custom-DS-02

The DataScripts can be used to apply custom scripts to data traffic. The order of evaluation of the DataScripts is in the same order they appear in the CRD definition.

Configure Backend Properties

The backendProperties section in the L4Rule can be used to configure pool settings such as custom health monitors, application persistence profiles, LB algorithms, and so on. The L4Rule CRD identifies the pools based on the port and protocol, and AKO applies the configuration to it. AKO logs a warning if the port and protocol do not match the service’s port and protocol configurations.

A sample backendProperties looks like this:

backendProperties:
  - port: 80
    protocol: TCP
    enabled: true
    applicationPersistenceProfileRef: Custom-Application-Persistence-Profile
    healthMonitorRefs:
      - Custom-HM-01
      - Custom-HM-02
    lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
    lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
    lbAlgorithmConsistentHashHdr: custom-string
    sslProfileRef: Custom-SSL-Profile
    sslKeyAndCertificateRef: Custom-Key-And-Certificate
    pkiProfileRef: Custom-PKI-Profile
    analyticsPolicy:
      enableRealtimeMetrics: true
    minServersUp: 1
Note:

The fields port and protocol are mandatory and AKO uses these fields to identify the pool. The port and protocol must equal the service’s port and protocol.

AKO defaults the fields enabled to true and lbAlgorithm to LB_ALGORITHM_LEAST_CONNECTIONS.

Enable/ Disable Pool

The field enabled in the L4Rule can be used to activate/deactivate pools attached to an L4 virtual service. By default, the value of the field is true and the user has to set the value to false to disable the pool.

enabled: true # or false

Express Custom Application Persistence Profile

The L4Rule CRD can be used to express a custom Application Persistence Profile reference. The Application Persistence Profile reference must have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

applicationPersistenceProfileRef: Custom-Application-Persistence-Profile

Express Custom Health Monitors

L4Rule CRD can be used to express custom health monitor references. The health monitor reference must have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

healthMonitorRefs:
  - Custom-HM-01
  - Custom-HM-02

The health monitors can be used to verify server health. A server (Kubernetes pods in this case) will be marked UP only when all the health monitors return successful responses. Health monitors provided here overwrite the default health monitor configuration set by AKO, that is, System-TCP for TCP traffic and System-UDP for UDP traffic based on the service configuration.

Configure LB Algorithm

The L4Rule CRD can be used to select suitable LB algorithms to effectively distribute traffic across healthy servers. The LB algorithm can be used for distributing TCP and UDP connections across servers.

A sample LB algorithm configuration is shown below:

lbAlgorithm: LB_ALGORITHM_CONSISTENT_HASH
lbAlgorithmHash: LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER
lbAlgorithmConsistentHashHdr: "custom-string"

The lbAlgorithm allows a predefined set of values, and the user can choose the desired one. Presently the following values are supported for the field lbAlgorithm:

  • LB_ALGORITHM_LEAST_CONNECTIONS

  • LB_ALGORITHM_ROUND_ROBIN

  • LB_ALGORITHM_FASTEST_RESPONSE

  • LB_ALGORITHM_CONSISTENT_HASH

  • LB_ALGORITHM_LEAST_LOAD

  • LB_ALGORITHM_FEWEST_SERVERS

  • LB_ALGORITHM_RANDOM

  • LB_ALGORITHM_FEWEST_TASKS

  • LB_ALGORITHM_NEAREST_SERVER

  • LB_ALGORITHM_CORE_AFFINITY

  • LB_ALGORITHM_TOPOLOGY

The lbAlgorithmHash field is used only when the algorithm is chosen as LB_ALGORITHM_CONSISTENT_HASH. The following values are supported for lbAlgorithmHash:

LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS_AND_PORT LB_ALGORITHM_CONSISTENT_HASH_URI LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_STRING LB_ALGORITHM_CONSISTENT_HASH_CALLID

The lbAlgorithmConsistentHashHdr field is used only when the lbAlgorithmHash is chosen as LB_ALGORITHM_CONSISTENT_HASH_CUSTOM_HEADER.

Express Custom SSL Profile

The custom SSL profile can be used to configure the desired set of SSL versions and ciphers to accept SSL/TLS terminated connections.

sslProfileRef: Custom-SSL-Profile

The custom SSL profile must have been created in the NSX Advanced Load Balancer Controller before the CRD creation.

Express Custom SSL Key And Certificate

The L4Rule CRD can be used to express a custom SSL key and certificate reference. The service engines present this certificate to the backend servers. The custom SSL key and certificate must have been created in the NSX Advanced Controller before the CRD creation.

sslKeyAndCertificateRef: Custom-Key-And-Certificate

Express Custom PKI Profile

The L4Rule CRD can be used to express a custom PKI profile reference. Once configured, the NSX Advanced Load Balancer Controller validates the SSL certificate present by a server against the custom PKI Profile configured in the CRD. The custom PKI Profile must be created in the NSX Advanced Load Balancer Controller before referring to it.

pkiProfileRef: Custom-PKI-Profile

Configure Analytics Policy

The L4Rule CRD can be used to configure the analytics settings for the pool. Set the enableRealtimeMetrics to true/false to enable/disable real-time metrics for server and pool metrics.

Configure Minimum Servers UP

The L4Rule CRD can be used to configure the minimum number of servers in the UP state for marking the pool UP.

minServersUp: 1
Note:

The value given must be equal to or less than the number of health monitors attached to the pool.

Configure Listener Properties

The listenerProperties section in the L4Rule can be used to enable or disable SSL support for L4 virtual services. Each item in the listenerProperties array corresponds to a port definition in the LoadBalancer service with the option to enable SSL termination in the service/listener settings created for that port as part of the NSX Advanced Load Balancer virtual service.

When an L4Rule object is created with listener properties, AKO identifies the service/listener setting on the virtual service based on the port and protocol, and applies the SSL configuration to it. AKO logs a WARNING if the port and protocol does not match the service's port and protocol configurations. There are also limitations and conditions for using listener properties, see Caveats section for more details.

A sample listenerProperties looks as shown below:

listenerProperties:
  - port: 80
    protocol: TCP
    enableSsl: true

The fields port and protocol are mandatory, and AKO uses these fields to identify the corresponding service/listener setting in the virtual service. The port and protocol must equal the service's port and protocol. Currently, only a single TCP port is allowed in the LoadBalancer service definition if SSL is required to be enabled. Hence, the same limitation also applies to listenerProperties, which can also have only one matching TCP based port definition.

Enable or Disable SSL

The field enableSsl in the L4Rule can be used to enable SSL termination and offload for traffic from clients for an L4 virtual service. The enableSsl field is specified for a port, and AKO configures the associated service/listener setting in the virtual service with the value. By default, the value of this field is false, and the user has to set the value to true to enable SSL.

enableSsl: true # or false

Express custom SSL Profile for Virtual Service

The custom SSL profile can be used to configure the desired set of SSL versions and ciphers to accept SSL/TLS terminated connections for the virtual service.

sslProfileRef: Custom-SSL-Profile

The custom SSL profile must be created in the NSX Advanced Load Balancer Controller before the CRD creation.

Note:

The sslProfileRef must only be specified when SSL is enabled for a virtual service. The L4Rule will otherwise be rejected.

Express custom SSL Keys And Certificates for Virtual Service

The L4Rule CRD can be used to express custom SSL key and certificate references for a virtual service. These certificates will be presented to SSL/TLS terminated connections. The custom SSL keys and certificates must be created in the NSX Advanced Load Balancer Controller before the CRD creation.

      sslKeyAndCertificateRefs:
      - "Custom-SSL-Key-Cert"
Note:

The sslKeyAndCertificateRefs must only be specified when SSL is enabled for a virtual service. The L4Rule will otherwise be rejected.

Status Messages

The status messages are used to give instantaneous feedback about the reference objects specified in the L4Rule CRD.

Following are some sample status messages:

  1. Accepted L4Rule Object

    $ kubectl get l4rule
NAME         STATUS     AGE
my-l4-rule   Accepted   3d5s

    An L4Rule is accepted only when all the reference objects specified inside it exist in the NSX Advanced Load Balancer Controller.

  2. Rejected L4Rule Object

    $ kubectl get l4rule
NAME            STATUS     AGE
my-l4-rule-alt  Rejected   2d23h

    The detailed reason for rejection can be obtained from the status:

    status: error: applicationprofile "My-L4-Application" not found on controller status: Rejected

Caveats

  1. Sharing L4Rule with Load Balancer IP

    The L4Rule CRD with load balancer IP can be shared among services only when the services contain the ako.vmware.com/enable-shared-vip annotation. However, L4Rule cannot be shared if SSL termination is required to be enabled for the services. So, if enableSsl is set to true for any port in listenerProperties section, then that L4Rule must only be applied to a single LoadBalancer service.

  2. L4Rule Deletion

    If an L4Rule is deleted, the L4 VSes and Pools in the NSX Advanced Load Balancer Controller will be configured with the default values.

  3. L4Rule Admission

    An L4Rule CRD is only admitted if all the objects referenced in it, exist in the NSX Advanced Load Balancer Controller. If after admission the object references are deleted out-of-band, then AKO does not re-validate the associated HostRule CRD objects. The user needs to manually edit or delete the object for new changes to take effect.

  4. Enabling SSL with L4Rule

    Following are the limitations when trying to enable SSL termintaion for an L4 virtual service with L4Rule:

    1. Currently, only a single TCP port is allowed in the LoadBalancer service definition if SSL is required to be enabled. Hence, the same limitation also applies to listenerProperties, which can also have only one matching TCP based port definition with enableSsl field. This is because NSX Advanced Load Balancer only supports SSL termination with TCP protocol, and also a virtual service of type L4 SSL can have only one back end pool configured.

    2. If enableSsl is set to true for any port in listenerProperties section, then applicationProfile must be of type L4 SSL/TLS. If application profile is not of type L4 SSL/TLS, then L4Rule will be rejected. If applicationProfile is not set, then it defaults to System-L4-Application in the CRD, but AKO internally sets the application profile as System-SSL-Application, which is the default value when SSL is enabled.

    3. If enableSsl is set to true for any port in listenerProperties section, then networkProfileRef must be of type TCP proxy. Currently, only a single TCP port definition is allowed in the LoadBalancer service and listener properties.

    4. The sslProfileRef and sslKeyAndCertificateRefs must be set for the virtual service only if SSL termination is enabled for any port in listenerProperties and application profile is of type L4 SSL/TLS, otherwise the L4Rule will be rejected. If enableSsl is set to true, but sslProfileRef and sslKeyAndCertificateRefs are not set specified in the L4Rule, then these fields will be set with their default values in NSX Advanced Load Balancer.