Ingress

NCP will create one layer-7 load balancer for Ingresses with TLS specification, and one layer-7 load balancer for Ingresses without TLS specification. You can also create CRDs (CustomResourceDefinitions) to handle Ingress scaling.

Note the following:

• All Ingresses will get a single IP address.
• The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools option in the [nsx_v3] section in ncp.ini. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
• The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).

  Note that after you set the external_ip_pools_lb option, you can add more IP pools. However, updating the IP pools is not supported. Removing IP pools is also not supported.

• You can change to a different IP pool by changing the configuration and restarting NCP.
• You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.
• Ingresses without TLS specification will be hosted on HTTP virtual server (port 80).
• Ingresses with TLS specification will be hosted on HTTPS virtual server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.
• Modification of Ingress by adding or removing the TLS section is supported. When the tls key is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS virtual server (port 443) to the HTTP virtual server (port 80). Similarly, when the tls key is added to Ingress specification, the Ingress rules are transferred from the HTTP virtual server (port 80) to the HTTPS virtual server (port 443).
• If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied. The other Ingresses with the duplicate rules will be annotated with error. For example, if you create two Ingresses with the same host and path, and one Ingress is TLS while and the other is non-TLS, and kubernetes.io/ingress.allow-http is false, the two rules will be created on different virtual servers and will not conflict with each other. However, if kubernetes.io/ingress.allow-http is true, the Ingress that is applied later will be annotated with an error. See the "Errors" section below for more information.
• Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.
• If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error. See the "Errors" section below for more information.
• The rules are applied in the following order:
  1. Rules with both host and path specified.
     1. Rules with the Exact pathType.
     2. Rules with the Prefix pathType.
     3. Rules with the Regex pathType.
  2. Rules with host or path specified.
     1. Rules with the Exact pathType.
     2. Rules with the Prefix pathType.
     3. Rules with the Regex pathType.

  Note: If multiple paths match a request, precedence will be given to the longest matching path.

  About pathType:

  • Starting with Kubernetes 1.19 pathType is mandatory.
  • If use-regex is set, it will only take effect if pathType is ImplementationSpecific.
  • If use-regex is not set, ImplementationSpecific is treated the same as Exact. pathType takes precedence over other NCP annotations.
  • Two matching paths with different pathTypes can co-exist.
  • For the Prefix type, /foo will match /foo/, /foo/bar but not /foo or /foobar. To match /foo, add the Exact path /foo to the Ingress rule.
• Hostname wildcards
  When specifying hostname, you can specify the exact name, such as abc.example.com, or use a wildcard, such as *.example.com. Note that the wildcard will match one or more DNS labels. The following table shows the match results if you specify *.example.com.

  Host Header	Match Result
  abc.example.com	Match
  abc.def.example.com	Match
  example.com	No match

• (This is applicable if you use Policy mode.) If a TLS Ingress is created with a default backend, it is recommended that you set up the default backend to respond to both HTTP and HTTPS requests because:
  • The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
  • The load balancer will re-encrypt and send HTTPS request to the backend server if there is no TLS Ingress (in the cluster for the Kubernetes/TKGI use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
• The IngressClass resource is supported.
• The pathType attribute is supported.
• JWT (JSON Web Token) client authentication is supported. This feature requires NSX 3.0.0 or later.
• The HTTP and HTTPS request body size is unlimited. NCP does not support setting a limit for the request body size.

Starting with Kubernetes 1.18, the kubernetes.io/ingress.class annotation is deprecated and replaced by the IngressClass resource. To specify NSX as the Ingress controller in the IngressClass resource, set the controller parameter to k8s.io/nsx. For example:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nsx-lb
  annotations:
    ingressclass.kubernetes.io/is-default-class: true
spec:
  controller: k8s.io/nsx

To specify a third-party Ingress controller, set the controller parameter to k8s.io/<controller name>. For example:

apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx-lb
  annotations:
    ingressclass.kubernetes.io/is-default-class: false
spec:
  controller: k8s.io/ingress-nginx

To set the IngressClass as the default for the cluster, set the annotation ingressclass.kubernetes.io/is-default-class to true. See the example above.

You must not use both the kubernetes.io/ingress.class annotation and the IngressClass resource.

If you manually configure an Endpoints object, be sure to set the targetRef parameter. The value should be the UID of the backend pod. For example:

apiVersion: networking.k8s.io/v1
kind: Endpoints
metadata:
  name: tea-svc
subsets:
  - addresses:
    - ip: 172.26.0.2
      targetRef:
        uid: 4378e0ae-5837-49c6-a0b2-178dced8eb1e
...

Feature Annotations

The following table lists annotations that NCP supports:

Annotation	Description	Supported Values	Default Value
kubernetes.io/ingress.allow-http	Enables HTTP requests in addition to HTTPS	true, false	true
ncp/use-regex	Enables path pattern matching	true, false	false
ingress.kubernetes.io/rewrite-target	Rewrites path of incoming request	Path starting with ‘/’ Path with a numbered placeholder for a group captured in a regular expression, for example, /$1	No default value
ncp/http-redirect	Redirects HTTP requests to HTTPS	true, false	false
kubernetes.io/ingress.class	Indicates which Ingress controller is responsible for this Ingress	nsx, nginx, etc.	nsx
nsx/loadbalancer	Places an Ingress on a dedicated load balancer	Name of a LoadBalancer CRD	No default value
ncp/allowed-source-range	Limits Ingress access by request source IP	Comma-separated list of CIDRs, IP addresses, or IP ranges.	No default value
ncp/ssl-mode	Selects SSL mode for all the rules in an Ingress	offload, reencrypt, passthrough	offload
ncp/jwt-alg	Determines the algorithm to be used to validate JWT signature. Enables JWT client authentication when set with ncp/jwt-secret.	HS256, RS256	No default value
ncp/jwt-secret	Specifies the name of the Kubernetes secret that contains the JWT secret or public key used for signature validation. Enables JWT client authentication when set with ncp/jwt-alg.	Name of a Kubernetes secret	No default value
ncp/jwt-token	Additional location to search for JWT in the HTTP request.	_arg_<param_name>, _cookie_<cookie_name>	No default value
ncp/jwt-realm	Specifies the Realm header returned with 401 when authentication failed.	String indicating the realm	Hostname of the ingress rule
ncp/jwt-preserve	Option to keep JWT and pass to backend after successful authentication.	true, false	true
ncp/connection_multiplexing_enabled	Enables TCP multiplexing.	true, false	false
ncp/connection_multiplexing_number	If TCP multiplxing is enabled, this specifies the number of idle TCP connections that are available.	0 - 2147483647	6

Details about the annotations:

• Path Regex (Regular Expression) Matching
  • You can enable or disable regular expression matching of the Ingress path (but not host) parameter using the annotation ncp/use-regex. If set to false, exact path matching will be performed by doing the equals match. If set to true, regular expression matching will be performed by adding the start of string character (^) and end of string character ($) to the path so that the entire request URI matches the pattern. Note that when using the OR operator (|), always specify the scope with parentheses so that ^ and $ apply to all the operands. For example, path: /(tea|coffee|milk). An Ingress specification example:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        kubernetes.io/ingress.class: "nsx"
        #/tea/cup will be served by tea-svc:80
        ncp/use-regex: "True"
    spec:
      rules:
      - host: cafe.example.com
        http:
          paths:
          # for this tea and coffee NCP will configure regex rule
          - path: /tea/(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: tea-svc
                port:
                  number: 80
          - path: /coffee/(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: coffee-svc
                port:
                  number: 80
          # for juice and alcohol NCP will configure Exact and Prefix rules
          - path: /juice
            pathType: Exact
            backend:
              service:
                name: juice-svc
                port:
                  number: 80
          - path: /alcohol
            pathType: Prefix
            backend:
              service:
                name: bar
                  number: 80

  • Updating Ingresses prior to upgrading NCP
    This is required only if you have Ingresses requiring all sub-path matching using the characters '.' and '*'.

    1. Update the Ingresses to include the annotation ncp/use-regex: true.
    2. For all sub-path matching, if you have paths such as /coffee/* or /*, change them to /coffee/.* and /.*.

       /coffee/.* will match /coffee/, /coffee/a, /coffee/b, /coffee/a/b, and so on. /.* will match /coffee, /tea, /coffee/a, and so on. Omitting the path will produce the same behavior as path: /.*.

• In the following example of the annotation ingress.kubernetes.io/rewrite-target, the paths /tea and /coffee will be rewritten to / before the URL is sent to the backend service.

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: cafe-ingress
    annotations:
      ingress.kubernetes.io/rewrite-target: /
  spec:
    rules:
    - host: cafe.example.com
      http:
        paths:
        - path: /tea
          pathType: ImplementationSpecific
          backend:
            service:
              name: tea-svc
              port:
                number: 80
        - path: /coffee
          pathType: ImplementationSpecific
          backend:
            service:
              name: coffee-svc
              port:
                number: 80

  If path is specified using a regular expression, the captured groups are saved in numbered placeholders in the form $1, $2, and so on. These placeholders can be used as parameters in the ingress.kubernetes.io/rewrite-target annotation. Named capture groups and named placeholders are not supported. For example,

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: cafe-ingress
    annotations:
      kubernetes.io/ingress.class: "nsx"
      ncp/use-regex: "true"
      #/tea/cup will be rewritten to /cup before sending request to endpoint
      ingress.kubernetes.io/rewrite-target: /$1
  spec:
    rules:
    - host: cafe.example.com
      http:
        paths:
        - path: /tea/(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: tea-svc
              port:
                number: 80
        - path: /coffee/(.*)
          pathType: ImplementationSpecific
          backend:
            service:
              name: coffee-svc
              port:
                number: 80

• About the annotation kubernetes.io/ingress.allow-http:
  • If the annotation is set to false, rules will be created for the HTTPS virtual server.
  • If the annotation is set to true or missing, rules will created for both HTTP and HTTPS virtual servers. Note that HTTPS rules will be created only if the TLS section is present in the Ingress specification.
• About the annotation ncp/http-redirect:
  • If the annotation is set to false, Incoming HTTP traffic (port 80) to HTTP Virtual Server will not be redirected to HTTPS Virtual Server.
  • If the annotation is set to true, Incoming HTTP traffic (port 80) to HTTP Virtual Server will be redirected to HTTPS Virtual Server (port 443).

  This annotation is only valid if the TLS section is present. This annotation takes precedence over kubernetes.io/ingress.allow-http. When set to true, it will direct matched HTTP traffic to HTTPS, regardless of how kubernetes.io/ingress.allow-http is set.

• About the annotation kubernetes.io/ingress.class:
  • If the value is nsx, this ingress will be handled by NCP. If any other value is specified, the Ingress will be ignored by NCP. For more info see Third-party Ingress Controllers.
• For more information about the annotation nsx/loadbalancer, see LoadBalancer CRDs to Handle Ingress Scaling.
• About the annotation ncp/allowed-source-range:
  • Supported in Policy mode only.
  • When set, an incoming request will only be accepted if its source IP is included in this annotation. Otherwise, traffic will be dropped.
  • You can specify a combination of CIDR, IP addresses, and IP ranges. For example, 1.1.1.1/24, 2.2.2.2, 3.3.3.3-4.4.4.4.
  • YAML example:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        kubernetes.io/ingress.class: "nsx"
        #Source IP addresses allowed to access ingress URL
        ncp/allowed-source-range: "192.168.128.0/17, 192.168.64.0-192.168.64.255, 192.168.16.32"
    spec:
      tls:
      - hosts:
        - cafe.example.com
      rules:
      - host: cafe.example.com
        http:
          paths:
          - path: /tea
            pathType: ImplementationSpecific
            backend:
              service:
                name: tea-svc
                port:
                  number: 80
          - path: /coffee
            pathType: ImplementationSpecific
            backend:
              service:
                name: coffee-svc
                port:
                  number: 80

• About the annotation ncp/ssl-mode:
  • Supported in Policy mode only.
  • This annotation applies to all the rules in an Ingress and the load balancer will operate in the selected SSL mode for these rules.

    If ncp/ssl-mode is set to reencrypt or passthrough, kubernetes.io/ingress.allow-http must be set to False. This annnotation cannot be set to passthrough if the ingress.kubernetes.io/rewrite-target, ncp/use-regex or ncp/allowed-source-range annotation is set.

    If ncp/ssl-mode is set to passthrough, the attribute path in the rules specification is not supported.

  • YAML example for SSL mode reencrypt:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      namespace: qe
      annotations:
        kubernetes.io/ingress.class: "nsx"
        kubernetes.io/ingress.allow-http: "False"
        ncp/ssl-mode: "reencrypt"
        ingress.kubernetes.io/rewrite-target: "/$1"
        ncp/use-regex: "true"
    spec:
      tls:
      - hosts:
        - cafe.example.com
        secretName: cafe-secret
      defaultBackend:
        service:
          name: default-svc
          port:
            number: 443
      rules:
      - host: cafe.example.com
        http:
          paths:
          - path: /fo{2}/[abc]+(r|R)/(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: tea-svc
                port:
                  number: 443
          - path: /coffee
            pathType: ImplementationSpecific
            backend:
              service:
                name: coffee-svc
                port:
                  number: 443

• YAML example for SSL mode passthrough:

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: cafe-ingress
    annotations:
      kubernetes.io/ingress.class: "nsx"
      #Loadbalancer in passthrough mode
      ncp/ssl-mode: "passthrough"
  spec:
    tls:
    - hosts:
      - cafe.example.com
    rules:
    - host: cafe.example.com
      http:
        paths:
        - backend:
            service:
              name: tea-svc
              port:
                number: 443
          pathType: ImplementationSpecific

• About JWT client authentication:
  • To enable JWT client authentication for all rules under an Ingress, both ncp/jwt-alg and ncp/jwt-secret need to be set to a valid value. When enabled, incoming HTTP traffic will only be passed to the backend if bearing valid JSON web token. Otherwise, traffic will be rejected with the 401 status.
  • This feature is incompatible with the following annotations:
    • kubernetes.io/ingress.allow-http: true
    • ncp/http-redirect: true
    • ncp/ssl-mode: passthrough
  • ncp/jwt-alg:
    • Supported symmetrical algorithms: HS256
    • Supported asymmetrical algorithms: RS256
  • ncp/jwt-secret:
    • A symmetrical key or public certificate must be configured in a Kubernetes secret with the name specified in this annotation under the same namespace as the Ingress.
    • For symmetrical algorithms, the secret must be stored in the jwt.key field.
    • For asymmetrical algorithms, the public cert must be stored in the tls.crt field.
    • JWT client authentication will be disabled if the symmetrical secret or public certificate is not stored in the locations mentioned above, or if the data stored in the secret is invalid.
  • ncp/jwt-token:
    • Only one item can be configured in this annotation.
    • _arg_<param_name>: For JWT passed in as a URI parameter. Specify the parameter name that contains JWT.
    • _cookie_<cookie_name>: For JWT passed in as a cookie. Specify the cookie name that contains JWT.
• About the annotations ncp/connection_multiplxing_enabled and ncp/connection_multiplxing_number:
  • TCP multiplexing cannot coexist with HTTP NTLM. In NSX, if if a load balancer pool with TCP multiplxing enabled is bound to an NTLM-enabled layer-7 server, the NTLM configuration will take precedence over TCP multiplexing.
  • YAML example:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: cafe-ingress
      annotations:
        ncp/connection_multiplexing_enabled: "true"
        ncp/connection_multiplexing_number: 100
    spec:
      tls:
      - hosts:
        - cafe.example.com
      rules:
      - host: cafe.example.com
        http:
          paths:
          - path: /tea
            backend:
              service:
                name: tea-svc
                port:
                  number: 80
          - path: /coffee
            backend:
              service:
                name: coffee-svc
                port:
                  number: 80

Errors

Errors are annotated to the Ingress resource. The error key is ncp/error.loadbalancer and the warning key is ncp/warning.loadbalancer. The possible error and warning are:

• ncp/error.loadbalancer: DEFAULT_BACKEND_IN_USE

  This error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. This error will occur if (1) this Ingress is for HTTP and another Ingress for HTTP with a default backend exists; (2) this Ingress is for HTTPS and another Ingress for HTTPS with a default backend exists; or (3) this Ingress is for HTTP and HTTPS and another Ingress for HTTP and HTTPS with a default backend exists. To fix the error, delete and recreate the Ingress with a correct specification.

• ncp/warning.loadbalancer: SECRET_NOT_FOUND

  This error indicates that the secret specified in the Ingress specification does not exist, or if ncp/jwt-alg and ncp/jwt-secret are anootated but the secret cannot be found under the same namespace as the Ingress. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.

• ncp/warning.loadbalancer: INVALID_INGRESS
  This error indicates that one of the following conditions is true. The Ingress will be inactive. To fix the error, delete and recreate the Ingress with a correct specification.

  • An Ingress rule conflicts with another Ingress rule in the same Kubernetes cluster. Conflicts are determined only for Ingresses with the same match strategy, that is, the same ncp/use-regex annotation value.
  • The kubernetes.io/ingress.allow-http annotation is set to false and the Ingress does not have a TLS section.
  • The ncp/http-redirect annotation is set to true and the Ingress does not have a TLS section.
  • An Ingress rule does not have host and path specified. Such an Ingress rule has the same functionality as the Ingress default backend. Use the Ingress default backend instead.
  • The Ingress has JWL annotations that cannot be correctly processed. For example:
    • Either ncp/jwt-alg or ncp/jwt-secret is missing.
    • ncp/jwt-alg is configured with an unsupported algorithm.
    • ncp/jwt-alg and ncp/jwt-secret are configured with other HTTP enabling annotations, or with ssl passthrough.