NCP will create one layer 7 load balancer for Ingresses with TLS specification, and one layer 7 load balancer for Ingresses without TLS specification. You can also create CRDs (CustomResourceDefinitions) to handle Ingress scaling.

Note the following:
  • All Ingresses will get a single IP address.
  • The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools option in the [nsx_v3] section in ncp.ini. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
  • The Ingress resource is allocated an IP address from the external IP pool specified by the external_ip_pools_lb option in the [nsx_v3] section in ncp.ini. If the external_ip_pools_lb option does not exist, the pool specified by external_ip_pools is used. The load balancer is exposed on this IP address and the HTTP and HTTPS ports (80 and 443).
  • You can change to a different IP pool by changing the configuration and restarting NCP.
  • You can specify a default certificate for TLS. See below for information about generating a certificate and mounting a certificate into the NCP pod.
  • Ingresses without TLS specification will be hosted on HTTP virtual server (port 80).
  • Ingresses with TLS specification will be hosted on HTTPS virtual server (port 443). The load balancer will act as an SSL server and terminate the client SSL connection.
  • Modification of Ingress by adding or removing the TLS section is supported. When the tls key is removed from the Ingress specification, the Ingress rules will be transferred from the HTTPS virtual server (port 443) to the HTTP virtual server (port 80). Similarly, when the tls key is added to Ingress specification, the Ingress rules are transferred from the HTTP virtual server (port 80) to the HTTPS virtual server (port 443).
  • If there are duplicate rules in Ingress definitions for a single cluster, only the first rule will be applied. The other Ingresses with the duplicate rules will be annotated with error. For example, if you create two Ingresses with the same host and path, and one Ingress is TLS while and the other is non-TLS, and kubernetes.io/ingress.allow-http is false, the two rules will be created on different virtual servers and will not conflict with each other. However, if kubernetes.io/ingress.allow-http is true, the Ingress that is applied later will be annotated with an error. See the "Errors" section below for more information.
  • Only a single Ingress with a default backend is supported per cluster. Traffic not matching any Ingress rule will be forwarded to the default backend.
  • If there are multiple Ingresses with a default backend, only the first one will be configured. The others will be annotated with an error. See the "Errors" section below for more information.
  • (NCP 3.0.0 and 3.0.1) The rules are applied in the following order:
    1. Rules with both host and path specified, and without regular expression matching.
    2. Rules with both host and path specified, and with regular expression matching (with the longest Ingress path first).
    3. Rules with host or path specified, and without regular expression matching.
    4. Rules with host or path specified, and with regular expression matching (with the longest Ingress path first).
  • (NCP 3.0.2 and later) The rules are applied in the following order:
    1. Rules with both host and path specified.
      1. Rules with the Exact pathType.
      2. Rules with the Prefix pathType.
      3. Rules with the Regex pathType.
    2. Rules with host or path specified.
      1. Rules with the Exact pathType.
      2. Rules with the Prefix pathType.
      3. Rules with the Regex pathType.

    Note: If multiple paths match a request, precedence will be given to the longest matching path.

    About pathType:
    • ImplementationSpecific is treated the same as Exact.
    • Two matching paths with different pathTypes can co-exist.
    • For the Prefix type, /foo will match /foo/, /foo/bar but not /foo or /foobar. To match /foo, add the Exact path /foo to the Ingress rule.
  • (This is applicable if you use Policy mode.) If a TLS Ingress is created with a default backend, it is recommended that you set up the default backend to respond to both HTTP and HTTPS requests because:
    • The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/PKS use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
    • The load balancer will re-encrypt and send HTTPS request to the backend server if there is no TLS Ingress (in the cluster for the Kubernetes/PKS use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.
  • (NCP 3.0.0 and 3.0.1) The IngressClass resource is not supported.
  • (NCP 3.0.2 and later) The IngressClass resource is supported.
  • (NCP 3.0.0 and 3.0.1) The pathType attribute is not supported.
  • (NCP 3.0.2 and later) The pathType attribute is supported.
  • (NCP 3.0.2 and later) JWT (JSON Web Token) client authentication is supported. This feature requires NSX-T Data Center 3.0.0 or later.
Starting with Kubernetes 1.18, the kubernetes.io/ingress.class annotation is deprecated and replaced by the IngressClass resource. To specify NSX-T as the Ingress controller in the IngressClass resource, set the controller parameter to k8s.io/nsx. For example: 
kind: IngressClass
metadata:
  name: nsx-lb
  annotations:
    ingressclass.kubernetes.io/is-default-class: true
spec:
  controller: k8s.io/nsx
To specify a third-party Ingress controller, set the controller parameter to k8s.io/<controller name>. For example: 
kind: IngressClass
metadata:
  name: nginx-lb
  annotations:
    ingressclass.kubernetes.io/is-default-class: false
spec:
  controller: k8s.io/ingress-nginx

To set the IngressClass as the default for the cluster, set the annotation ingressclass.kubernetes.io/is-default-class to true. See the example above.

You must not use both the kubernetes.io/ingress.class annotation and the IngressClass resource.

Feature Annotations

The following table lists annotations that NCP supports:
Annotation Description Supported Values Default Value NCP Version
kubernetes.io/ingress.allow-http Enables HTTP requests in addition to HTTPS true, false true 2.5 and later
ncp/use-regex Enables path pattern matching true, false false 2.5.1 and later
ingress.kubernetes.io/rewrite-target Rewrites path of incoming request
  • (NCP 2.5 and later) Path starting with ‘/’
  • (NCP 2.5.1 and later, and NSX-T 2.5.1 and later) Path with a numbered placeholder for a group captured in a regular expression, for example, /$1
 No default value See the Supported Values column
ncp/http-redirect Redirects HTTP requests to HTTPS true, false false 2.5.1 and later
kubernetes.io/ingress.class Indicates which Ingress controller is responsible for this Ingress nsx, nginx, etc. nsx 2.5 and later
nsx/loadbalancer Places an Ingress on a dedicated load balancer Name of a LoadBalancer CRD No default value 2.5.1 and later
ncp/whitelist-source-range Limits Ingress access by request source IP Comma-separated list of CIDRs, IP addresses, or IP ranges. No default value 3.0.0 and later
ncp/ssl-mode Selects SSL mode for all the rules in an Ingress offload, reencrypt, passthrough offload 3.0.1 and later
ncp/jwt-alg Determines the algorithm to be used to validate JWT signature. Enables JWT client authentication when set with ncp/jwt-secret. HS256, RS256 No default value 3.0.2 and later
ncp/jwt-secret Specifies the name of the Kubernetes secret that contains the JWT secret or public key used for signature validation. Enables JWT client authentication when set with ncp/jwt-alg. Name of a Kubernetes secret No default value 3.0.2 and later
ncp/jwt-token Additional location to search for JWT in the HTTP request. _arg_<param_name>, _cookie_<cookie_name> No default value 3.0.2 and later
ncp/jwt-realm Specifies the Realm header returned with 401 when authentication failed. String indicating the realm Hostname of the ingress rule 3.0.2 and later
ncp/jwt-preserve Option to keep JWT and pass to backend after successful authentication. true, false true 3.0.2 and later
Details about the annotations:
  • Path Regex (Regular Expression) Matching
    • For NCP 2.5.0 and earlier

      In NCP 2.5.0 and earlier, all sub-path matching is automatically enabled using the regular expression characters '.' and '*'. For example, the path /coffee/.* matches /coffee/ followed by zero, one or more characters, such as /coffee/, /coffee/a, and /coffee/b, but not /coffee, /coffeecup or /coffeecup/a.

      An Ingress specification example: 
      kind: Ingress
metadata:
  name: cafe-ingress
spec:
  rules:
  - http:
      paths:
      - path: /coffee/.*    #Matches /coffee/, /coffee/a but NOT /coffee, /coffeecup, etc.
        backend:
          serviceName: coffee-svc
          servicePort: 80
    • For NCP 2.5.1 and later
      You can enable or disable regular expression matching of the Ingress path (but not host) parameter using the annotation ncp/use-regex. If set to false, exact path matching will be performed by doing the equals match. If set to true, regular expression matching will be performed by adding the start of string character (^) and end of string character ($) to the path so that the entire request URI matches the pattern. Note that when using the OR operator (|), always specify the scope with parentheses so that ^ and $ apply to all the operands. For example, path: /(tea|coffee|milk). An Ingress specification example: 
      kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nsx"
    ncp/use-regex: "true"
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea/.*
        backend:
          serviceName: tea-svc
          servicePort: 80
    • Updating Ingresses prior to upgrading NCP to 2.5.1 or later
      This is required only if you have Ingresses requiring all sub-path matching using the characters '.' and '*'.
      1. Update the Ingresses to include the annotation ncp/use-regex: true.
      2. For all sub-path matching, if you have paths such as /coffee/* or /*, change them to /coffee/.* and /.*.

        /coffee/.* will match /coffee/, /coffee/a, /coffee/b, /coffee/a/b, and so on. /.* will match /coffee, /tea, /coffee/a, and so on. Omitting the path will produce the same behavior as path: /.*.

  • Example of the annotation ingress.kubernetes.io/rewrite-target: 
    kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

    The paths /tea and /coffee will be rewritten to / before the URL is sent to the backend service.

    Starting with NCP 2.5.1 and NSX-T 2.5.1, if path is specified using a regular expression, the captured groups are saved in numbered placeholders in the form $1, $2, and so on. These placeholders can be used as parameters in the ingress.kubernetes.io/rewrite-target annotation. Named capture groups and named placeholders are not supported. For example, 
    kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nsx"
    ncp/use-regex: "true"
    #/tea/cup will be rewritten to /cup before sending request to endpoint
    ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea/(.*)
        backend:
          serviceName: tea-svc
          servicePort: 80
  • About the annotation kubernetes.io/ingress.allow-http:
    • If the annotation is set to false, rules will be created for the HTTPS virtual server.
    • If the annotation is set to true or missing, rules will created for both HTTP and HTTPS virtual servers. Note that HTTPS rules will be created only if the TLS section is present in the Ingress specification.
  • About the annotation ncp/http-redirect:
    • If the annotation is set to false, Incoming HTTP traffic (port 80) to HTTP Virtual Server will not be redirected to HTTPS Virtual Server.
    • If the annotation is set to true, Incoming HTTP traffic (port 80) to HTTP Virtual Server will be redirected to HTTPS Virtual Server (port 443).

    This annotation is only valid if the TLS section is present. This annotation takes precedence over kubernetes.io/ingress.allow-http. When set to true, it will direct matched HTTP traffic to HTTPS, regardless of how kubernetes.io/ingress.allow-http is set.

  • About the annotation kubernetes.io/ingress.class:
    • If the value is nsx, this ingress will be handled by NCP. If any other value is specified, the Ingress will be ignored by NCP. For more info see Third-party Ingress Controllers.
  • For more information about the annotation nsx/loadbalancer, see LoadBalancer CRDs to Handle Ingress Scaling.
  • About the annotation ncp/whitelist-source-range:
    • When set, an incoming request will only be accepted if its source IP is included in this annotation. Otherwise, traffic will be dropped.
    • You can specify a combination of CIDR, IP addresses, and IP ranges. For example, 1.1.1.1/24, 2.2.2.2, 3.3.3.3-4.4.4.4.
    • Example YAML:
      kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nsx"
    ncp/whitelist-source-range: "192.168.128.0/17, 192.168.64.0-192.168.64.255, 192.168.16.32"
spec:
  tls:
  - hosts:
    - cafe.example.com
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
  • About the annotation ncp/ssl-mode:
    • This annotation applies to all the rules in an Ingress and the load balancer will operate in the selected SSL mode for these rules.

      Note: If ncp/ssl-mode is set to reencrypt or passthrough, kubernetes.io/ingress.allow-http must be set to False. This annnotation cannot be set to passthrough if the ingress.kubernetes.io/rewrite-target, ncp/use-regex or ncp/whitelist-source-range annotation is set.

  • About JWT client authentication:
    • To enable JWT client authentication for all rules under an Ingress, both ncp/jwt-alg and ncp/jwt-secret need to be set to a valid value. When enabled, incoming HTTP traffic will only be passed to the backend if bearing valid JSON web token. Otherwise, traffic will be rejected with the 401 status.
    • This feature is incompatible with the following annotations:
      • kubernetes.io/ingress.allow-http: true
      • ncp/http-redirect: true
      • ncp/ssl-mode: passthrough
    • ncp/jwt-alg:
      • Supported symmetrical algorithms: HS256
      • Supported asymmetrical algorithms: RS256
    • ncp/jwt-secret:
      • A symmetrical key or public certificate must be configured in a Kubernetes secret with the name specified in this annotation under the same namespace as the Ingress.
      • For symmetrical algorithms, the secret must be stored in the jwt.key field.
      • For asymmetrical algorithms, the public cert must be stored in the tls.crt field.
      • JWT client authentication will be disabled if the symmetrical secret or public certificate is not stored in the locations mentioned above, or if the data stored in the secret is invalid.
    • ncp/jwt-token:
      • Only one item can be configured in this annotation.
      • _arg_<param_name>: For JWT passed in as a URI parameter. Specify the parameter name that contains JWT.
      • _cookie_<cookie_name>: For JWT passed in as a cookie. Specify the cookie name that contains JWT.

Errors

Errors are annotated to the Ingress resource. The error key is ncp/error.loadbalancer and the warning key is ncp/warning.loadbalancer. The possible error and warning are:
  • ncp/error.loadbalancer: DEFAULT_BACKEND_IN_USE

    This error indicates that an Ingress with a default backend already exists. The Ingress will be inactive. This error will occur if (1) this Ingress is for HTTP and another Ingress for HTTP with a default backend exists; (2) this Ingress is for HTTPS and another Ingress for HTTPS with a default backend exists; or (3) this Ingress is for HTTP and HTTPS and another Ingress for HTTP and HTTPS with a default backend exists. To fix the error, delete and recreate the Ingress with a correct specification.

  • ncp/warning.loadbalancer: SECRET_NOT_FOUND

    This error indicates that the secret specified in the Ingress specification does not exist, or if ncp/jwt-alg and ncp/jwt-secret are anootated but the secret cannot be found under the same namespace as the Ingress. The Ingress will be partially active. To fix the error, create the missing secret. Note that once a warning is in the annotation, it will not be cleared during the life cycle of the Ingress resource.

  • ncp/warning.loadbalancer: INVALID_INGRESS
    This error indicates that one of the following conditions is true. The Ingress will be inactive. To fix the error, delete and recreate the Ingress with a correct specification.
    • An Ingress rule conflicts with another Ingress rule in the same Kubernetes cluster. Conflicts are determined only for Ingresses with the same match strategy, that is, the same ncp/use-regex annotation value.
    • The kubernetes.io/ingress.allow-http annotation is set to false and the Ingress does not have a TLS section.
    • The ncp/http-redirect annotation is set to true and the Ingress does not have a TLS section.
    • An Ingress rule does not have host and path specified. Such an Ingress rule has the same functionality as the Ingress default backend. Use the Ingress default backend instead.
    • The Ingress has JWL annotations that cannot be correctly processed. For example:
      • Either ncp/jwt-alg or ncp/jwt-secret is missing.
      • ncp/jwt-alg is configured with an unsupported algorithm.
      • ncp/jwt-alg and ncp/jwt-secret are configured with other HTTP enabling annotations, or with ssl passthrough.