Third-party Ingress Controllers

You can configure NCP to support third-party Ingress controllers.

Editing the ncp.ini file

You must edit the configuration file /var/vcap/data/jobs/ncp/xxxxxxxx/config/ncp.ini (where xxxxxxxx is the BOSH deployment ID). This file will then be copied to rootfs and used by NCP every time NCP restarts. The file must be edited on each master node.

Important: Changes to ncp.ini are not persistent across TKGI cluster updates. If you make changes through the TKGI tile and then update the TKGI deployment, the changes to ncp.ini will be lost.

The relevant options are in the nsx_v3 section.

• use_native_loadbalancer - If set to False, NCP will not process any Ingress or service of type Loadbalancer updates, regardless of its annotations. This setting applies to the whole TKGI cluster. The default is True.
• default_ingress_class_nsx - If set to True, NCP becomes the default Ingress controller and will handle both Ingresses annotated with kubernetes.io/ingress.class: "nsx" and Ingresses without any annotation. If set to False, NCP will only handle Ingresses annotated with kubernetes.io/ingress.class: "nsx". The default is True.
  Starting with NCP 3.2.1, default_ingress_class_nsx is deprecated. NCP will look at the following when resolving the Ingress class:

  • annotations
  • ingressClass objects
  • If no annotation is specified and use_native_loadbalancer is True, then NSX-LB will handle the Ingress. Otherwise, NSX-LB will not handle it.

If you want NCP to assign a floating IP to the NGINX controller pod and update the status of Ingresses with the floating IP, do the following:

• In the k8s section in ncp.ini, set ingress_mode=nat.
• Add the annotation ncp/ingress-controller: "True" to the NGINX Ingress controller pod.

NCP will update the status of Ingresses that have the annotation kubernetes.io/ingress.class: "nginx" with the NGINX Ingress controller pod's floating IP. If default_ingress_class_nsx=False, NCP will also update the status of Ingresses without the kubernetes.io/ingress.class annotation with the NGINX Ingress controller pod's floating IP.

Note: Even if the NGINX Ingress controller pod does not have the annotation ncp/ingress-controller: "True", NCP will update the status of the Ingresses mentioned above to loadBalancer: {}. The Ingresses could then be stuck in a loop where the NGINX controller updates the Ingress status to loadBalancer: {ingress: [{ip: <IP>}]} and NCP updates the Ingress status to loadBalancer: {}. To avoid this situation, perform the following steps:

• If the Ingress controller is from https://github.com/kubernetes/ingress-nginx,
  • On the Ingress controller, change the ingress-class to something other than "nginx".
  • If there is an Ingress with the annotation kubernetes.io/ingress-class: "nginx", change the annotation to a different value.
  • For more information, see https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress.
• If the Ingress controller is from https://github.com/nginxinc/kubernetes-ingress,
  • On the Ingress controller, change the ingress-class to something other than "nginx".
  • If there is an Ingress with the annotation kubernetes.io/ingress-class: "nginx", change the annotation to a different value.
  • On the Ingress controller pod, set use-ingress-class-only to True. This will stop this controller from updating Ingresses without the kubernetes.io/ingress-class annotation.
  • For more information, see https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/multiple-ingress.md.

For third-party Ingress controllers deployed in NAT mode, you can modify the http_ingress_port and https_ingress_port parameters in the k8s section to specify custom ports for the NAT rules exposed for the Ingress controller.

Scenario 1: NCP handles Ingresses but is not the default Ingress controller.

Follow this procedure to let NCP handle nsx-class Ingresses.

1. Edit the nsx_v3 section in ncp.ini on each master node.
   1. Set default_ingress_class_nsx to False.
   2. Leave use_native_loadbalancer set to True, the default value.
2. Restart NCP on each master node. This might cause a master failover.
3. Annotate all the Ingresses that you want NCP to handle with kubernetes.io/ingress.class: "nsx".

Scenario 2: NCP is the default Ingress controller.

Follow this procedure:

1. No need to edit ncp.ini, but ensure that every Ingress is annotated.
2. Ingresses to be handled by NCP should be annotated with kubernetes.io/ingress.class: "nsx".

   Although NCP will handle Ingresses without the kubernetes.io/ingress.class annotation, in the case of multiple Ingress controllers, the best practice is to always have the kubernetes.io/ingress.class annotation and not to rely on the default Ingress controller behavior.

3. Ingresses to be handled by third-party Ingress controllers must be annotated with the value required by those Ingress controllers.

Important: Unless the goal is to make NGINX the default Ingress controller, do not use nginx as the NGINX Ingress controller, because this will make NGINX the default Ingress controller.

Scenario 3: NCP does not handle any Ingress regardless of its annotation.

Follow this procedure:

1. Edit the nsx_v3 section in ncp.ini on each master node.
   1. Set use_native_loadbalancer to False. The value of default_ingress_class_nsx is now irrelevant.
2. Restart NCP on each master node. This might cause a master failover.

Note that NCP will also not handle services of type LoadBalancer