NSX-T resources that you need to configure include an overlay transport zone, a tier-0 logical router, a logical switch to connect the node VMs, IP blocks for Kubernetes nodes, and an IP block or pool for SNAT.

Overlay Transport Zone

The overlay Transport Zone for a cluster is identified by the tag {'ncp/cluster': '<cluster_name>'}. Log in to NSX Manager and navigate to Fabric > Transport Zones. Find the overlay transport zone that is used for container networking, or create a new one. Tag the transport zone with the name of the cluster being configured. Specifically, <cluster_name> must match the value of the cluster option in the [coe] section in ncp.ini. You can add more than one tag to the transport zone to make it shared.

Tier-0 Logical Routing

The tier-0 logical router for a cluster is identified by the tag {'ncp/cluster': '<cluster_name>'}. Log in to NSX Manager and navigate to Routing > ROUTERS. You can either create a new tier-0 logical router for the Kubernetes cluster, or use an existing one. After you identify the router, tag it with {'ncp/cluster': '<cluster_name>'}.

The <cluster_name> value must match the value of the cluster option in the [coe] section in ncp.ini. You can add more than one tag to the router to make it shared.

Note:

The router must be created in active-standby mode.

Logical Switch

The vNICs used by the node for data traffic must be connected to an overlay logical switch. It is not mandatory for the node's management interface to be connected to NSX-T, although doing so will make setting up easier. You can create a logical switch by logging in to NSX Manager and navigating to Switching > Switches. On the switch, create logical ports and attach the node vNICs to them. The logical ports must be tagged with {'ncp/cluster': '<cluster_name>'} and {'ncp/node_name': '<node_name>'}. The <cluster_name> value must match the value of the cluster option in the [coe] section in ncp.ini.

IP Blocks for Kubernetes Pods

Create one or more IP blocks for the Kubernetes pods. You can log in to NSX Manager and navigate to DDI > IPAM to create IP blocks. Specify the IP block in CIDR format. Also specify the tag ncp/cluster for the block.

You can also create IP blocks specifically for no-SNAT namespaces. These IP blocks require the tag {'ncp/no_snat': '<cluster_name>'} in addition to the ncp/cluster tag. If you create no-SNAT IP blocks while NCP is running, you must restart NCP. Otherwise, NCP will keep using the shared IP blocks until they are exhausted.

IP Block or IP Pool for SNAT

These resources will be used for allocating IP addresses which will be used for translating Pod IPs via SNAT rules, and for exposing ingress controllers via SNAT/DNAT rules - just like Openstack floating IPs. In this guide, these IP addresses are also referred to as *external IPs*. Users can either configure a *global* external IP block or a cluster specific external IP pool.

To set up an external IP block, log in to NSX Manager and navigate to DDI > IPAM. Specify a CIDR value with a network address and not a host address. For example, specify 4.3.0.0/16 instead of 4.3.2.1/16. Tag the IP block with the following key and value to indicate that the IP block is for external IP allocation.

{'ncp/external': 'true'}

Multiple Kubernetes clusters use the same external IP pool. Each NCP instance uses a subset of this pool for the Kubernetes cluster that it manages. By default, the same subnet prefix for pod subnets will be used. To use a different subnet size, update the external_subnet_prefix option in the [nsx_v3] section in ncp.ini.

To use a cluster-specific IP pool for allocating external IPs, log in to NSX Manager and navigate to Inventory > Groups > IP POOL. Create or use an existing pool. Apply the following tags to the pool.

{'ncp/cluster': 'true'}
{'ncp/external': 'true'}

(Optional) Firewall Marker Section

To allow the administrator to create firewall rules and not have them interfere with NCP-created firewall sections, log in to NSX Manager, navigate to Firewall > General and create an empty firewall section and tag it with {'ncp/fw_sect_marker': 'true'}. With this marker firewall section created, all subsequent firewall sections created by NCP for network policies and namespace isolation will be placed above this firewall section, and firewall rules created by the administrator will be placed below this marker firewall section.

If this marker section is not created, all isolation rules will be created at the bottom. Multiple marker firewall sections per cluster is not supported and will cause an error.