NSX-T Data Center resources that you need to configure include an overlay transport zone, a tier-0 logical router, a logical switch to connect the node VMs, IP blocks for Kubernetes nodes, and an IP pool for SNAT.

You configure NSX-T Data Center resources using UUIDs or names in the configuration file ncp.ini.

Overlay Transport Zone

Log in to NSX Manager and navigate to Fabric > Transport Zones. Find the overlay transport zone that is used for container networking or create a new one.

Specify an overlay transport zone for a cluster by setting the overlay_tz option in the [nsx_v3] section of ncp.ini. This step is optional. If you do not set overlay_tz, NCP will automatically retrieve the overlay transport zone ID from the tier-0 router.

Tier-0 Logical Routing

Log in to NSX Manager and navigate to Networking > Routing > Routers. Find the router that is used for container networking or create a new one.

Specify a tier-0 logical router for a cluster by setting the tier0_router option in the [nsx_v3] section of ncp.ini.

Note:

The router must be created in active-standby mode.

Logical Switch

The vNICs used by the node for data traffic must be connected to an overlay logical switch. It is not mandatory for the node's management interface to be connected to NSX-T Data Center, although doing so will make setting up easier. You can create a logical switch by logging in to NSX Manager and navigating to Networking > Switching > Switches. On the switch, create logical ports and attach the node vNICs to them. The logical ports must have the following tags:

  • tag: <cluster_name>, scope: ncp/cluster

  • tag: <node_name>, scope: ncp/node_name

The <cluster_name> value must match the value of the cluster option in the [coe] section in ncp.ini.

IP Blocks for Kubernetes Pods

Log in to NSX Manager and navigate to Networking > IPAM to create one or more IP blocks. Specify the IP block in CIDR format.

Specify IP blocks for Kubernetes pods by setting the container_ip_blocks option in the [nsx_v3] section of ncp.ini.

You can also create IP blocks specifically for no-SNAT namespaces.

Specify no-SNAT IP blocks by setting the no_snat_ip_blocks option in the [nsx_v3] section of ncp.ini.

If you create no-SNAT IP blocks while NCP is running, you must restart NCP. Otherwise, NCP will keep using the shared IP blocks until they are exhausted.

Note:

When you create an IP block, the prefix must not be larger than the value of the parameter subnet_prefix in NCP's configuration file ncp.ini.

IP Pool for SNAT

The IP pool is used for allocating IP addresses which will be used for translating pod IPs via SNAT rules, and for exposing ingress controllers via SNAT/DNAT rules, just like Openstack floating IPs. These IP addresses are also referred to as external IPs.

Multiple Kubernetes clusters use the same external IP pool. Each NCP instance uses a subset of this pool for the Kubernetes cluster that it manages. By default, the same subnet prefix for pod subnets will be used. To use a different subnet size, update the external_subnet_prefix option in the [nsx_v3] section in ncp.ini.

Log in to NSX Manager and navigate to Inventory > Groups > IP Pools to create a pool or find an existing pool.

Specify IP pools for SNAT by setting the external_ip_pools option in the [nsx_v3] section of ncp.ini.

You can also configure SNAT for a specific service by adding an annotation to the service. For example,

    apiVersion: v1
    kind: Service
    metadata:
      name: svc-example
      annotations:
        ncp/snat_pool: <external IP pool ID or name>
      selector:
        app: example
    ...

NCP will configure the SNAT rule for this service. The rule's source IP is the set of backend pods. The destination IP is the SNAT IP allocated from the specified external IP pool. Note the following:

  • The IP pool specified by ncp/snat_pool should already exist in NSX-T Data Center before the service is configured. Starting with NCP 2.3.1, the IP pool must have the tag {"ncp/owner": cluster:<cluster>}.

  • In NSX-T Data Center, the priority of the SNAT rule for the service is higher than that for the project.

  • If a pod is configured with multiple SNAT rules, only one will work.

You can specify which namespace can be allocated IPs from the SNAT IP pool by adding the following tag to the IP pool.

  • scope: ncp/owner, tag: ns:<namespace_UUID>

You can get the namespace UUID with one of the following command:

oc get ns -o yaml

Note the following:

  • Each tag should specify one UUID. You can create multiple tags for the same pool.

  • If you change the tags after some namespaces have been allocated IPs based on the old tags, those IPs will not be reclaimed until the SNAT configurations of the services change or NCP restarts..

  • The namespace owner tag is optional. Without this tag, any namespace can have IPs allocated from the SNAT IP pool.

(Optional) Firewall Marker Sections

To allow the administrator to create firewall rules and not have them interfere with NCP-created firewall sections based on network policies, log in to NSX Manager, navigate to Security > Distributed Firewall > General and create two firewall sections.

Specify marker firewall sections by setting the bottom_firewall_section_marker and top_firewall_section_marker options in the [nsx_v3] section of ncp.ini.

The bottom firewall section must be below the top firewall section. With these firewall sections created, all firewall sections created by NCP for isolation will be created above the bottom firewall section, and all firewall sections created by NCP for policy will be created below the top firewall section. If these marker sections are not created, all isolation rules will be created at the bottom, and all policy sections will be created at the top. Multiple marker firewall sections with the same value per cluster are not supported and will cause an error.