NSX-T Data Center resources that you need to configure include an overlay transport zone, a tier-0 logical router, a logical switch to connect the node VMs, IP blocks for Kubernetes nodes, and an IP pool for SNAT.
In the NCP configuration file ncp.ini, the NSX-T Data Center resources are specified using their UUIDs or names.
Overlay Transport Zone
Log in to NSX Manager and find the overlay transport zone that is used for container networking or create a new one.
Specify an overlay transport zone for a cluster by setting the overlay_tz option in the [nsx_v3] section of ncp.ini. This step is optional. If you do not set overlay_tz, NCP will automatically retrieve the overlay transport zone ID from the tier-0 router.
Tier-0 Logical Routing
Log in to NSX Manager and find the router that is used for container networking or create a new one.
Specify a tier-0 logical router for a cluster by setting the tier0_router option in the [nsx_v3] section of ncp.ini.
- tag: <cluster_name>, scope: ncp/cluster
- tag: <node_name>, scope: ncp/node_name
IP Blocks for Kubernetes Pods
Log in to NSX Manager and create one or more IP blocks. Specify the IP block in CIDR format.
Specify IP blocks for Kubernetes pods by setting the container_ip_blocks option in the [nsx_v3] section of ncp.ini.
You can also create IP blocks specifically for no-SNAT namespaces.
Specify no-SNAT IP blocks by setting the no_snat_ip_blocks option in the [nsx_v3] section of ncp.ini.
If you create no-SNAT IP blocks while NCP is running, you must restart NCP. Otherwise, NCP will keep using the shared IP blocks until they are exhausted.
IP Pool for SNAT
The IP pool is used for allocating IP addresses which will be used for translating pod IPs via SNAT rules, and for exposing ingress controllers via SNAT/DNAT rules, just like Openstack floating IPs. These IP addresses are also referred to as external IPs.
Multiple Kubernetes clusters use the same external IP pool. Each NCP instance uses a subset of this pool for the Kubernetes cluster that it manages. By default, the same subnet prefix for pod subnets will be used. To use a different subnet size, update the external_subnet_prefix option in the [nsx_v3] section in ncp.ini.
Log in to NSX Manager and create a pool or find an existing pool.
Specify IP pools for SNAT by setting the external_ip_pools option in the [nsx_v3] section of ncp.ini.
apiVersion: v1 kind: Service metadata: name: svc-example annotations: ncp/snat_pool: <external IP pool ID or name> selector: app: example ...
- The IP pool specified by ncp/snat_pool should already exist in NSX-T Data Center before the service is configured. The IP pool must have the tag
scope: ncp/owner, tag: cluster:<cluster_name>.
- In NSX-T Data Center, the priority of the SNAT rule for the service is higher than that for the project.
- If a pod is configured with multiple SNAT rules, only one will work.
scope: ncp/owner, tag: ns:<namespace_UUID>
oc get ns -o yaml
- Each tag should specify one UUID. You can create multiple tags for the same pool.
- If you change the tags after some namespaces have been allocated IPs based on the old tags, those IPs will not be reclaimed until the SNAT configurations of the services change or NCP restarts..
- The namespace owner tag is optional. Without this tag, any namespace can have IPs allocated from the SNAT IP pool.
(Optional) Firewall Marker Sections
To allow the administrator to create firewall rules and not have them interfere with NCP-created firewall sections based on network policies, log in to NSX Manager and create two firewall sections.
Specify marker firewall sections by setting the bottom_firewall_section_marker and top_firewall_section_marker options in the [nsx_v3] section of ncp.ini.
The bottom firewall section must be below the top firewall section. With these firewall sections created, all firewall sections created by NCP for isolation will be created above the bottom firewall section, and all firewall sections created by NCP for policy will be created below the top firewall section. If these marker sections are not created, all isolation rules will be created at the bottom, and all policy sections will be created at the top. Multiple marker firewall sections with the same value per cluster are not supported and will cause an error.