Overlay Design for a Virtual Infrastructure Workload Domain

ESXi Host Transport Nodes

A transport node in NSX-T Data Center is a node that is capable of participating in an NSX data plane. The VI workload domain contains multiple ESXi hosts in a vSphere cluster to support workloads. You register these ESXi hosts as transport nodes so that networks and workloads on that host can use the capabilities of NSX-T Data Center. During the preparation process, the native vSphere Distributed Switch for the cluster in the VI workload domain is extended with NSX-T Data Center capabilities.

Table 1. Design Decisions on ESXi Host Transport Nodes
Decision ID	Design Decision	Design Justification	Design Implication
VCF-WLD-NSX-SDN-022	Enable all ESXi hosts in the VI workload domain as transport nodes in NSX-T Data Center.	Enables distributed routing, logical segments, and distributed firewall.	None.
VCF-WLD-NSX-SDN-023	Configure each ESXi host as a transport node without using transport node profiles.	Enables the participation of ESXi hosts and the virtual machines on them in NSX overlay and VLAN networks. Transport node profiles can only be applied at the cluster level. Because in an environment with multiple availability zones each availability zone is connected to a different set of VLANs, you cannot use a transport node profile.	You must configure each transport node with an uplink profile individually.

Host TEP IP Assignment

A host TEP (tunnel end-point) interface is created for each uplink on the allocated distributed switch. This TEP interface is used for data plane communication between transport nodes. The IP address for the TEP interface can be statically assigned by using SDDC Manager or dynamically assigned over DHCP.


IP Assignment Method	Process	Advantages	Disadvantages
DHCP by using a DHCP server in the data center	A DHCP scope with a pool of available IP addresses, gateway, and subnet information for each overlay VLAN is configured in a DHCP server. Newly-configured hosts require a DHCP address on the overlay VLAN.	Enables clusters to span racks and Layer 2 domains. Enables clusters to span availability zones.	Requires a highly available DHCP server for each overlay VLAN. Potential configuration of DHCP-Helper on the top of rack switch.
Static by using SDDC Manager	A group or pool of available IP addresses, gateway, and subnet information for each overlay VLAN is provided to SDDC Manager. Newly-configured hosts are allocated one or more addresses from that pool during the initial host configuration.	Removes the need for a DHCP server.	Not compatible with some deployment topologies of VMware Cloud Foundation, such as multiple availability zones or clusters which span Layer 3 domains. Requires careful IP management.

Table 2. Design Decision on Host TEP Addressing for NSX-T Data Center
Decision ID	Design Decision	Design Justification	Design Implication
VCF-WLD-NSX-SDN-024	Use DHCP to assign IP addresses to the host TEP interfaces.	Required for deployments where a cluster spans Layer 3 network domains such as multiple availability zones and clusters in the VI workload domain that span Layer 3 domains.	DHCP server is required for the host overlay VLANs.

Virtual Switches

NSX segments are logically abstracted segments to which you can connect customer workloads. A single segment is mapped to a unique Geneve segment that is distributed across the ESXi hosts in a transport zone. The segment supports line-rate switching in the ESXi host without the constraints of VLAN sprawl or spanning tree issues.

Consider the following limitations of distributed switches:

Distributed switches are manageable only when the vCenter Server instance is available. You can consider vCenter Server a Tier-1 application.
Distributed switches with NSX-T Data Center capabilities are manageable only when the vCenter Server instance and NSX Manager cluster is available. You can consider vCenter Server and NSX Manager as Tier-1 applications.
N-VDS instances are manageable only when the NSX Manager cluster is available. You can consider the NSX Manager cluster as a Tier-1 application.

Table 3. Design Decision on Virtual Switches for NSX-T Data Center
Decision ID	Design Decision	Design Justification	Design Implication
VCF-WLD-NSX-SDN-025	Use a vSphere Distributed Switch for the shared edge and workload cluster that is enabled for NSX-T Data Center.	Uses the existing vSphere Distributed Switch. Provides NSX logical segment capabilities to support advanced use cases. To use features such as distributed routing, customer workloads must be connected to NSX segments.	Management occurs jointly from the vSphere Client to NSX Manager. However, you must perform all network monitoring in the NSX Manager user interface or another solution.

Configuration of the vSphere Distributed Switch with NSX-T Data Center

A cluster in the VI workload domain uses a single vSphere Distributed Switch with a configuration for system traffic types, NIC teaming, and MTU size. See vSphere Networking Design for a Virtual Infrastructure Workload Domain.

Note:

VMware Cloud Foundation supports one distributed switch with NSX per cluster. Custom distributed switches are not supported.

To support uplink and overlay traffic for the NSX Edge nodes for the VI workload domain, you must create several port groups on the vSphere Distributed Switch for the VI workload domain. The VMkernel adapter for the host TEP is connected to the host overlay VLAN but does not require a dedicated port group on the distributed switch. The VMkernel network adapter for the host TEP is automatically created when you configure the ESXi host as a transport node.

The NSX Edge appliances and the VMkernel adapter for the host TEP should be connected to different VLANs and subnets. The VLAN IDs for the NSX Edge nodes are mapped to the VLAN trunk port groups for the edge uplinks on the host.

Table 4. vSphere Distributed Switch Configuration for the Virtual Infrastructure Workload Domain
Function	Number of Physical NIC Ports	Teaming Policy	MTU
Management vSphere vMotion vSAN NFS Host Overlay Edge Uplinks and Overlay - VLAN Trunking	2	Load balance source for the ESXi traffic Failover order for the edge uplinks	9000

Table 5. Switch Configuration per Physical NIC
vmnic	Function	Connected to
0	Uplink	Top of rack switch 1
1	Uplink	Top of rack switch 2

Table 6. Segments for a Single Availability Zone
Segment Name	Type	Purpose
Management - first availability zone	VLAN	Management traffic
vMotion - first availability zone	VLAN	vSphere vMotion traffic
vSAN - first availability zone	VLAN	vSAN traffic
Edge Uplink 1	VLAN Trunk	Edge node overlay and uplink traffic to the first top of rack switch
Edge Uplink 2	VLAN Trunk	Edge node overlay and uplink traffic to the second top of rack switch
NFS storage - first availability zone	VLAN	NFS traffic
auto created (Host TEP)	-	Host overlay
auto created (Host TEP)	-	Host overlay
auto created (Hyperbus)	-	-

In a deployment that has multiple availability zones, you must provide new or extend existing networks.

Table 7. Additional Segments for a Second Availability Zone
Segment Name	Type	Availability Zone	Purpose
Management - second availability zone	VLAN	Second availability zone	Management traffic in the second availability zone
vMotion - second availability zone	VLAN	Second availability zone	vSphere vMotion traffic in the second availability zone
vSAN - second availability zone	VLAN	Second availability zone	vSAN traffic in the second availability zone
Edge Uplink 1	VLAN Trunk	Stretched between both availability zones	Edge node overlay and uplink traffic to the first top of rack switch in the second availability zone Required only in vSphere clusters that contain NSX Edge appliances.
Edge Uplink 2	VLAN Trunk	Stretched between both availability zones	Edge node overlay and uplink traffic to the first top of rack switch in the second availability zone Required only in vSphere clusters that contain NSX Edge appliances.
NFS storage - second availability zone	VLAN	Second availability zone	-
auto created (Host TEP)	-	-	Host overlay
auto created (Host TEP)	-	-	Host overlay
auto created (Hyperbus)	-	-	-

Virtual Segments

Geneve provides the overlay capability in NSX-T Data Center to create isolated, multi-tenant broadcast domains across data center fabrics, and enables customers to create elastic, logical networks that span physical network boundaries, and physical locations.

The first step in creating these logical networks is to isolate and pool the networking resources. By using the Geneve overlay, NSX-T Data Center isolates the network into a pool of capacity and separates the consumption of these services from the underlying physical infrastructure. This model is similar to the model vSphere uses to abstract compute capacity from the server hardware to create virtual pools of resources that can be consumed as a service. You can then organize the pool of network capacity in logical networks that are directly attached to specific applications.

Geneve is a tunneling mechanism which provides extensibility while still using the offload capabilities of NICs for performance improvement.

Geneve works by creating Layer 2 logical networks that are encapsulated in UDP packets. A Segment ID in every frame identifies the Geneve logical networks without the need for VLAN tags. As a result, many isolated Layer 2 networks can coexist on a common Layer 3 infrastructure using the same VLAN ID.

In the vSphere architecture, the encapsulation is performed at the TEP VMkernel adapter of the ESXi host and before being sent on the physical network, making the Geneve overlay transparent to both the guest virtual machines and the underlying Layer 3 network. The Tier-0 Gateway provides gateway services between overlay and non-overlay hosts, for example, a physical server or the Internet router. The NSX Edge node translates overlay segment IDs to VLAN IDs, so that non-overlay hosts can communicate with virtual machines on an overlay network.

The NSX Edge cluster hosts all NSX Edge node instances that connect to the corporate network for secure and centralized network administration.

Table 8. Design Decisions on Geneve Overlay
Decision ID	Design Decision	Design Justification	Design Implication
VCF-WLD-NSX-SDN-026	To provide virtualized network capabilities to customer workloads, use overlay networks with NSX Edge nodes and distributed routing.	Creates isolated, multi-tenant broadcast domains across data center fabrics to deploy elastic, logical networks that span physical network boundaries. Enables advanced deployment topologies by introducing Layer 2 abstraction from the data center networks.	Requires configuring transport networks with an MTU size of at least 1,700 bytes.

Transport Zones

Transport zones determine which hosts can be connected to a particular network. A transport zone identifies the type of traffic, VLAN or overlay, and the vSphere Distributed Switch name. You can configure one or more VLAN transport zones and a single overlay transport zone per virtual switch. A transport zone does not represent a security boundary.

The ESXi hosts in a VI workload domain and NSX edge nodes are connected to one overlay transport zone. The edge uplinks are connected to a separate VLAN transport zone. — Figure 1. Transport Zone Design

Table 9. Design Decision on the Transport Zone Configuration for NSX-T Data Center
Decision ID	Design Decision	Design Implication	Design Justification
VCF-WLD-NSX-SDN-027	Create a single overlay transport zone for all overlay traffic across the VI workload domain and NSX Edge nodes.	Ensures that overlay segments are connected to an NSX Edge node for services and north-south routing. Ensures that all segments are available to all ESXi hosts and NSX Edge nodes configured as transport nodes.	None.
VCF-WLD-NSX-SDN-028	Create a single VLAN transport zone for uplink VLAN traffic that is applied only to NSX Edge nodes.	Ensures that uplink VLAN segments are configured on the NSX Edge transport nodes.	If VLAN segments are needed on hosts, you must create another VLAN transport zone for the host transport nodes only.

Uplink Policy for ESXi Host Transport Nodes

Uplink profiles define policies for the links from ESXi hosts to NSX-T segments or from NSX Edge appliances to top of rack switches. By using uplink profiles, you can apply consistent configuration of capabilities for network adapters across multiple ESXi hosts or NSX Edge nodes.

Uplink profiles can use either load balance source or failover order teaming. If using load balance source, multiple uplinks can be active. If using failover order, only a single uplink can be active.

Table 10. Design Decisions on the Uplink Profile for ESXi Transport Nodes
Decision ID	Design Decision	Decision Justification	Decision Implication
VCF-WLD-NSX-SDN-029	Create an uplink profile with the load balance source teaming policy with two active uplinks for ESXi hosts.	For increased resiliency and performance, supports the concurrent use of both physical NICs on the ESXi hosts that are configured as transport nodes.	None.

Replication Mode of Segments

The control plane decouples NSX-T Data Center from the physical network. The control plane handles the broadcast, unknown unicast, and multicast (BUM) traffic in the virtual segments.

The following options are available for BUM replication on segments.

Table 11. BUM Replication Modes of NSX-T Segments
BUM Replication Mode	Description
Hierarchical Two-Tier	The ESXi host transport nodes are grouped according to their TEP IP subnet. One ESXi host in each subnet is responsible for replication to an ESXi host in another subnet. The receiving ESXi host replicates the traffic to the ESXi hosts in its local subnet. The source ESXi host transport node knows about the groups based on information it has received from the NSX-T Controller. The system can select an arbitrary ESXi host transport node as the mediator for the source subnet if the remote mediator ESXi host node is available.
Head-End	The ESXi host transport node at the origin of the frame to be flooded on a segment sends a copy to every other ESXi host transport node that is connected to this segment.

Table 12. Design Decisions on Segment Replication Mode
Decision ID	Design Decision	Design Justification	Design Implications
VCF-WLD-NSX-SDN-030	Use hierarchical two-tier replication on all NSX-T overlay segments.	Hierarchical two-tier replication is more efficient by reducing the number of ESXi hosts the source ESXi host must replicate traffic to.	None.