Design of the physical data center network includes defining the network topology for connecting physical switches and ESXi hosts, determining switch port settings for VLANs and link aggregation, and designing routing.

A software-defined network (SDN) both integrates with and uses components of the physical data center. SDN integrates with your physical network to support east-west transit in the data center and north-south transit to and from the SDDC networks.

Several typical data center network deployment topologies exist:

  • Core-Aggregation-Access

  • Leaf-Spine

  • Hardware SDN

Note:

Leaf-Spine is the default data center network deployment topology used for VMware Cloud Foundation

VLANs and Subnets for VMware Cloud Foundation

Configure your VLANs and subnets according to the guidelines and requirements for VMware Cloud Foundation.

When designing the VLAN and subnet configuration for your VMware Cloud Foundation deployment, consider the following guidelines:

Table 1. VLAN and Subnet Guidelines for VMware Cloud Foundation

All Deployment Topologies

Multiple Availability Zones

NSX Federation Between Multiple VMware Cloud Foundation Instances

  • Ensure your subnets are scaled appropriately to allow for expansion as expanding at a later time can be disruptive.

  • Use the IP address of the floating interface for Virtual Router Redundancy Protocol (VRPP) or Hot Standby Routing Protocol (HSRP) as the gateway.

  • Use the RFC 1918 IPv4 address space for these subnets and allocate one octet by VMware Cloud Foundation instance and another octet by function.

  • For network segments which are stretched between availability zones, the VLAN ID must meet the following requirements:

    • Be the same in both availability zones with the same Layer 3 network segments.

    • Have a Layer 3 gateway at the first hop that is highly available such that it tolerates the failure of an entire availability zone.

  • For network segments of the same type which are not stretched between availability zones, the VLAN ID can be the same or different between the zones.

  • An RTEP network segment should have a VLAN ID and Layer 3 range that are specific to the VMware Cloud Foundation instance.

  • In a VMware Cloud Foundation instance with multiple availability zones, the RTEP network segment must be stretched between the zones and assigned the same VLAN ID and IP range.

  • All Edge RTEP networks must be reachable from each other.

When deploying VLANs and subnets for VMware Cloud Foundation, they must conform to the following requirements according to the VMware Cloud Foundation topology:

Table 2. VLANs and Subnets for VMware Cloud Foundation

Function

VMware Cloud Foundation Instances with a Single Availability Zone

VMware Cloud Foundation Instances with Multiple Availability Zones

Management - first availability zone

  • Required

  • Highly available gateway within the instance

  • Required

  • Must be stretched within the instance

  • Highly available gateway across availability zones within the instance

vSphere vMotion - first availability zone

  • Required

  • Highly available gateway within the instance

  • Required

  • Highly available gateway in first availability zone within the instance

vSAN - first availability zone

  • Required

  • Highly available gateway within the instance

  • Required

  • Highly available gateway in first availability zone within the instance

Host overlay - first availability zone

  • Required

  • Highly available gateway within the instance

  • Required

  • Highly available gateway in first availability zone within the instance

Uplink01

  • Required

  • Gateway optional

  • Required

  • Gateway optional

  • Must be stretched within the instance

Uplink02

  • Required

  • Gateway optional

  • Required

  • Gateway optional

  • Must be stretched within the instance

Edge overlay

  • Required

  • Highly available gateway within the instance

  • Required

  • Must be stretched within the instance

  • Highly available gateway across availability zones within the instance

Management - second availability zone

  • Not required

  • Required

  • Highly available gateway in second availability zone within the instance

vSphere vMotion - second availability zone

  • Not required

  • Required

  • Highly available gateway in second availability zone within the instance

vSAN - second availability zone

  • Not required

  • Required

  • Highly available gateway in second availability zone within the instance

Host overlay - second availability zone

  • Not required

  • Required

  • Highly available gateway in second availability zone within the instance

Edge RTEP

  • Required for NSX Federation only

  • Highly available gateway within the instance

  • Required for NSX Federation only

  • Must be stretched within the instance

  • Highly available gateway across availability zones within the instance

Management and Witness - witness appliance at a third location

  • Not required

  • Required

  • Highly available gateway at the witness location

Leaf-Spine Physical Network Design Requirements and Recommendations for VMware Cloud Foundation

Leaf-Spine is the default data center network deployment topology used for VMware Cloud Foundation. Consider network bandwidth, access port configuration, jumbo frames, routing configuration, and DHCP availability for NSX in a deployment with a single or multiple VMware Cloud Foundation instances.

Leaf-Spine Physical Network Logical Design

Each ESXi host is connected redundantly to the top-of-rack (ToR) switches of the SDDC network fabric by two 25-GbE ports. The ToR switches are configured to provide all necessary VLANs using an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and NSX to guarantee that no physical interface is overrun and available redundant paths are used.
Figure 1. Leaf-Spine Physical Network Logical Design
Each ESXi host is connected redundantly to the ToR switches of the SDDC network fabric by two 25-GbE ports. ToR switches are connected to the spine.

Leaf-Spine Physical Network Design Requirements and Recommendations

The requirements and recommendations for the leaf-spine network configuration determine the physical layout and use of VLANs. They also include requirements and recommendations on jumbo frames, and on network-related requirements such as DNS and NTP.

Table 3. Leaf-Spine Physical Network Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NET-REQD-CFG-001

Do not use EtherChannel (LAG, LACP, or vPC) configuration for ESXi host uplinks.

  • Simplifies configuration of top of rack switches.

  • Teaming options available with vSphere Distributed Switch provide load balancing and failover.

  • EtherChannel implementations might have vendor-specific limitations.

None.

VCF-NET-REQD-CFG-002

Use VLANs to separate physical network functions.

  • Supports physical network connectivity without requiring many NICs.

  • Isolates the different network functions in the SDDC so that you can have differentiated services and prioritized traffic as needed.

Requires uniform configuration and presentation on all the trunks that are made available to the ESXi hosts.

VCF-NET-REQD-CFG-003

Configure the VLANs as members of a 802.1Q trunk.

All VLANs become available on the same physical network adapters on the ESXi hosts.

Optionally, the management VLAN can act as the native VLAN.

VCF-NET-REQD-CFG-004

Set the MTU size to at least 1,700 bytes (recommended 9,000 bytes for jumbo frames) on the physical switch ports, vSphere Distributed Switches, vSphere Distributed Switch port groups, and N-VDS switches that support the following traffic types:

  • Overlay (Geneve)

  • vSAN

  • vSphere vMotion

  • Improves traffic throughput.

  • Supports Geneve by increasing the MTU size to a minimum of 1,600 bytes.

  • Geneve is an extensible protocol. The MTU size might increase with future capabilities. While 1,600 bytes is sufficient, an MTU size of 1,700 bytes provides more room for increasing the Geneve MTU size without the need to change the MTU size of the physical infrastructure.

When adjusting the MTU packet size, you must also configure the entire network path (VMkernel network adapters, virtual switches, physical switches, and routers) to support the same MTU packet size.

In an environment with multiple availability zones, the MTU must be configured on the entire network path between the zones.
Table 4. Leaf-Spine Physical Network Design Requirements for NSX Federation in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NET-REQD-CFG-005

Set the MTU size to at least 1,500 bytes (1,700 bytes preferred; 9,000 bytes recommended for jumbo frames) on the components of the physical network between the VMware Cloud Foundation instances for the following traffic types.

  • NSX Edge RTEP

  • Jumbo frames are not required between VMware Cloud Foundation instances. However, increased MTU improves traffic throughput.

  • Increasing the RTEP MTU to 1,700 bytes minimizes fragmentation for standard-size workload packets between VMware Cloud Foundation instances.

When adjusting the MTU packet size, you must also configure the entire network path, that is, virtual interfaces, virtual switches, physical switches, and routers to support the same MTU packet size.

VCF-NET-REQD-CFG-006

Ensure that the latency between VMware Cloud Foundation instances that are connected in an NSX Federation is less than 150 ms.

A latency lower than 150 ms is required for the following features:

  • Cross vCenter Server vMotion

  • NSX Federation

None.

VCF-NET-REQD-CFG-007

Provide a routed connection between the NSX Manager clusters in VMware Cloud Foundation instances that are connected in an NSX Federation.

Configuring NSX Federation requires connectivity between the NSX Global Manager instances, NSX Local Manager instances, and NSX Edge clusters.

You must assign unique routable IP addresses for each fault domain.
Table 5. Leaf-Spine Physical Network Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NET-RCMD-CFG-001

Use two ToR switches for each rack.

Supports the use of two 10-GbE (25-GbE or greater recommended) links to each server, provides redundancy and reduces the overall design complexity.

Requires two ToR switches per rack which might increase costs.

VCF-NET-RCMD-CFG-002

Implement the following physical network architecture:

  • One 25-GbE (10-GbE minimum) port on each ToR switch for ESXi host uplinks (Host to ToR).

  • Layer 3 device that supports BGP.

  • Provides availability during a switch failure.

  • Provides support for BGP dynamic routing protocol

  • Might limit the hardware choices.

  • Requires dynamic routing protocol configuration in the physical network.

VCF-NET-RCMD-CFG-003

Use a physical network that is configured for BGP routing adjacency.

  • Supports design flexibility for routing multi-site and multi-tenancy workloads.

  • BGP is the only dynamic routing protocol that is supported for NSX Federation.

  • Supports failover between ECMP Edge uplinks.

Requires BGP configuration in the physical network.

VCF-NET-RCMD-CFG-004

Assign persistent IP configurations for NSX tunnel endpoints (TEPs) that use dynamic IP allocation rather than static IP pool addressing.

  • Ensures that endpoints have a persistent TEP IP address.

  • In VMware Cloud Foundation, TEP IP assignment over DHCP is required for advanced deployment topologies such as a management domain with multiple availability zones.

  • Static pools in NSX Manager support only a single availability zone.

Requires a DHCP server availability on the TEP VLAN.

VCF-NET-RCMD-CFG-005

Set the lease duration for the DHCP scope for the host overlay network to at least 7 days.

The IP addresses of the host overlay VMkernel ports are assigned by using a DHCP server.

  • Host overlay VMkernel ports do not have an administrative endpoint. As a result, they can use DHCP for automatic IP address assignment. If you must change or expand the subnet, changing the DHCP scope is simpler than creating an IP pool and assigning it to the ESXi hosts.

  • DHCP simplifies the configuration of a default gateway for host overlay VMkernel ports if hosts within same cluster are on separate Layer 2 domains.

Requires configuration and management of a DHCP server.

VCF-NET-RCMD-CFG-006

Designate the trunk ports connected to ESXi NICs as trunk PortFast.

Reduces the time to transition ports over to the forwarding state.

Although this design does not use the STP, switches usually have STP configured by default.

VCF-NET-RCMD-CFG-007

Configure VRRP, HSRP, or another Layer 3 gateway availability method for these networks.

  • Management

  • Edge overlay

Ensures that the VLANs that are stretched between availability zones are connected to a highly- available gateway. Otherwise, a failure in the Layer 3 gateway will cause disruption in the traffic in the SDN setup.

Requires configuration of a high availability technology for the Layer 3 gateways in the data center.
Table 6. Leaf-Spine Physical Network Design Recommendations for Stretched Clusters with VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NET-RCMD-CFG-008

Provide high availability for the DHCP server or DHCP Helper used for VLANs requiring DHCP services that are stretched between availability zones.

Ensures that a failover operation of a single availability zone will not impact DHCP availability.

Requires configuration of highly available DHCP server.
Table 7. Leaf-Spine Physical Network Design Recommendations for NSX Federation in VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NET-RCMD-CFG-009

Provide BGP routing between all VMware Cloud Foundation instances that are connect in an NSX Federation setup.

BGP is the supported routing protocol for NSX Federation.

None.