Design of the physical data center network includes defining the network topology for connecting the physical switches and the ESXi hosts, determining switch port settings for VLANs and link aggregation, and designing routing.

A software-defined network (SDN) both integrates with and uses components of the physical data center. SDN integrates with your physical network to support east-west transit in the data center and north-south transit to and from the SDDC networks.

VMware Validated Design uses the leaf-spine networking topology, because in a single data center deployment, it provides predictable performance, scalable nature, and applicability across multiple vendors.

Switch Types and Network Connectivity

Follow the best practices for physical switches, switch connectivity, VLANs and subnets, and access port settings.

Figure 1. Host-to-ToR Connectivity


Table 1. Design Components for Physical Switches in the SDDC

Design Component

Configuration Best Practices

Top of rack (ToR) physical switches

  • Configure redundant physical switches to enhance availability.

  • Configure switch ports that connect to ESXi hosts manually as trunk ports.

  • Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce the time to transition ports over to the forwarding state, for example using the Trunk PortFast feature found in a Cisco physical switch.

  • Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports. This setup simplifies the configuration by using DHCP to assign IP address based on the IP subnet in use.

  • Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual interfaces (SVIs).

Top of rack connectivity and network settings

Each ESXi host is connected redundantly to the top of rack switches SDDC network fabric by two 25 GbE ports. Configure the top of rack switches to provide all necessary VLANs using an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and NSX-T to guarantee that no physical interface is overrun, and available redundant paths are used.

VLANs and Subnets for Single Region and Single Availability Zone

Each ESXi host uses VLANs and corresponding subnets.

Follow these guidelines:

  • Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.

  • Use the IP address .254 as the (floating) interface with .252 and .253 for Virtual Router Redundancy Protocol (VRPP) or Hot Standby Routing Protocol (HSRP).

  • Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region and another octet by function.

Note:

Implement VLAN and IP subnet configuration according to the requirements of your organization.

Table 2. VLANs and IP Ranges in This Design

Function

VLAN ID

IP Range

Management

1631

172.16.31.0/24

vSphere vMotion

1632

172.16.32.0/24

vSAN

1633

172.16.33.0/24

Host Overlay

1634

172.16.34.0/24

NFS

1635

172.16.35.0/24

Uplink01

2731

172.27.31.0/24

Uplink02

2732

172.27.32.0/24

Edge Overlay

2733

172.27.33.0/24

Physical Network Requirements

Physical requirements determine the MTU size for networks that carry overlay traffic, dynamic routing support, time synchronization through an NTP server, and forward and reverse DNS resolution.

Requirement

Comment

Use 25 GbE (10 GbE minimum) port on each ToR switch for ESXi host uplinks. Connect each host to two ToR switches.

25 GbE provides appropriate bandwidth for hyperconverged networking traffic. Connection to two ToR switches provides redundant physical network paths to each host.

Provide an MTU size of 1700 bytes or greater for host overlay traffic.

Geneve packets cannot be fragmented. The MTU size must be large enough to support extra encapsulation overhead.

Geneve is an extensible protocol, therefore the MTU might increase with future capabilities. While 1600 is sufficient, an MTU size of 1700 bytes provides more room for increasing the Geneve MTU without the need to change physical infrastructure MTU.

This design uses an MTU size of 9000 bytes for Geneve traffic.

Enable BGP dynamic routing support on the upstream Layer 3 devices.

You use BGP on the upstream Layer 3 devices to establish routing adjacency with the Tier-0 service routers (SRs). NSX-T supports only the BGP routing protocol.

Dynamic routing enables ECMP failover for upstream connectivity.

BGP Autonomous System Number (ASN) allocation

A BGP ASN must be allocated for the NSX-T SDN. Use a private ASN according to RFC1930.

Physical Network Design Decisions

The physical network design decisions determine the physical layout and use of VLANs. They also include decisions on jumbo frames and on other network-related requirements such as DNS and NTP.

Table 3. Design Decisions on the Physical Network Infrastructure for NSX-T Data Center in a Workload Domain

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-KUBWLD-VI-SDN-001

Use two ToR switches for each rack.

Supports the use of two 10 GbE (25 GbE or greater recommended) links to each server and provides redundancy and reduces the overall design complexity.

Requires two ToR switches per rack which can increase costs.

SDDC-KUBWLD-VI-SDN-002

Implement the following physical network architecture:

  • One 25 GbE (10 GbE minimum) port on each ToR switch for ESXi host uplinks.

  • Layer 3 device that supports BGP.

  • Guarantees availability during a switch failure.

  • Provides support for BGP as the only dynamic routing protocol that is supported by NSX-T Data Center.

  • Might limit the hardware choices.

  • Requires dynamic routing protocol configuration in the physical network

SDDC-KUBWLD-VI-SDN-003

Do not use EtherChannel (LAG, LACP, or vPC) configuration for ESXi host uplinks

  • Simplifies configuration of top of rack switches.

  • Teaming options available with vSphere Distributed Switch and N-VDS provide load balancing and failover.

  • EtherChannel implementations might have vendor-specific limitations.

None.

SDDC-KUBWLD-VI-SDN-004

Use a physical network that is configured for BGP routing adjacency

  • Supports flexibility in network design for routing multi-site and multi-tenancy workloads.

  • Uses BGP as the only dynamic routing protocol that is supported by NSX-T.

  • Supports failover between ECMP Edge uplinks.

Requires BGP configuration in the physical network.

Access Port Network Settings

Configure additional network settings on the access ports that connect the ToR switches to the corresponding servers.

Table 4. Access Port Network Configuration

Setting

Value

Spanning Tree Protocol (STP)

Although this design does not use the Spanning Tree Protocol, switches usually include STP configured by default. Designate the access ports as trunk PortFast.

Trunking

Configure the VLANs as members of an 802.1Q trunk. Optionally, the management VLAN can act as the native VLAN.

MTU

  • Set MTU for management VLANs and SVIs to 1500 bytes.
  • Set MTU for vMotion, vSAN, NFS, Uplinks, Host Overlay, and Edge Overlay VLANs and SVIs to 9000 bytes.

DHCP Helper

Configure a DHCP helper (sometimes called a DHCP relay) on all Overlay VLANs. Set the DHCP helper (relay) to point to a DHCP server by IPv4 address.

Table 5. Design Decisions on Access Ports for NSX-T Data Center in a Workload Domain

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-KUBWLD-VI-SDN-005

Assign persistent IP configurations to each management component in the SDDC with the exception for NSX-T tunnel endpoints (TEPs) that use dynamic IP allocation.

Ensures that endpoints have a persistent management IP address. In VMware Cloud Foundation, you assign storage (vSAN and NFS) and vSphere vMotion IP configurations by using user-defined network pools.

Requires precise IP address management.

SDDC-KUBWLD-VI-SDN-006

Set the lease duration for the NSX-T Host Overlay network DHCP scope to at least 7 days.

NSX-T Host Overlay VMkernel port IP addresses are assigned by using a DHCP server.

  • NSX-T Host Overlay VMkernel ports do not have an administrative endpoint. As a result, they can use DHCP for automatic IP address assignment. IP pools are an option, but the NSX-T administrator must create them. If you must change or expand the subnet, changing the DHCP scope is simpler than creating an IP pool and assigning it to the ESXi hosts.

  • DHCP simplifies the configuration of default gateway for Host Overlay VMkernel ports if hosts within same cluster are on separate Layer 2 domains.

Requires configuration and management of a DHCP server.

SDDC-KUBWLD-VI-SDN-007

Use VLANs to separate physical network functions.

  • Supports physical network connectivity without requiring many NICs.

  • Isolates the different network functions of the SDDC so that you can have differentiated services and prioritized traffic as needed.

Requires uniform configuration and presentation on all the trunks that are made available to the ESXi hosts.

Jumbo Frames

IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-frame payload from 1500 bytes to the jumbo frame setting improves the efficiency of data transfer. You must configure jumbo frames end-to-end. Select an MTU that matches the MTU of the physical switch ports.

  • According to the purpose of the workload, determine whether to configure jumbo frames on a virtual machine. If the workload consistently transfers large amounts of network data, configure jumbo frames, if possible. In that case, confirm that both the virtual machine operating system and the virtual machine NICs support jumbo frames.

  • Using jumbo frames also improves the performance of vSphere vMotion.

  • The Geneve overlay requires an MTU value of 1700 bytes or greater.

Table 6. Design Decisions on the Jumbo Frames for NSX-T Data Center for the Workload Domain

Decision ID

Design Decision

Decision Justification

Decision Implication

SDDC-KUBWLD-VI-SDN-008

Set the MTU size to at least 1700 bytes (recommended 9000 bytes for jumbo frames) on the physical switch ports, vSphere Distributed Switches, vSphere Distributed Switch port groups, and N-VDS switches that support the following traffic types:

  • Geneve (overlay)

  • vSAN

  • vSphere vMotion

  • NFS

  • Improves traffic throughput.

  • Supports Geneve by increasing the MTU size to a minimum of 1600 bytes.

  • Geneve is an extensible protocol. The MTU size might increase with future capabilities. While 1600 is sufficient, an MTU size of 1700 bytes provides more room for increasing the Geneve MTU size without the need to change the MTU size of the physical infrastructure.

When adjusting the MTU size, you must also configure the entire network path (VMkernel ports, virtual switches, physical switches, and routers) to support the same MTU size.