This section describes best practises for designing the physical network for NSX-T workload domains and Enterprise PKS.

Top of Rack Physical Switches

Consider the following best practises when configuring the top of rack (ToR) switches.
  • Configure redundant physical switches to enhance availability.
  • Configure switch ports that connect to ESXi hosts manually as trunk ports.
  • Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce the time to transition ports over to the forwarding state, for example using the Trunk PortFast feature in Cisco physical switches.
  • Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports. This setup simplifies the configuration by using DHCP to assign IP address based on the IP subnet in use.
  • Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual interfaces (SVIs).

Top of Rack Connectivity and Network Settings

Each ESXi host is connected redundantly to the ToR switches SDDC network fabric by two 25 GbE ports. Configure the ToR switches to provide all necessary VLANs using an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and NSX-T to guarantee that no physical interface is overrun and available redundant paths are used.

VLANs and Subnets

Each ESXi host uses VLANs and corresponding subnets. Follow these guidelines for VLANs and subnets.
  • Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.
  • Use the IP address .254 as the (floating) interface with .252 and .253 for Virtual Router Redundancy Protocol (VRPP) or Hot Standby Routing Protocol (HSRP).
  • Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region and another octet by function.

Access Port Network Settings

Configure additional network settings listed in the table below on the access ports that connect the ToR switches to the corresponding servers.
Table 1. Additional Network Settings to be Configured on VLANs and Subnets
Setting Description
Spanning Tree Protocol (STP) Although this design does not use the Spanning Tree Protocol, switches usually include STP configured by default. Designate the access ports as trunk PortFast.
Trunking Configure the VLANs as members of a 802.1Q trunk with the management VLAN acting as the native VLAN.
MTU Set MTU for all VLANs and SVIs (Management, vMotion, Geneve, and Storage) to jumbo frames for consistency.
DHCP Helper Configure a DHCP helper (DHCP relay) on all TEP VLANs.

Routing Protocols

NSX-t supports BGP only. Configure BGP per the guidelines below. The Layer 3 device (for example, the ToR switch) must supports BGP.

DHCP

Set the DHCP helper (relay) to point to a DHCP server by IPv4 address.

Physical and Logical Networking

  • Implement the following physical network architecture:
    • One 25 GbE (10 GbE minimum) port on each ToR switch for ESXi host uplinks.
    • No EtherChannel (LAG/LACP/vPC) configuration for ESXi host uplink.
    • Use two ToR switches for each rack for redundancy.
  • Implement the following logical network architecture:

    Use VLANs to segment physical network functions.

Static IP Addresses, DNS records, and NTP time source

  • Use a physical network that is configured for BGP routing adjacency.
  • Use two ToR switches for each rack.
  • Use VLANs to segment physical network functions.

Jumbo Frames

You must configure jumbo frames end-to-end. Select an MTU that matches the MTU of the physical switch ports. Note that the Geneve overlay requires an MTU value of 1600 bytes or greater.