Follow best practices for physical switches, switch connectivity, VLANs and subnets, and access port settings.
Top of Rack Physical Switches
When configuring top of rack (ToR) switches, consider the following best practices:
Configure redundant physical switches to enhance availability.
Configure switch ports that connect to ESXi hosts manually as trunk ports. Virtual switches are passive devices and do not support trunking protocols, such as Dynamic Trunking Protocol (DTP).
Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce the time to transition ports over to the forwarding state, for example using the Trunk PortFast feature found in a Cisco physical switch.
Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports. This setup simplifies the configuration by using DHCP to assign IP address based on the IP subnet in use.
Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual interfaces (SVIs).
Top of Rack Connectivity and Network Settings
Each ESXi host is connected redundantly to the ToR switches SDDC network fabric by two 10 GbE ports. Configure the ToR switches to provide all necessary VLANs via an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and NSX-T to guarantee that no physical interface is overrun and available redundant paths are used.
VLANs and Subnets
Each ESXi host uses VLANs and corresponding subnets.
Follow these guidelines:
Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.
Use the IP address .253 as the (floating) interface with .251 and .252 for Virtual Router Redundancy Protocol (VRPP) or Hot Standby Routing Protocol (HSRP).
Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region and another octet by function.
Access Port Network Settings
Configure additional network settings on the access ports that connect the ToR switches to the corresponding servers.
Spanning Tree Protocol (STP)
Although this design does not use the Spanning Tree Protocol, switches usually include STP configured by default. Designate the access ports as trunk PortFast.
Configure the VLANs as members of a 802.1Q trunk with the management VLAN acting as the native VLAN.
Set MTU for all VLANS and SVIs (Management, vMotion, VXLAN, and Storage) to jumbo frames for consistency purposes.
Configure a DHCP helper (sometimes called a DHCP relay) on all TEP VLANs.