Network Architecture

You can implement the switch fabric at the physical layer by providing Layer 2 or Layer 3 transport services. For a scalable and vendor-neutral data solution, use a Layer 3 transport.

Both layer 2 and layer 3 transport have their own sets of benefits and drawbacks. When deciding on an architecture, consider the following benefits and drawbacks of each transport.

Layer 2 Transport Considerations

A design using Layer 2 transport has these considerations:

• Top-of-Rack (ToR) switches and upstream Layer 3 devices such as core switches and routers form a switched fabric.

• The upstream Layer 3 devices terminate each VLAN and provide the default gateway functionality.

• Uplinks from the ToR switch to the upstream Layer 3 devices are 802.1Q trunks carrying all required VLANs.

Figure 1. Layer 2 Transport

Table 1. Benefits and Drawbacks of a Layer 2 Transport

Characteristic	Description
Benefits	More design flexibility. You can span VLANs across racks.
Drawbacks	The size of this deployment is limited because the fabric elements share a limited number of VLANs. You might have to rely on a specialized switching fabric product from a single vendor. Typically requires some implementation to prevent L2 loops

Layer 3 Transport Considerations

A design using Layer 3 transport has these considerations:

• Layer 2 connectivity is limited to the ToR switches.

• The ToR switch terminates each VLAN and provides the default gateway functionality. That is, it has a Switch Virtual Interface (SVI) for each VLAN.

• Uplinks from the ToR switch to the upstream layer are routed point-to-point links. You cannot use VLAN trunking on the uplinks.

• A dynamic routing protocol such as eBGP connects the ToR switches and upstream switches. Each ToR switch advertises the prefixes, typically one per VLAN or subnet. In turn, the ToR switch calculates equal-cost paths to the prefixes received from the upstream layer it peers with.

Figure 2. Layer 3 Transport

Table 2. Benefits and Drawbacks of a Layer 3 Transport

Characteristic	Description
Benefits	You can select from many Layer 3 capable switch products for the physical switching fabric. You can mix switches from different vendors because of the general interoperability between the implementation of routing protocols. This approach is cost-effective because it uses only the basic functionality of the physical switches.
Drawbacks	VLANs are restricted to a single rack. The restriction can affect IP-based storage networks, such as iSCSI and NFS.

Note:

When leveraging NSX for VLAN or Overlay networking, the compute vSwitch on workload domains must use the converged VDS model. The converged VDS model is supported from vSphere 7.0 onwards.

Physical Network Interfaces

The ESXi hosts must contain four or more physical NICs of the same speed. Use at least two physical NICs as uplinks with VLANs trunked to the interfaces.

VMware vSphere Distributed Switch supports several NIC teaming options. Load-based NIC teaming supports the optimal use of available bandwidth and redundancy in case of a link failure. Use a minimum of 10 GbE connections, with 25 GbE or larger connections recommended, for each ESXi host in combination with a pair of ToR switches. 802.1Q network trunks can support as many VLANs as required.