You can implement the switch fabric at the physical layer by providing Layer 2 or Layer 3 transport services. For a scalable and vendor-neutral data solution, use a Layer 3 transport.

Both layer 2 and layer 3 transport have their own sets of benefits and drawbacks. When deciding on an architecture, consider the following benefits and drawbacks of each transport.

Layer 2 Transport Considerations

A design using Layer 2 transport has these considerations:

  • Top-of-Rack (ToR) switches and upstream Layer 3 devices such as core switches and routers form a switched fabric.

  • The upstream Layer 3 devices terminate each VLAN and provide the default gateway functionality.

  • Uplinks from the ToR switch to the upstream Layer 3 devices are 802.1Q trunks carrying all required VLANs.

    Figure 1. Layer 2 Transport
    Layer 2 Transport
Table 1. Benefits and Drawbacks of a Layer 2 Transport

Characteristic

Description

Benefits

  • More design flexibility.

  • You can span VLANs across racks.

Drawbacks

  • The size of this deployment is limited because the fabric elements share a limited number of VLANs.

  • You might have to rely on a specialized switching fabric product from a single vendor.

Layer 3 Transport Considerations

A design using Layer 3 transport has these considerations:

  • Layer 2 connectivity is limited to the ToR switches.

  • The ToR switch terminates each VLAN and provides the default gateway functionality. That is, it has a Switch Virtual Interface (SVI) for each VLAN.

  • Uplinks from the ToR switch to the upstream layer are routed point-to-point links. You cannot use VLAN trunking on the uplinks.

  • A dynamic routing protocol, such as eBGP, connects the ToR switches and upstream switches. Each ToR switch advertises the prefixes, typically one per VLAN or subnet. In turn, the ToR switch calculates equal-cost paths to the prefixes received from the upstream layer it peers with.

    Figure 2. Layer 3 Transport
    Layer 3 Transport
Table 2. Benefits and Drawbacks of a Layer 3 Transport

Characteristic

Description

Benefits

  • You can select from many Layer 3 capable switch products for the physical switching fabric.

  • You can mix switches from different vendors because of the general interoperability between the implementation of routing protocols.

  • This approach is cost-effective because it uses only the basic functionality of the physical switches.

Drawbacks

  • VLANs are restricted to a single rack. The restriction can affect IP-based storage networks, such as iSCSI and NFS.

Physical Network Interfaces

The ESXi hosts must contain four or more physical NICs of the same speed. Switch ports supporting the ESXi hosts must have the required VLANs tagged

The VMware vSphere Distributed Switch supports several NIC teaming options. Load-based NIC teaming supports the optimal use of available bandwidth and redundancy in case of a link failure. Use a minimum of four physical network interfaces for compute and two for management and edge clusters. 25 GbE or greater physical network interfaces are recommended in combination with a pair of ToR switches. 802.1Q network trunks can support as many VLANs as required. Link aggregation, such as LACP, must not be used between the ESXi hosts and the physical switches.