You can implement the switch fabric at the physical layer by providing Layer 2 or Layer 3 transport services. For a scalable and vendor-neutral data solution, use Layer 3 transport.

Both Layer 2 and Layer 3 transport have their own sets of benefits and drawbacks. When deciding on an architecture, consider the following benefits and drawbacks of each transport.

Layer 2 Transport Considerations

The following considerations apply for a design that uses Layer 2 transport:

  • Top-of-Rack (ToR) switches and upstream Layer 3 devices such as core switches and routers form a switched fabric.

  • The upstream Layer 3 devices terminate each VLAN and provide the default gateway functionality.

  • Uplinks from the ToR switch to the upstream Layer 3 devices are 802.1Q trunks carrying all required VLANs.

    Figure 1. Layer 2 Transport
    Layer 2 Transport
Table 1. Benefits and Drawbacks of a Layer 2 Transport

Characteristic

Description

Benefits

  • Additional design flexibility

  • You can span VLANs across racks.

Drawbacks

  • The size of this deployment is limited because the fabric elements share a limited number of VLANs.

  • You might have to rely on a specialized switching fabric product from a single vendor.

Layer 3 Transport Considerations

The following considerations apply for a design using Layer 3 transport:

  • Layer 2 connectivity is limited to the ToR switches.

  • The ToR switch terminates each VLAN and provides the default gateway functionality. That is, it has a Switch Virtual Interface (SVI) for each VLAN.

  • Uplinks from the ToR switch to the upstream layer are routed point-to-point links. You cannot use VLAN trunking on the uplinks.

  • A dynamic routing protocol, such as eBGP, connects the ToR switches and upstream switches. Each ToR switch advertises the prefixes, typically one per VLAN or subnet. In turn, the ToR switch calculates equal-cost paths to the prefixes received from the upstream layer it peers with.

    Figure 2. Layer 3 Transport
    Layer 3 Transport
Table 2. Benefits and Drawbacks of a Layer 3 Transport

Characteristic

Description

Benefits

  • You can select from many Layer 3 capable switch products for the physical switching fabric.

  • You can mix switches from different vendors because of the general interoperability between the implementation of routing protocols.

  • This approach is cost-effective because it uses only the basic functionality of the physical switches.

Drawbacks

  • VLANs are restricted to a single rack. The restriction can affect IP-based storage networks, such as iSCSI and NFS.

Physical Network Interfaces

The Telco Cloud Infrastructure requires that the ESXi hosts contain four or more physical NICs of the same speed. Switch ports supporting the ESXi hosts must have the required VLANs tagged.

The VMware vSphere Distributed Switch supports several NIC teaming options. Load-based NIC teaming ensures the optimal use of available bandwidth and redundancy in case of a link failure. Use a minimum of four physical network interfaces for compute and two for management and edge clusters. Use physical network interfaces with at least 25 GbE throughput in combination with a pair of ToR switches. Depending on the switch vendor, 802.1Q network trunks can support a maximum of 4095 or 4096 VLANs. Link aggregation, such as LACP, must not be used between the ESXi hosts and the physical switches.