You can implement the physical layer switch fabric for a SDDC by offering Layer 2 transport services or Layer 3 transport services. For a scalable and vendor-neutral data center network, use a Layer 3 transport.

A VMware Validated Design supports both Layer 2 and Layer 3 transports. When deciding to use Layer 2 or Layer 3 keep the following in mind:

  • NSX ECMP Edge devices establish layer 3 routing adjacency with the first upstream Layer 3 device to provide equal cost routing for management and workload virtual machine traffic.

  • The investment you have today in your current physical network infrastructure.

  • The following benefits and drawbacks for both Layer 2 and Layer 3 designs.

Benefits and Drawbacks for Layer 2 Transport

A design using Layer 2 transport requires these considerations:

  • In a design that uses Layer 2 transport, top-of-rack switches and upstream Layer 3 devices, such as core switches or routers, form a switched fabric.

  • The upstream Layer 3 devices terminate each VLAN and provide default gateway functionality.

  • Uplinks from the top-of-rack switch to the upstream Layer 3 devices are 802.1Q trunks carrying all required VLANs.

Using a Layer 2 transport has the following benefits and drawbacks:

  • The benefit of Layer 2 is more design freedom. You can span VLANs, which can be useful in some circumstances.

  • The drawback is that the size of such a deployment is limited because the fabric elements have to share a limited number of VLANs. In addition, you may have to rely on a specialized data center switching fabric product from a single vendor.

Figure 1. Example Layer 2 Transport

Example of Layer 2 Transport

Benefits and Drawbacks for Layer 3 Transport

A design using Layer 3 transport requires these considerations for setup:

  • Layer 2 connectivity is limited within the data center rack up to the top-of-rack switches.

  • The top-of-rack switch terminates each VLAN and provides default gateway functionality. That is, the top-of-rack switch has a switch virtual interface (SVI) for each VLAN.

  • Uplinks from the top-of-rack switch to the upstream layer are routed point-to-point links. VLAN trunking on the uplinks is not allowed.

  • A dynamic routing protocol, such as OSPF, IS-IS, or BGP, connects the top-of-rack switches and upstream switches. Each top-of-rack switch in the rack advertises a small set of prefixes, typically one per VLAN or subnet. In turn, the top-of-rack switch calculates equal cost paths to the prefixes it receives from other top-of-rack switches.

Using Layer 3 routing has the following benefits and drawbacks:

  • The benefit of Layer 3 is that you can chose from a wide array of Layer 3 capable switch products for the physical switching fabric. You can mix switches from different vendors due to general interoperability between implementation of OSPF, IS-IS or BGP. This approach is typically more cost effective because it makes use of only the basic functionality of the physical switches.

  • A design restriction, and thereby a drawback of using Layer 3 routing, is that VLANs are restricted to a single rack. This can affect, vSphere Fault Tolerance, and storage networks. This limitation can be overcome by the use of Layer 2 bridging in NSX.

Figure 2. Example Layer 3 Transport

Example of Layer 3 Transport