You can implement the physical layer switch fabric for a SDDC by offering Layer 2 transport services or Layer 3 transport services. For a scalable and vendor-neutral data center network, use a Layer 3 transport.
A VMware Validated Design supports both Layer 2 and Layer 3 transports. When deciding to use Layer 2 or Layer 3 keep the following in mind:
NSX ECMP Edge devices establish layer 3 routing adjacency with the first upstream Layer 3 device to provide equal cost routing for management and workload virtual machine traffic.
The investment you have today in your current physical network infrastructure.
The following benefits and drawbacks for both Layer 2 and Layer 3 designs.
Benefits and Drawbacks for Layer 2 Transport
A design using Layer 2 transport requires these considerations:
In a design that uses Layer 2 transport, top-of-rack switches and upstream Layer 3 devices, such as core switches or routers, form a switched fabric.
The upstream Layer 3 devices terminate each VLAN and provide default gateway functionality.
Uplinks from the top-of-rack switch to the upstream Layer 3 devices are 802.1Q trunks carrying all required VLANs.
Using a Layer 2 transport has the following benefits and drawbacks:
The benefit of Layer 2 is more design freedom. You can span VLANs, which can be useful in some circumstances.
The drawback is that the size of such a deployment is limited because the fabric elements have to share a limited number of VLANs. In addition, you may have to rely on a specialized data center switching fabric product from a single vendor.
Benefits and Drawbacks for Layer 3 Transport
A design using Layer 3 transport requires these considerations for setup:
Layer 2 connectivity is limited within the data center rack up to the top-of-rack switches.
The top-of-rack switch terminates each VLAN and provides default gateway functionality. That is, the top-of-rack switch has a switch virtual interface (SVI) for each VLAN.
Uplinks from the top-of-rack switch to the upstream layer are routed point-to-point links. VLAN trunking on the uplinks is not allowed.
A dynamic routing protocol, such as OSPF, IS-IS, or BGP, connects the top-of-rack switches and upstream switches. Each top-of-rack switch in the rack advertises a small set of prefixes, typically one per VLAN or subnet. In turn, the top-of-rack switch calculates equal cost paths to the prefixes it receives from other top-of-rack switches.
Using Layer 3 routing has the following benefits and drawbacks:
The benefit of Layer 3 is that you can chose from a wide array of Layer 3 capable switch products for the physical switching fabric. You can mix switches from different vendors due to general interoperability between implementation of OSPF, IS-IS or BGP. This approach is typically more cost effective because it makes use of only the basic functionality of the physical switches.
A design restriction, and thereby a drawback of using Layer 3 routing, is that VLANs are restricted to a single rack. This can affect, vSphere Fault Tolerance, and storage networks. This limitation can be overcome by the use of Layer 2 bridging in NSX.