You can implement the physical layer switch fabric for a SDDC by offering Layer 2 transport services or Layer 3 transport services to all components. For a scalable and vendor-neutral data center network, use Layer 3 transport.
Benefits and Drawbacks for Layer 2 Transport
In a design that uses Layer 2 transport, leaf switches and spine switches form a switched fabric, effectively acting like one large switch. Using modern data center switching fabric products such as Cisco FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol (STP). Such networks are particularly suitable for large virtualization deployments, private clouds, and high-performance computing (HPC) environments.
Using Layer 2 routing has the following benefits and drawbacks:
The benefit of this approach is more design freedom. You can span VLANs, which is useful for vSphere vMotion or vSphere Fault Tolerance (FT).
The drawback is that the size of such a deployment is limited because the fabric elements have to share a limited number of VLANs. In addition, you have to rely on a specialized data center switching fabric product from a single vendor because these products are not designed for interoperability between vendors.
Benefits and Drawbacks for Layer 3 Transport
A design using Layer 3 transport requires these considerations:
Layer 2 connectivity is limited within the data center rack up to the leaf switch.
The leaf switch terminates each VLAN and provides default gateway functionality. That is, it has a switch virtual interface (SVI) for each VLAN.
Uplinks from the leaf switch to the spine layer are routed point-to-point links. VLAN trunking on the uplinks is not allowed.
A dynamic routing protocol, such as OSPF, ISIS, or BGP, connects the leaf switches and spine switches. Each leaf switch in the rack advertises a small set of prefixes, typically one per VLAN or subnet. In turn, the leaf switch calculates equal cost paths to the prefixes it received from other leaf switches.
Using Layer 3 routing has the following benefits and drawbacks:
The benefit is that you can chose from a wide array of Layer 3 capable switch products for the physical switching fabric. You can mix switches from different vendors due to general interoperability between implementation of OSPF, ISIS or BGP. This approach is typically more cost effective because it makes use of only the basic functionality of the physical switches.
A design restriction, and thereby a drawback of using Layer 3 routing, is that VLANs are restricted to a single rack. This affects vSphere vMotion, vSphere Fault Tolerance, and storage networks.