The routing design considers different levels of routing within the environment from which to define a set of principles for designing a scalable routing solution.

North/south

The Provider Logical Router (PLR) handles the north/south traffic to and from a tenant and management applications inside of application virtual networks.

East/west

Internal east/west routing at the layer beneath the PLR deals with the application workloads.

Table 1. Routing Model Design Decisions

Decision ID

Design Decision

Design Justification

Design Implications

CSDDC-VI-SDN-011

Deploy NSX Edge Services Gateways in an ECMP configuration for north/south routing.

The NSX ESG is the recommended device for managing north/south traffic. Using ECMP provides multiple paths in and out of the SDDC. This results in faster failover times than deploying Edge service gateways in HA mode.

ECMP requires 2 VLANS for uplinks which adds an additional VLAN over traditional HA ESG configurations.

CSDDC-VI-SDN-012

Deploy a single NSX UDLR to provide east/west routing.

Using the UDLR reduces the hop count between nodes attached to it to 1. This reduces latency and improves performance.

Using the UDLR allows seamless migration to the two pod validated design.

UDLRs are limited to 1,000 logical interfaces. When that limit is reached, a new UDLR must be deployed.

CSDDC-VI-SDN-013

Deploy all NSX UDLRs without the local egress option enabled.

When local egress is enabled, control of ingress traffic is also necessary (for example using NAT). This becomes hard to manage for little to no benefit.

All north/south traffic is routed through Region A until those routes are no longer available. At that time, all traffic dynamically changes to Region B.

CSDDC-VI-SDN-014

Use BGP as the dynamic routing protocol inside the SDDC.

Using BGP as opposed to OSPF eases the implementation of dynamic routing. There is no need to plan and design access to OSPF area 0 inside the SDDC.

The use of BGP for the SDDC components doesn't prohibit the continued use of another protocol on the physical network.

BGP requires configuring each ESG and UDLR with the remote router that it exchanges routes with.

CSDDC-VI-SDN-015

Configure BGP Keep Alive Timer to 1 and Hold Down Timer to 3 between the UDLR and all ESGs that provide north/south routing.

With Keep Alive and Hold Timers between the UDLR and ECMP ESGs set low, a failure is detected quicker, and the routing table is updated faster.

If an ESXi host becomes resource constrained, the ESG running on that host might no longer be used even though it is still up.

CSDDC-VI-SDN-016

Configure BGP Keep Alive Timer to 4 and Hold Down Timer to 12 between the ESGs and the upstream Layer 3 device providing north/south routing.

This provides a good balance between failure detection between the physical network and the ESGs without overburdening the physical network with keep alive traffic.

By using longer timers to detect when a router is dead, a dead router stays in the routing table longer and continues to send traffic to a dead router.

CSDDC-VI-SDN-017

Create one or more static routes on ECMP enabled edges for subnets behind the UDLR with a higher admin cost then the dynamically learned routes.

When the UDLR control VM fails over router adjacency is lost and routes from upstream devices to subnets behind the UDLR are lost.

This requires each ECMP edge device be configured with static routes to the UDLR. If any new subnets are added behind the UDLR the routes must be updated on the ECMP edges.

Transit Network and Dynamic Routing

Dedicated networks are needed to facilitate traffic between the universal distributed logical routers and edge gateways, and to facilitate traffic between edge gateways and the upstream layer 3 devices. These networks are used for exchanging routing tables and for carrying transit traffic.

Table 2. Transit Network Design Decisions

Decision ID

Design Decision

Design Justification

Design Implications

CSDDC-VI-SDN-018

Create a universal virtual switch for use as the transit network between the UDLR and ESGs. The UDLR provides east/west routing while the ESG's provide north/south routing.

The universal virtual switch allows the UDLR and all ESGs to exchange routing information.

Using a universal virtual switch allows seamless migration to the two pod validated design.

Only the primary NSX Manager can create and manage universal objects.

CSDDC-VI-SDN-019

Create two VLANs to enable ECMP between the north/south ESGs and the upstream layer 3 devices.

The upstream layer 3 devices have an SVI on one of the two VLANS and each north/south ESG has an interface on each VLAN.

This enables the ESGs to have multiple equal-cost routes and provides more resiliency and better bandwidth utilization in the network.

Extra VLANs are required.