The routing design considers different levels of routing within the environment from which to define a set of principles for designing a scalable routing solution.

North-south

The Provider Logical Router (PLR) handles the North-South traffic to and from a tenant and management applications inside of application virtual networks.

East-west

Internal East-West routing at the layer beneath the PLR deals with the application workloads.

Table 1. Design Decisions on the Routing Model of NSX

Decision ID

Design Decision

Design Justification

Design Implications

SDDC-VI-SDN-017

Deploy a minimum of two NSX Edge services gateways (ESGs) in an ECMP configuration for North-South routing in both management and shared edge and compute clusters.

  • You use an NSX ESG for directing North-South traffic. Using ECMP provides multiple paths in and out of the SDDC.

  • Failover is faster than deploying ESGs in HA mode.

ECMP requires 2 VLANS in each availability zone and region for uplinks which adds an extra VLAN over traditional HA ESG configurations.

When using two availability zones, deploy a minimum of two NSX Edge services gateways in an ECMP configuration in each availability zone.

Because the upstream physical Layer 3 devices reside in a single availability zone, you must deploy ECMP edge devices in each availability zone for North-South routing.

SDDC-VI-SDN-018

Deploy a single NSX UDLR for the management cluster to provide East-West routing across all regions.

Using the UDLR reduces the hop count between nodes attached to it to 1. This reduces latency and improves performance.

UDLRs are limited to 1,000 logical interfaces. If that limit is reached, you must deploy a new UDLR.

SDDC-VI-SDN-019

Deploy a single NSX UDLR for the shared edge and compute, and compute clusters to provide East-West routing across all regions for workloads that require mobility across regions.

Using the UDLR reduces the hop count between nodes attached to it to 1. This reduces latency and improves performance.

UDLRs are limited to 1,000 logical interfaces. If that limit is reached , you must deploy a new UDLR.

SDDC-VI-SDN-020

Deploy a single DLR for the shared edge and compute and compute clusters to provide East-West routing for workloads that require on demand network objects from vRealize Automation.

Using the DLR reduces the hop count between nodes attached to it to 1. This reduces latency and improves performance.

DLRs are limited to 1,000 logical interfaces. If that limit is reached, you must deploy a new DLR.

SDDC-VI-SDN-021

Deploy all NSX UDLRs without the local egress option enabled.

When local egress is enabled, control of ingress traffic is also necessary, for example, using NAT). This configuration is hard to manage for little benefit.

All North-South traffic is routed through Region A until those routes are no longer available. At that time, all traffic dynamically changes to Region B.

SDDC-VI-SDN-022

Use BGP as the dynamic routing protocol inside the SDDC.

Using BGP as opposed to OSPF eases the implementation of dynamic routing. There is no need to plan and design access to OSPF area 0 inside the SDDC. OSPF area 0 varies based on customer configuration.

BGP requires configuring each ESG and UDLR with the remote router that it exchanges routes with.

SDDC-VI-SDN-023

Configure BGP Keep Alive Timer to 1 and Hold Down Timer to 3 between the UDLR and all ESGs that provide North-South routing.

With Keep Alive and Hold Timers between the UDLR and ECMP ESGs set low, a failure is detected quicker, and the routing table is updated faster.

If an ESXi host becomes resource constrained, the ESG running on that ESXi host might no longer be used even though it is still up.

SDDC-VI-SDN-024

Configure BGP Keep Alive Timer to 4 and Hold Down Timer to 12 between the ToR switches and all ESGs providing North-South routing.

This provides a good balance between failure detection between the ToR switches and the ESGs and overburdening the ToRs with keep alive traffic.

By using longer timers to detect when a router is dead, a dead router stays in the routing table longer and continues to send traffic to a dead router.

SDDC-VI-SDN-025

Create one or more static routes on ECMP enabled edges for subnets behind the UDLR and DLR with a higher admin cost than the dynamically learned routes.

When the UDLR or DLR control VM fails over router adjacency is lost and routes from upstream devices such as ToR switches to subnets behind the UDLR are lost.

This requires each ECMP edge device be configured with static routes to the UDLR or DLR. If any new subnets are added behind the UDLR or DLR the routes must be updated on the ECMP edges.

SDDC-VI-SDN-026

Disable Graceful Restart on all ECMP Edges and Logical Router Control Virtual Machines.

Graceful Restart maintains the forwarding table which in turn will forward packets to a down neighbor even after the BGP timers have expired causing loss of traffic.

None.

SDDC-VI-SDN-027

In the management and shared edge and compute clusters, do not create anti-affinity rules to separate ECMP edges and Logical Router control virtual machines.

  • Because these clusters contain four hosts, creating an anti-affinity rule that contains four virtual machines results in not being able to enter maintenance mode to perform life cycle activities.

  • During a host failure, vSphere HA cannot restart the virtual machine because of the anti-affinity rule.

If the active Logical Router control virtual machine and an ECMP edge reside on the same host and that host fails, a dead path in the routing table appears until the standby Logical Router control virtual machine starts its routing process and updates the routing tables.

To avoid this situation, add an additional host to the cluster and create an anti-affinity rule to keep these virtual machines separated.

Transit Network and Dynamic Routing

Dedicated networks are needed to facilitate traffic between the universal dynamic routers and edge gateways, and to facilitate traffic between edge gateways and the top of rack switches. These networks are used for exchanging routing tables and for carrying transit traffic.

Table 2. Design Decisions on the Transit Network

Decision ID

Design Decision

Design Justification

Design Implications

SDDC-VI-SDN-028

Create a universal virtual switch for use as the transit network between the UDLR and ESGs.

The universal virtual switch allows the UDLR and all ESGs across regions to exchange routing information. The UDLR provides East-West routing in both compute and management stacks while the ESGs provide North-South routing.

Only the primary NSX Manager can create and manage universal objects including this UDLR.

SDDC-VI-SDN-029

Create a global virtual switch in each region for use as the transit network between the DLR and ESGs.

The global virtual switch allows the DLR and ESGs in each region to exchange routing information. The DLR provides East-West routing in the compute stack while the ESGs provide North-South routing.

A global virtual switch for use as a transit network is required in each region.

SDDC-VI-SDN-030

Create two VLANs in each availability zone. Use those VLANs to enable ECMP between the North-South ESGs and the L3 device (ToR or upstream device).

The ToR switches or upstream Layer 3 devices have an SVI on one of the two VLANS and each North-South ESG has an interface on each VLAN.

This enables the ESGs to have multiple equal-cost routes and provides more resiliency and better bandwidth use in the network.

Extra VLANs are required.