Suppose two ESGs are deployed to provide a DLR instance with 2-way ECMP uplinks with the physical environment.

High-Level ESG and DLR Packet Flow with ECMP shows the ESG and DLR packet flow when equal-cost multipath (ECMP) routing is enabled between two ESGs and the physical infrastructure.

VM1 thus has access to 2x bi-directional throughput compared with a deployment with a single ESG.

VM1 is connected to a Logical Switch with the VNI 5000.

The DLR has two LIFs – Internal on VNI 5000, and Uplink on VNI 5001.

The DLR has ECMP enabled and is receiving equal cost routes toward the IP subnet of VLAN 20 from a pair of ESGs, ESG A and ESG B via a dynamic routing protocol (BGP or OSPF).

The two ESGs are connected to a VLAN-backed dvPortgroup associated with VLAN 10, where a physical router that provides connectivity to VLAN 20 is also connected.

The ESGs receive external routes for VLAN 20, via a dynamic routing protocol from the physical router.

The physical router in exchange learns about the IP subnet associated with VXLAN 5000 from both ESGs, and performs ECMP load balancing for the traffic toward VMs in that subnet.

Figure 1. High-Level ESG and DLR Packet Flow with ECMP
The image is described in the surrounding text.

The DLR can receive up to eight equal-cost routes and balance traffic across the routes. ESG A and ESG B in the diagram provide two equal-cost routes.

ESGs can do ECMP routing toward the physical network, assuming multiple physical routers are present. For simplicity, the diagram shows a single physical router.

There is no need for ECMP to be configured on ESGs toward the DLR, because all DLR LIFs are “local” on the same host where ESG resides. There would be no additional benefit provided by configuring multiple uplink interfaces on a DLR.

In situations where more North-South bandwidth is required, multiple ESGs can be placed on different ESXi hosts to scale up to ~80Gbps with 8 x ESGs.

The ECMP packet flow (not including ARP resolution):
  1. VM1 sends a packet to the physical server, which is sent to VM1’s IP gateway (which is a DLR LIF) on ESXi Host A.
  2. The DLR performs a route lookup for the IP of the physical server, and finds that it is not directly connected, but matches two ECMP routes received from ESG A and ESG B.
  3. The DLR calculates an ECMP hash, and decides on a next hop, which could be either ESG A or ESG B, and sends the packet out the VXLAN 5001 LIF.
  4. The DVS delivers the packet to the selected ESG.
  5. The ESG performs the routing lookup and finds that the physical server’s subnet is accessible via the physical router’s IP address on VLAN 10, which is directly connected to one of ESG’s interfaces.
  6. The packet is sent out through the DVS, which passes it on to the physical network after tagging it with the correct 801.Q tag with VLAN ID 10.
  7. The packet travels through the physical switching infrastructure to reach the physical router, which performs a lookup to find that the physical server is directly connected to an interface on VLAN 20.
  8. The physical router sends the packet to the physical server.
On the way back:
  1. The physical server sends the packet to VM1, with the physical router as the next hop.
  2. The physical router performs a lookup for VM1’s subnet, and sees two equal-cost paths to that subnet with the next hops, ESG A's and ESG B’s VLAN 10 interface, respectively.
  3. The physical router selects one of the paths and sends the packet toward the corresponding ESG.
  4. The physical network delivers the packet to the ESXi host where the ESG resides, and delivers it to DVS, which decapsulates the packet and forwards it on the dvPortgroup associated with VLAN 10 to the ESG.
  5. The ESG performs a routing lookup and finds that VM1’s subnet is accessible via its interface associated with VXLAN 5001 with the next hop being DLR’s uplink interface IP address.
  6. The ESG sends the packet to the DLR instance on the same host as the ESG.
  7. The DLR performs a routing lookup to find that VM1 is available via its VXLAN 5000 LIF.
  8. The DLR sends the packet out its VXLAN 5000 LIF to the DVS, which performs the final delivery.