VMware HCX Proximity Routing optimizes connectivity for VMs that are migrated in and out of Proximity Routed Networks. Proximity Routing builds on VMware HCX Network Extension by integrating with NSX Routers at the VMware HCX Cloud destination site. By dynamically injecting Virtual Machine routes into the existing routed topology, proximity routed network traffic always traverses a symmetric path to the network target.

VMware HCX-PR allows extended VMs to route traffic optimally through the cloud-side, first-hop gateway. By dynamically injecting VM routes into the routing protocols, incoming traffic from the local and remote data centers use an optimal path to reach the extended Virtual Machine, while ensuring all flows remain symmetric. The Proximity Routing feature is toggled on during the VMware HCX Network Extension operation, but it carries distinct requirements and results in enhanced traffic patterns, which are covered in this section.

Virtual Machine Reachability on Extended Networks without Proximity Routing

  • In the following figure, Network A has been extended without enabling Proximity Routing. Virtual Machine B has been migrated.

  • Reaching Virtual Machine B:

    • Traffic sourcing from Virtual Machine A in data center A traverses Local HCX-NET-EXT and Remote HCX-NET-EXT to reach Virtual Machine B in data center B, and conversely for Virtual Machine B to Virtual Machine A.

    • Traffic sourcing from Network B, Network C, or any other network, must first travel to the Network A gateway in data center A. It then traverses Local HCX-NET-EXT and Remote HCX-NET-EXT to reach VM-B in data center B.

    • Traffic sourcing from VM-B to Network B, Network C, or any other network, must first travel to the Network A gateway to be routed to its destination.

Figure 1. Network A Is Extended from Data Center A to Data Center B

VMware HCX Proximity Routing In Action

The following events take place when VMware HCX-PR is used, with VMware HCX Migrations.

Prerequisites

Requirements and Restrictions for HCX Proximity Routing

  • Dynamic Routing

    • The HCX Enterprise site (on-premises data center) must be configured to learn routes from the VMware HCX target site dynamically. A routing protocol like BGP or OSPF must be configured between the two sites.

    • The VMware HCX-enabled Cloud target site must be running NSX 6.3GA+ (allows host routes to learn from the dynamic routing protocol).

  • Private Lines/Direct Link/ Direct Connect Networks 

    • Private lines are typically implemented as the transport for the Dynamic Routing configuration (if there is OSPF, to meet security requirements, and adjacency requirements).

  • VMware HCX-Cloud Providers and Proximity Routing

    • VMware HCX is available for IBM Clouds, OVH Private Cloud, and the VMware Cloud on AWS (VMC). The IBM and OVH cloud services can be configured to meet all VMware HCX-PR requirements. If the Cloud is configured to use NSX Distributed Logical Router for virtual machine networking, there are additional requirements in the following section that must be met.

    • At the time, of this writing, the VMC cloud cannot be configured to meet the VMware HCX-PR requirements.

  • After all requirements are met, Proximity Routing can simply be enabled during the Network Extension operations.

Additional Configuration Requirements for VMware HCX Proximity Routing when NSX Distributed Logical Router (DLR or UDLR) is present.

  • Dynamic Routing Between the Cloud Site Edge Gateway and the Cloud Site DLR

    • Cloud destination site NSX Edge Services Gateway Configuration

      • Enable BGP.

      • Add the Cloud Site DLR as a Neighbor.

      • Add the on-premises peer as a neighbor.

      • Configure Redistribution of Static routes.

    • Cloud destination site NSX DLR Configuration

      • Enable BGP.

      • Add the Cloud Site NSX Edge as a Neighbor.

      • Add BGP Filters (in this specific order):

        • Deny Out Network Extended/Stretch Prefix Lists.

        • Permit Out Any (this filter advertises native Virtual Machine networks).

        • Deny Any In (The NSX DLR must be configured to reach the ESG using its default route).

Procedure

  1. VMware HCX Network Extension is triggered at the source site.
    • The extended subnet's details are provided, including the extended subnet's gateway IP. The system enables VMware HCX-PR for the extension.

    • Upon completion, a routed port group and isolated port group are created for the extended network, at the cloud site.

    • The VMware HCX Network Extension appliance becomes connected to both the routed and isolated extended port groups.

    • VMware HCX creates the extended subnet's gateway on the Cloud destination site DLR using the same IP as the on-premises gateway. ARP filters are applied to prevent the new gateway from being reached from the source site.

  2. A Virtual Machine is migrated into the PR-Extended Network.
    • A virtual machine is migrated to the extended network with VMware HCX Proxy vMotion placed into the ISO segment. The ARP table shows the on-premises gateway's MAC address, the VM continues sending routing requests to the on-premises gateway mac address.

      Note:

      VMware HCX handles virtual machine membership in the isolated "ISO" network as an internal configuration operation. Virtual machine membership in the ISO port groups is not reflected in the vCenter Network view.

    • Whenever the vMotion VM is rebooted (at the administrator's discretion), VMware HCX connects the VM to the routed (non-ISO) version of the network. Post-reboot, the VM ARP table shows the Cloud Site DLR MAC address for its gateway and being routing locally, without hair-pinning.

      Note:

      Rewiring into the non-ISO/routed port group requires VM tools to be running and detecting the virtual machine's IP address.

    • A Virtual Machine migrated to the extended network with VMware HCX Bulk Migration is placed directly on the non-ISO tagged/routed extended network. The ISO tagged isolated network is not used in this case.

Results

  • VMware HCX Injected Routes

    • After the system places the virtual machine on the non-ISO/routed extended network, VMware HCX adds a /32 host route for the VM. The VM is then added to the destination site NSX Edge Gateway with the destination site DLR Uplink IP as the next hop.

    • Also, VMware HCX adds exclusion static routes covering every IP from the extended subnet that does not belong to migrated VMs to the destination site DLR. With the destination site NSX Edge Gateway Downlink IP as the next hop.