VMware SD-WAN supports interconnection of multiple Hub Edges or Hub Clusters to increase the range of Spoke Edges that can communicate with each other. This feature allows communication between the Spoke Edges connected to one Hub Edge or Hub Cluster and the Spoke Edges connected to another Hub Edge or Hub Cluster, using multiple overlay and underlay connections.

When a Spoke Edge tries to connect to a Hub Cluster, one of the members from the Hub Cluster is selected as the Hub to the Spoke Edge. If this Hub goes down, another member from the same Hub Cluster is automatically selected to serve the Spoke Edge, without any user configuration. The Hub Cluster members are connected to each other via underlay (BGP), and can exchange the routes and data using this underlay connection. Spoke Edges connected to different members of the same Hub Cluster can then communicate with each other using this underlay connection. This solution provides better resiliency.

When two Hub Clusters are connected to each other, one Cluster acts as a Hub to the other Cluster (and the reverse relation can also exist, depending on the configuration). The VCEs from one Cluster get their own Hubs from the other Cluster. The end Spoke Edges connected to these Hub Clusters can then communicate with each other through these two Hub Clusters and the intermediate VCRP (VeloCloud Route Protocol) hops.

The below diagram explains this concept:
In this example, Cluster 1 (C1) and Cluster 2 (C2) are Hub Clusters, and S1 and S2 are the set of Spoke Edges connected to C1 and C2 respectively. S1 can communicate with S2 through the following connections:
  • Overlay connection between S1 and C1.
  • Overlay connection between S2 and C2.
  • Overlay connection between C1 and C2.
  • Underlay connection within C1.
  • Underlay connection within C2.

In this way, the Hub Clusters can exchange routes with each other, providing a way for the packets to flow between Spoke Edges connected to different Hub Clusters.

Note: Customers can deactivate this feature if they do not want all their Spoke Edges to communicate with all other Spoke Edges connected across Hub Clusters.

Limitations

When the Hub or Cluster Interconnect feature is activated:
  • Only those branches that are configured for Edge-to-Hub can still get Edge-to-Edge routes.
  • When a route is exchanged between the Hub Clusters with a common Layer 3 device, the BGP metric is overwritten by the Cluster metric.
  • Dynamic tunnels between Spoke Edges connected to different Hub Clusters are not supported.
  • Hub or Cluster Interconnect through Gateway is not supported.
  • Edge-to-Edge through Hub and Gateway is not supported.
  • Exchanging routes between Hub Cluster members using OSPF is not supported.
  • Community strings are added to all the routes to assist with interconnect routing.

Configuring Hub or Cluster Interconnect

Prerequisites

Ensure that the Cloud VPN service is activated for the Cluster Profile associated with the Edge Cluster.

Note: Activating Hub or Cluster Interconnect feature introduces a fundamental change to the VMware SD-WAN Routing Protocol where it allows packets to traverse more than one hop in the network. While this change has been tested in representative topologies, it is not possible to test this change for all the encountered routing scenarios. As a result, VMware is releasing this feature as an Early Access and will closely monitor the deployments, where it is activated, for unexpected routing behavior.

Procedure

  1. Create new Clusters:
    1. In the Enterprise portal, go to Configure > Network Services > Clusters and Hubs.
    2. Click New to create new Clusters.
    3. Associate the available Edges to these Clusters.
    4. Click Save Changes.
  2. Create a Profile for each of these Clusters:
    1. Go to Configure > Profiles.
    2. Create a separate Profile for each new Cluster. For information on how to create a Profile, see Create Profile with New Orchestrator UI.
  3. Designate Hub to the Cluster Profile:
    1. On the Profile Device Settings screen, go to VPN Services and turn on the Cloud VPN service.
    2. Select Enable Branch to Hubs and Enable Branch to Branch VPN check boxes.
    3. Select Hubs for VPN, and then click Edit Hubs located under Branch to Branch VPN Hub Designation.
    4. You can choose the Clusters to act as Hubs to each other as shown below:
      In this example, Cluster 1 (C1) acts as a Hub to Cluster 2 (C2).
    5. Click Update Hubs.
  4. Assign Profiles to the Edges: Navigate to Configure > Edges to assign Profiles to the available Edges.
  5. Activate 'Hub or Cluster Interconnect' feature: On the Profile Device Settings screen, navigate to Hub or Cluster Interconnect located under VPN Services, and then select the Enable check box.
    Caution: Activating or deactivating the Hub or Cluster Interconnect feature causes all Edge devices associated with the Profile to restart. Hence, it is recommended to configure the feature only in a maintenance mode to prevent traffic disruption.
    This activates the feature and creates a tunnel between the Hub Clusters which allows their respective Spoke Edges to communicate with each other.