VMware SD-WAN Routing Overview

Overview

VMware SD-WAN routing is built on a proprietary protocol called VCRP, which is multi-path capable and secured through VCMP transport. The SD-WAN endpoints are connected using VCRP in a manner similar to iBGP full mesh. The SD-WAN Gateway acts as a BGP route reflector which reflects the routes from one SD-WAN Edge to another SD-WAN Edge within the customer enterprise based on the profile settings.

The following diagram depicts a typical SD-WAN deployment with Multi Cloud Non SDWAN Destinations.

SD-WAN Components

VMware SD-WAN routing uses three components: Edge, Gateway, and Orchestrator.

The SD-WAN Edge is an enterprise-class device or virtualized cloud instance that provides secure and optimized connectivity to private, public and hybrid applications, and virtualized services. In SD-WAN routing the Edge is a Border Gateway. An Edge can function as a regular Edge (with no Hub configuration), as a Hub by itself or as part of a cluster, or as a Spoke (when Hubs are configured).
The SD-WAN Gateway is autonomous, stateless, horizontally scalable, and cloud-delivered to which Edges from multiple tenants can connect. For any SD-WAN deployment, several SD-WAN Gateways are deployed as a geographically distributed (for lower latency) and horizontally scalable (for capacity) network with each Gateway acting as a Route Reflector for their connected Edges.
All routes that are locally learned on an Edge are sent to the Gateway based on the configuration. The Gateway then reflects these routes to other Edges in the enterprise, allowing for efficient full mesh VPN connectivity without building a full mesh of tunnels.
The SASE Orchestrator is a multi-tenant cloud-based configuration and monitoring portal. In SD-WAN routing the Orchestrator manages routes for all enterprises and can override default routing behavior.

Route Types

There are two general types of routes for SD-WAN:

Local Routes: Any route that is learned locally on a SD-WAN Edge. This can either be a connected subnet, statically configured route, or any route that is learnt via BGP or OSPF.

Remote Routes: Any route that is learned from VCRP, in other words a route that is not locally present on an Edge is a remote route. This route originated from a different Edge and is reflected by the Gateway to other Edges in the customer enterprise based on the configuration.

There is a strict order that SD-WAN uses to route traffic for non-dynamic routes (BGP and OSPF) that cannot be altered. However, in some scenarios you can use the technique of Longest Prefix Match to manipulate how the routing flows.

Table 1. Order of Route Types
1. Longest Prefix Match
2. Connected Local
3. Static LAN/WAN Local
4. Connected Remote
5. Static LAN/WAN Remote
6. Static Non SD-WAN Destination
7. Static Partner Gateway
8. Overlay Flow Control (OFC) Driven Route Order

Note: Between local and remote routes of the same type, VMware SD-WAN will prefer the local over the remote. For example a local connected route is preferred over a remote connected route. Similarly, for a local static route versus a remote static route, the local static route is preferred.

Connected and Static Routes

This section includes essential information regarding connected and static routes. A connected route is a route to a network that is directly attached to the interface. Information about static routes can be found in Configure Static Route Settings.

Connected Routes

For a connected route to be visible in SD-WAN, configure the following settings on the Orchestrator UI:
- Cloud VPN must be activated.
- The connected route must be configured with a valid IP address.
- The Edge interface for this route must be up at Layer 1, and functional at Layers 2 and 3.
- VLANs associated with this Edge interface must also be up.
- The Advertise flag must be set on the Edge interface under Interface IP settings for which the connected route is configured.

Static Routes

For a static route to be visible in SD-WAN, configure the following settings on the Orchestrator UI:
- Cloud VPN must be activated.
- The Advertise flag must be set on the Edge interface for which the static route is configured.
- The static route configuration must have Preferred and Advertised checked.
Static Routes can forward traffic to the LAN or WAN Underlay.
Adding a static route bypasses the NAT on the Edge interface.
ECMP (Equal-cost multi-path routing) with a static route is not supported, and only the first static route would be used.
Use an ICMP Probe to avoid blackholing traffic.
Static routes that do not have the Preferred flag checked are not advertised into the Overlay.
A static route with the Preferred flag checked is preferred over any VPN route learned over the Overlay.

Note: The difference between the Preferred flag, and the Advertise flag:

When the Preferred check box is selected, the static route will always be matched first, even if a VPN route with a lower cost is available.

Not selecting this option means that any available VPN route is matched over the static route, even when the VPN route has a higher cost than the static route. The static route is matched only when corresponding VPN routes are not available.

The Preferred option is not available for an IPv6 address type.

When the Advertise check box is selected, the static route is advertised over VPN routes and other SD-WAN Edges in the network will have access to the resource.

Do not select this option when a private resource like a remote worker's personal printer is configured as a static route and other users should be prevented from accessing the resource.

The Advertise option is not available for an IPv6 address type.

Note: There are two additional route types: Self Routes and Cloud Routes which are installed on an Edge depending on the Edge's configuration. Each route has a narrow application outlined below and require no additional treatment beyond their mention here:

A Self Route refers to an interface-based prefix using IP Longest prefix match (LPM) (for example: 172.16.1.10/32) which is installed locally on the Edge but is not advertised to remote Edges. Another term for Self Routes is "Interface Routes". When looking in an Edge's logs, a user would see these Self Routes with the route flag "s".

A Self Route is distinguished from a Connected Route as a Connected Route can be advertised into the overlay so that the remote Edge clients can reach back to clients belonging to the connected route on the source Edge side. Self Routes are strictly local to the Edge itself.

A Cloud Route is indicated with a "v" flag and refers to a route installed on an Edge pointing to a VMware SD-WAN Gateway for multi-path traffic destined for the internet (in other words, internet traffic using Dynamic Multi-Path Optimization (DMPO) which leverages a Gateway prior to reaching the internet).

The Edge also uses a Cloud Route via a corresponding Gateway for management traffic destined for a VMware Orchestrator which is hosted on the public cloud.

Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC)

This section explains how a route order using OFC with DCC works.

Important: This material is valid only for customers who have Distributed Cost Control (DCC) activated. DCC was first made available in SD-WAN Release 3.4.0 and is now expected to be activated for all customers. This feature will automatically be activated for new customers in an upcoming release. For more information about DCC including best practices, see Configure Distributed Cost Calculation.

Distributed Cost Calculation Overview

Distributed Cost Calculation (DCC) is a feature that leverages the SD-WAN Edges and Gateways for route preference calculation instead of relying on the SASE Orchestrator. The Edge and Gateway each insert the routes instantly upon learning them and then convey these preferences to the Orchestrator.

DCC resolves an issue seen in large scale deployments where relying solely on the Orchestrator can prevent timely route preference updates either because it could not be reached by an Edge or Gateway to receive updated routing preferences, or because the Orchestrator could not deliver route updates quickly when it is calculating a large number of them at one time. Distributing the responsibilities for route preference calculation to the Edges and Gateways ensures fast and reliable route updates.

How Distributed Cost Calculation Preference is Done

Table 1-2 includes the types of dynamic routes supported in SD-WAN while table 1-3 is a glossary of route types. A dynamic route is first categorized by whether it is learned on the Edge or the Gateway.

Table 2. Dynamic Route Types
Edge	Partner Gateway / Hosted Gateway
NSD E BGP	NSD E/I BGP
NSD I BGP	E/I BGP
NSD Uplink BGP
OSPF O
OSPF IA
E BGP
I BGP
OSPF OE1
OSPF OE2
Uplink BGP

Table 3. Route Type Meanings
O = OSPF Intra area
IA = OSPF Inter area
OE1 = OSPF External Type-1
OE2 = OSPF External Type-2
E BGP = External BGP
I BGP = Internal BGP
NSD = Non SD-WAN Destination

Note: Non SD-WAN Destination (NSD) support with OFC is available from Release 4.3.0 and forward. For more information on NSDs, consult Configure a Non VMware SD-WAN Site

Each route type has a preference value, and each learned route is assigned a preference value based on the route's type. The lower the preference value, the higher the priority. Table 1-4 lists the default preference value for each route type.

Table 4. Preference Values
Device	Route Type	Default Preference
Edge	NSD E BGP	997
Edge	NSD I BGP	998
Gateway	NSD E/I BGP	999
Edge	NSD Uplink BGP	1000
Edge	OSPF O	1001
Edge	OSPF IA	1002
Edge	E BGP	1003
Edge	I BGP	1004
Partner Gateway	E/I BGP	1005
Hub	OSFP OE1	1001006
Hub	OSPF OE2	1001007
Hub	BGP Uplink	1001008

Dynamic Route Workflow

The Edge or Gateway learns a dynamic route.
SD-WAN internally identifies what type of route it is and its default preference value.
SD-WAN assigns the correct preference value and installs the route in the routing information base (RIB) and forwarding information base (FIB).
SD-WAN considers the default advertising action configured for this route. Based on the advertising action, SD-WAN either advertises the route across the customer enterprise (advertised) or takes no action apart from adding the route locally into the RIB and FIB (not advertised).
SD-WAN then synchronizes this route to the Orchestrator which displays it on the Orchestrator UI.

Preferred VPN Exit Points

This section covers Preferred VPN Exit Points: what they are, what routes can fall into which categories, and using route pinning to override default values.

On the Orchestrator UI, when navigating to Configure > Overlay Flow Control you will see a table titled Preferred VPN Exits. This table displays default priorities and marks some route categories to be preferred over others. The images below show the default settings for Preferred VPN Exits as seen on the Classic UI, and the New UI.

Screenshot of Overlay Flow Control screen showing Preferred VPN Exits on the Classic UI.

The Preferred VPN Exit categories:

Edge: Any internal route that can be learned either on a Hub or Spoke Edge falls under this category and is marked with the highest priority. An internal route cannot be an OSPF OE 1 / 2 or BGP Uplink type route.
Hub: Any external site that is learned on an Edge falls into the Hub category and typically has a lower priority. Hub routes include OSPF OE1/2 and BGP Uplink.
Partner Gateway: Any route learned on a Partner Gateway.
Router: This is used as a marker to determine the preference that is assigned to a dynamic route. Router is not a type of route. Typically, all exit points above Router in the VPN Exit are assigned a low preference value and are thus more preferred, while all exit points below Router are assigned a higher preference value and are thus less preferred.
- For example: When DCC is activated, all routes that belong to VPN Exit Points (Edge, Partner Gateway, or Hub) that are above Router get a preference value of less than 1,000,000, and the routes that are below Router get a preference value greater than 1,000,000.
- In the example below, the VPN Exit Points above Router, which are NSD, Edge, and Partner Gateway will get a preference value less than 1,000,000 and Hub will get a preference value greater than 1,000,000.

Pinning a Route to Override a Default Preference Value

SD-WAN has a route pinning feature that allows a user to override the default preference value assigned to any dynamic route. Once a dynamic route is learned and synchronized with the Orchestrator, the user can navigate to the Overlay Flow Control page and override the default order. The workflow for this is as follows:

A user pins a route on the Overlay Flow Control page by either:
1. On the Routes List, select one or more routes and then click the Pin Learned Route Preference option.
2. Modifying the order of the Preferred VPN Exits by clicking Edit under the table.
The Orchestrator sends this routing event to the relevant Edges in the customer enterprise.
The Edges override the previous preference value to match the pinned order.
The preference values that get assigned to pinned routes start from 1, 2, 3, and so on (the lowest values and thus the highest preferences), and this matches the order of the routes on the Overlay Flow Control page.
Note: For more information on pinning a route, consult Configure Subnets.

Tie-Breaking Scenarios for Dynamic Routes

What happens when an Edge receives the same prefix for two or more sources/neighbors?

A potential scenario in SD-WAN deployments is for the same prefix to be advertised from two different Edges or Partner Gateways. With VMware SD-WAN, if the subnets are within the same category (Edge, Hub, or Partner Gateway) and have the same preference value, the BGP attributes or OSPF metrics are first considered for route sorting.

If there is still a tie, SD-WAN uses the logical ID (which is derived from the Edge or Gateway's universally unique identifier (UUID)) of the next hop device to break the tie. The next hop device can be a Gateway or a Hub Edge depending on the type of Branch to Branch VPN being used. If the customer enterprise is using Branch to Branch via Gateway, the next hop is a Gateway, while a customer using Branch to Hub would have the next hop be a Hub Edge.