This section provides an overview of VMware SASE routing functionality including route types, connected and static routes, and dynamic routes with tie-breaking scenarios and preference values in Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC).
Overview
VMware SASE routing is built on a proprietary protocol called VCRP, which is multi-path capable and secured through VCMP transport. The SD-WAN endpoints are connected using VCRP similar to iBGP full mesh. The SD-WAN Gateway acts as a BGP route reflector which reflects the routes from one SD-WAN Edge to another SD-WAN Edge within the customer enterprise based on the profile settings.
The following diagram illustrates a typical SD-WAN deployment with Multi-Cloud Non SD-WAN Destinations, where the Orchestrator performs the route calculation (as contrasted with the newer and preferred method using Dynamic Cost Calculation (DCC).
SD-WAN Components for Routing Purposes
- The SD-WAN Edge is an Enterprise-class device or virtualized cloud instance that provides secure and optimized connectivity to private, public and hybrid applications, and virtualized services. In SD-WAN routing, the Edge is a Border Gateway. An Edge can function as a regular Edge (with no Hub configuration), as a Hub by itself or as part of a cluster, or as a spoke (when Hubs are configured).
- The SD-WAN Gateway is autonomous, stateless, horizontally scalable, and cloud-delivered to which Edges from multiple tenants can connect. For any SD-WAN deployment, several SD-WAN Gateways are deployed as a geographically distributed (for lower latency) and horizontally scalable (for capacity) network with each Gateway acting as a Route Reflector for their connected Edges.
All routes that are locally learned on an Edge are sent to the Gateway based on the configuration. The Gateway then reflects these routes to other Edges in the enterprise, allowing for efficient full mesh VPN connectivity without building a full mesh of tunnels.
- The SASE Orchestrator is a multi-tenant cloud-based configuration and monitoring portal. In SD-WAN routing, the Orchestrator manages routes for all enterprises and can override default routing behavior.
See the image below for an illustration of the VMware SD-WAN components for routing purposes.
Route Types
- Local Routes: Any route that is learned locally on a SD-WAN Edge. This can either be a connected subnet, statically configured route, or any route that is learned via BGP or OSPF.
- Remote Routes: Any route that is learned from VCRP, in other words, a route that is not locally present on an Edge is a remote route. This route originated from a different Edge and is reflected by the Gateway to other Edges in the customer enterprise based on the configuration.
SD-WAN uses a strict order to route traffic for non-dynamic routes (BGP and OSPF) that cannot be altered. However, in some scenarios, you can use the Longest Prefix Match technique to manipulate how the routing flows.
- Longest prefix length.
- Local connected.
- Local static if preferred option enabled (LAN static < WAN static).
- If preferred option is not enabled, overlay routes would be preferred.
- NSD static routes local.
- NSD IPsec wins over NSD GRE.
- Remote NSD static.
- Remote Edge connected.
- Remote Edge LAN/WAN static.
- PG static.
- PG secure static > PG non-secure static.
- Dynamic routes (Overlay Flow Control (OFC) or Distributed Cost Calculation Driven route order).
- Site Local (OSPF Inter/Intra, BGP non-uplink) is preferred than overly dynamic routes.
- Local OSPF inter/intra area routes wins over Local BGP.
- Local BGP wins over Local OSPF-external (OE1/OE2).
- Remote routes with preferred cost wins over non-preferred local route (OE1,OE2,UPLINK BGP).
- Within the remote dynamic routes preference is considered(lower preference wins).
- If preference is same, BGP attributes and OSPF metrics are compared).
- OSPF INTRA> INTER > OE1 > OE2
- BGP
- Higher Local preference
- Lower AS_PATH Length
- Smaller BGP metric
- For more details on preference calculation, please refer to the DCC section.
Connected and Static Routes
This section includes essential information regarding connected and static routes. A connected route is a route configured to a network that is directly attached to the interface. A static route is useful for special cases in which static routes are needed for existing network attached devices, such as printers. More information about static routes can be found at Configure Static Route Settings.
Connected Routes
- For a connected route to be visible in SD-WAN, configure the following settings on the Orchestrator:
- Cloud VPN must be activated.
- The connected route must be configured with a valid IP address.
- The Edge interface for this route must be up at Layer 1, and functional at Layers 2 and 3.
- VLANs associated with this Edge interface must also be up.
- The Advertise flag must be set on the Edge interface under Interface IP settings for the configured connected route.
- For a static route to be visible in SD-WAN, configure the following settings on the Orchestrator:
- Cloud VPN must be activated.
- The static route configuration must have Advertised checked.
- Static routes can forward the traffic to the WAN underlay or LAN.
- Adding a static route bypasses the NAT on the Edge interface.
- ECMP (Equal-cost multi-path routing) with a static route is not supported, and only the first static route would be used.
- Use an ICMP Probe to avoid blackholing traffic in case of failure in next hop.
- A static route with the Preferred flag checked is preferred over any VPN route learned over the Overlay.
When the Preferred check box is selected, the static route will always be matched first, even if a VPN route with a lower cost is available.
Not selecting this option means that any available VPN route is matched over the static route, even when the VPN route has a higher cost than the static route. The static route is matched only when corresponding VPN routes are not available.
When the Advertise check box is selected, the static route is advertised over VPN and other SD-WAN Edges in the network will have access to the resource. This also enables static route redistribution into a routing protocol like local BGP/OSPF.
Do not select this option when a private resource like a remote worker's personal printer is configured as a static route and other users should be prevented from accessing the resource.
The OFC Global Advertise Flags control which routes are added to the overlay. By default, the following route types are not advertised into the overlay: External OSPF and Non SD-WAN Destination iBGP. In addition, if an Edge is acting as both Hub and Branch, the Global Advertise Flags configured for the Branch will be used, not the Hub.
A Self Route refers to an interface-based prefix using IP Longest prefix match (LPM) (for example: 172.16.1.10/32) which is installed locally on the Edge but is not advertised to remote Edges. Another term for Self Routes is "Interface Routes." In the Edge logs, self routes are displayed as route flag "s."
A self route differs from a connected route, as a connected route can be advertised into the overlay so that the remote Edge clients can reach back to clients belonging to the connected route on the source Edge side. Self routes are strictly local to the Edge itself.
A Cloud Route is indicated with a "v" flag and refers to a route installed on an Edge pointing to Primary VMware SD-WAN Gateway for multi-path traffic destined for the Internet (in other words, Internet traffic using Dynamic Multi-Path Optimization (DMPO) which leverages a Gateway prior to reaching the Internet).The Edge also uses a cloud route via a corresponding Gateway for management traffic destined for a VMware Orchestrator, which is hosted on the public cloud.
Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC)
Distributed Cost Calculation Overview
Distributed Cost Calculation (DCC) is a feature that leverages the SD-WAN Edges and Gateways for route preference calculation instead of relying on the SASE Orchestrator. The Edge and Gateway each insert the routes instantly upon learning them and then convey these preferences to the Orchestrator.
DCC resolves an issue seen in large scale deployments where relying solely on the Orchestrator can prevent timely route preference updates either because it could not be reached by an Edge or Gateway to receive updated routing preferences, or because the Orchestrator could not deliver route updates quickly when it is calculating a large number of them at one time. Distributing the responsibilities for route preference calculation to the Edges and Gateways ensures fast and reliable route updates.
How Distributed Cost Calculation Preference is Done
| Edge | Partner Gateway / Hosted Gateway |
|---|---|
| NSD E BGP | NSD E/I BGP |
| NSD I BGP | E/I BGP |
| NSD Uplink BGP | |
| OSPF O | |
| OSPF IA | |
| E BGP | |
| I BGP | |
| OSPF OE1 | |
| OSPF OE2 | |
| Uplink BGP |
| O = OSPF Intra area |
| IA = OSPF Inter area |
| OE1 = OSPF External Type-1 |
| OE2 = OSPF External Type-2 |
| E BGP = External BGP |
| I BGP = Internal BGP |
| NSD = Non SD-WAN Destination |
Each route type has a preference value (consider the preference as the cost in this document), and each learned route is assigned a preference value based on the route's type. The lower the preference value, the higher the priority. Table 1-3 lists the default preference value for each route type.
| Device | Route Type | Default Preference |
| Edge/Hub | NSD E BGP | 997 |
| Edge/Hub | NSD I BGP | 998 |
| Gateway | NSD E/I BGP | 999 |
| Edge/Hub | NSD Uplink BGP | 1000 |
| Edge/Hub | OSPF O | 1001 |
| Edge/Hub | OSPF IA | 1002 |
| Edge/Hub | E BGP | 1003 |
| Edge/Hub | I BGP | 1004 |
| Partner Gateway | E/I BGP | 1005 |
| Edge/Hub | OSFP OE1 | 1001006 |
| Edge/Hub | OSPF OE2 | 1001007 |
| Hub/Edge | BGP Uplink | 1001008 |
The preference values displayed in the table above are based on the default priority order in the Overlay Flow Control configuration. The values will be adjusted accordingly if the default order is changed.
Dynamic Route Workflow
- The Edge or Gateway learns a dynamic route.
- SD-WAN internally identifies what type of route it is and its default preference value.
- SD-WAN assigns the correct preference value and installs the route in the routing information base (RIB) and forwarding information base (FIB).
- SD-WAN considers the default advertising action configured for this route. Based on the advertising action, SD-WAN either advertises the route across the customer enterprise (advertised) or takes no action apart from adding the route locally into the RIB and FIB (not advertised).
- SD-WAN then synchronizes this route to the Orchestrator which displays it on the Orchestrator.
Preferred VPN Exit Points
This section covers Preferred VPN Exit Points: what they are, what routes can fall into which categories, and using route pinning to override default values.
In the SD-WAN service of the Enterprise portal, when navigating to , you can see a section titled Preferred VPN Exits. This section displays default priorities and marks some route categories to be preferred over others.
- Edge: Any internal route that can be learned either on a Hub or Spoke Edge falls under this category and is marked with the highest priority. An internal route cannot be an OSPF OE 1 / OE 2 or BGP Uplink type route.
- Hub: Any external Route that is learned on an Edge/Hub falls into the Hub category and typically has a lower priority. Hub routes include OSPF OE1/2 and BGP Uplink.
- Partner Gateway: Any route learned on a Partner Gateway.
- Router: A router represents any route prefix learned by an Edge with a BGP or OSPF and determines the preference that is assigned to a dynamic route. Typically, all exit points above the Router in the VPN Exit are assigned a low preference value (preferred cost) and are more preferred, while all exit points below the Router are assigned a higher preference value and are less preferred.
- For example: When DCC is activated, all routes that belong to VPN Exit Points (Edge, Partner Gateway, or Hub) that are above Router get a preference value of less than 1,000,000, and the routes that are below Router get a preference value greater than 1,000,000.
- In the example below, the VPN Exit Points above Router, which are NSD, Edge, and Partner Gateway will get a preference value less than 1,000,000 and Hub will get a preference value greater than 1,000,000.
Pinning a Route to Override a Default Preference Value
- A user pins a route on the Overlay Flow Control page by either:
- On the Routes List, select one or more routes and then click the Pin Learned Route Preference option.
- Modifying the order of the Preferred VPN Exits by clicking Edit under the table.
- The Orchestrator sends this routing event to the relevant Edges in the customer enterprise.
- The Edges override the previous preference value to match the pinned order.
- The preference values that get assigned to pinned routes start from 1, 2, 3, and so on (the lowest values and thus the highest preferences), and this matches the order of the routes on the Overlay Flow Control page.
Note: For more information on pinning a route, consult Configure Subnets.
Tie-Breaking Scenarios for All Types of Routes
What happens when an Edge receives the same prefix for two or more sources/neighbors?
A potential scenario in SD-WAN deployments is for the same prefix to be advertised from two different Edges or Partner Gateways. With VMware SD-WAN, if the subnets are within the same category (Edge, Hub, or Partner Gateway) and have the same preference value, the BGP attributes or OSPF metrics are first considered for route sorting.
If there is still a tie, SD-WAN uses the logical ID (which is derived from the Edge or Gateway's universally unique identifier (UUID)) of the next hop device to break the tie. The next hop device can be a Gateway or a Hub Edge depending on the type of Branch to Branch VPN being used. If the customer enterprise is using Branch to Branch via Gateway, the next hop is a Gateway, while a customer using Branch to Hub would have the next hop be a Hub Edge.
There is a final tie-breaker if multiple Gateways advertise the same exact route type and preference. This final tie-breaker prefers the oldest route learned. To ensure the routing outcome you want, you can either pin certain routes or configure the BGP attributes and costs to favor some routes over others.
See the image below for an illustration of preference calculation and route sorting for dynamic routes.
- Spoke1 and Spoke2 learn the route as BGP routes (non-uplink).
- Hub1 and Hub2 learn the routes as uplink BGP routes.
- PG1 also learns the same route.
- Branch to branch via Hub1 and Hub2 is enabled in spoke profile.
- Since spoke1 and spoke2 learn the route as BGP, they pick the preferred cost value (the preference value is referred to as cost in this section) is 1003, as per the DCC preference mapping table.
- Route 9.9.9.9/32 will be installed in FIB of Spoke1 and Spoke2 with a reference cost of 1000000. As always, the underlay route will be installed in FIB with a reference cost only. The derived cost/preference from the DCC preference table is for remote SD-WAN entities (Edges/Gateway) to use for route sorting.
- Spoke1 and Spoke2 redistribute the route over VCRP with a derived cost of 1003 to the Gateway and remote Edges/Hubs. The below output image shows the derived cost/preference in spokes.
- Similarly Hub1 and Hub2 learn the route and derive the non-preferred cost (1001008), since they learn the route as an uplink route. Hubs redistribute the route to Gateways and other Edges with this cost. The below output shows the derived cost/preference in Hubs.
- PG1 learns the same route from BGP and uses cost 1005 and redistributes it to the Edges. The below output shows the derived cost/preference in PG.
- Spoke1 receives the route from Hub1 and Hub2 with the non-preferred cost of 1001008. Spoke1 has the preferred cost of 003. Hence, spoke1’s own underlay route will be preferred and Hub routes will be installed below the underlay route (SB). Within the Hub routes, if the preference (cost) is the same, BGP attributes will be compared for route sorting. If BGP attributes are also the same, then the Hub order will be used to install the routes.
- Spoke1 receives a route from Spoke2 and PG1 with costs 1003 and 1005, respectively. Since Spoke1 is having preferred cost 1003, and receives routes from Spoke2 and PG1 with a preferred cost (<100000), Spoke1 adds the reference cost 1000000 to the incoming preferred cost and install the routes in FIB. In this case, Spoke2’s route will be installed with a cost of 1001003 and PG1’s route will be installed with a cost of 1001005.
- The same route sorting logic is applied in Spoke2 or even Hubs if they learn the route as non-uplink route.
- If there is no underlay route learned in any entity, there will not be any correction to the received route preference/cost. The routes will be installed as per the received preference/cost.
- Hubs install their own underlay route (SB) with a reference cost of 1000000 in FIB.
- Hubs receive spoke routes with a preferred cost of 1003. Since cost is same between the spokes, BGP attributes will be compared and sorted based on that. If BGP attributes are also same, then spoke logical id will be used for sorting(lower destination logical ID wins the tie-breaker). The spoke’s routes will be installed with received cost as it is.
- The Hub receives PG1’s route with preferred cost. Therefore, it installs with that cost as is.
- PG1 installs its own underlay route (PB) with preference 100000.
- PG1 receives spoke routes and Hub route with corresponding preference. Routes are placed in the FIB based on the preference value. If preference are same, BGP attributes are considered. If they are also same, then logical ID will be used for sorting.
- In PG, there is no preference/cost correction.
- If DCC is not enabled, the advertisement verdict and the preference calculation is performed by the Orchestrator. Each entity (Edge or Gateway) sends the learned routes to the Orchestrator and expects to receive a reply from the Orchestrator. Upon receiving the reply from the Orchestrator, Edge, or Gateway would begin redistributing the routes to other SDWAN entities if the advertise flag is "true" in the reply.
- The route ordering remains the same, as is the case of DCC being enabled, but the preference values are not fixed in this scenario of DCC being disabled.
- The reference preference/cost is 512 for the Orchestrator based preference calculation. The preference/cost < 512 is the preferred cost, whereas > 512 is given to non-preferred routes (UPLINK routes, OSPF external routes). Other route sorting logic remains the same as when DCC is enabled.
- If spoke2 learns the route first and sends it to the Orchestrator, the Orchestrator will begin assigning the preference based on the entity and route type. Since spoke2 learns as non-uplink, the Orchestrator will assign the preference value (for example,64). Later, when spoke1 sends the same route to the Orchestrator, the Orchestrator will compare the entity, route type, and route attributes. If it is better, it will assign the preference to < 64. If it is worse, it will assign the preference to > 64.
- Hubs learn the routes as uplink routes and send them to the Orchestrator. The Orchestrator assigns a non-preferred cost (>512); in this example, it is 4096. If the preference is the same, the Hub order will be used to sort the routes in the spokes.
- When DCC is disabled, the route order in spoke1 (with a non-uplink route) will look like the following image.
- The router order in Hubs with an uplink route will look like the following image.
- The route order in PG will look like the following image.
- Longest prefix length.
- NSD static routes local .
- Remote NSD static.
- PG secure static.
- Enterprise level PG static route wins over Global Level PG static.
- Remote connected/static.
- Edge logical_id will be the tie breaker (higher logical ID wins).
- Dynamic routes (Overlay Flow Control (OFC) or Distributed Cost Calculation Driven route order).
- Dynamic route sorting will be based on preference value. Lower preference wins.
- Unlike an Edge, there is no preference in auto correction in Gateway. For dynamic routes, the Gateway installs the routes with the received preference. The local route will always be installed with the reference preference of 1000000.
Note: For more information about Preference Calculation, see the “Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC)” section. - PG non-secure static.
