This section provides an overview of VMware SASE routing functionality including route types, connected and static routes, dynamic routes with tie-breaking scenarios for them, and preference values in Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC).
Overview
VMware SASE routing is built on a proprietary protocol called VCRP, which is multi-path capable and secured through VCMP transport. The SD-WAN endpoints are connected using VCRP in a manner similar to iBGP full mesh. The SD-WAN Gateway acts as a BGP route reflector which reflects the routes from one SD-WAN Edge to another SD-WAN Edge within the customer enterprise based on the profile settings.
The following diagram depicts a typical SD-WAN deployment with Multi-Cloud Non SD-WAN Destinations where the Orchestrator performs the route calculation (as contrasted with the newer and preferred method using Dynamic Cost Calculation (DCC).
SD-WAN Components
- The SD-WAN Edge is an Enterprise-class device or virtualized cloud instance that provides secure and optimized connectivity to private, public and hybrid applications, and virtualized services. In SD-WAN routing the Edge is a Border Gateway. An Edge can function as a regular Edge (with no Hub configuration), as a Hub by itself or as part of a cluster, or as a Spoke (when Hubs are configured).
- The SD-WAN Gateway is autonomous, stateless, horizontally scalable, and cloud-delivered to which Edges from multiple tenants can connect. For any SD-WAN deployment, several SD-WAN Gateways are deployed as a geographically distributed (for lower latency) and horizontally scalable (for capacity) network with each Gateway acting as a Route Reflector for their connected Edges.
All routes that are locally learned on an Edge are sent to the Gateway based on the configuration. The Gateway then reflects these routes to other Edges in the enterprise, allowing for efficient full mesh VPN connectivity without building a full mesh of tunnels.
- The SASE Orchestrator is a multi-tenant cloud-based configuration and monitoring portal. In SD-WAN routing the Orchestrator manages routes for all enterprises and can override default routing behavior.
Route Types
- Local Routes: Any route that is learned locally on a SD-WAN Edge. This can either be a connected subnet, statically configured route, or any route that is learnt via BGP or OSPF.
- Remote Routes: Any route that is learned from VCRP, in other words a route that is not locally present on an Edge is a remote route. This route originated from a different Edge and is reflected by the Gateway to other Edges in the customer enterprise based on the configuration.
There is a strict order that SD-WAN uses to route traffic for non-dynamic routes (BGP and OSPF) that cannot be altered. However, in some scenarios you can use the technique of Longest Prefix Match to manipulate how the routing flows.
| 1. Longest Prefix Match |
| 2. Connected Local |
| 3. Static LAN/WAN Local |
| 4. Connected Remote |
| 5. Static LAN/WAN Remote |
| 6. Static Non SD-WAN Destination |
| 7. Static Partner Gateway |
| 8. Overlay Flow Control (OFC) Driven Route Order |
Connected and Static Routes
This section includes essential information regarding connected and static routes. A connected route is a route to a network that is directly attached to the interface. Information about static routes can be found in Configure Static Route Settings.
Connected Routes
- For a connected route to be visible in SD-WAN, configure the following settings on the Orchestrator:
- Cloud VPN must be activated.
- The connected route must be configured with a valid IP address.
- The Edge interface for this route must be up at Layer 1, and functional at Layers 2 and 3.
- VLANs associated with this Edge interface must also be up.
- The Advertise flag must be set on the Edge interface under Interface IP settings for which the connected route is configured.
- For a static route to be visible in SD-WAN, configure the following settings on the Orchestrator:
- Cloud VPN must be activated.
- The Advertise flag must be set on the Edge interface for which the static route is configured.
- The static route configuration must have Preferred and Advertised checked.
- Static Routes can forward traffic to the WAN Underlay for Global Segments, and to the LAN or WAN Underlay for Non-Global Segments.
- Adding a static route bypasses the NAT on the Edge interface.
- ECMP (Equal-cost multi-path routing) with a static route is not supported, and only the first static route would be used.
- Use an ICMP Probe to avoid blackholing traffic.
- A static route with the Preferred flag checked is preferred over any VPN route learned over the Overlay.
When the Preferred check box is selected, the static route will always be matched first, even if a VPN route with a lower cost is available.
Not selecting this option means that any available VPN route is matched over the static route, even when the VPN route has a higher cost than the static route. The static route is matched only when corresponding VPN routes are not available.
The Preferred option is not available for an IPv6 address type.
When the Advertise check box is selected, the static route is advertised over VPN routes and other SD-WAN Edges in the network will have access to the resource.
Do not select this option when a private resource like a remote worker's personal printer is configured as a static route and other users should be prevented from accessing the resource.
The Advertise option is not available for an IPv6 address type.
The OFC Global Advertise Flags control which routes are added to the overlay. By default, the following route types are not advertised into the overlay: External OSPF and Non SD-WAN Destination iBGP. In addition, if an Edge is acting as both Hub and Branch, the Global Advertise Flags configured for the Branch will be used, not the Hub.
A Self Route refers to an interface-based prefix using IP Longest prefix match (LPM) (for example: 172.16.1.10/32) which is installed locally on the Edge but is not advertised to remote Edges. Another term for Self Routes is "Interface Routes". When looking in an Edge's logs, a user would see these Self Routes with the route flag "s".
A Self Route is distinguished from a Connected Route as a Connected Route can be advertised into the overlay so that the remote Edge clients can reach back to clients belonging to the connected route on the source Edge side. Self Routes are strictly local to the Edge itself.
A Cloud Route is indicated with a "v" flag and refers to a route installed on an Edge pointing to a VMware SD-WAN Gateway for multi-path traffic destined for the internet (in other words, internet traffic using Dynamic Multi-Path Optimization (DMPO) which leverages a Gateway prior to reaching the internet).The Edge also uses a Cloud Route via a corresponding Gateway for management traffic destined for a VMware Orchestrator which is hosted on the public cloud.
Overlay Flow Control (OFC) with Distributed Cost Calculation (DCC)
Distributed Cost Calculation Overview
Distributed Cost Calculation (DCC) is a feature that leverages the SD-WAN Edges and Gateways for route preference calculation instead of relying on the SASE Orchestrator. The Edge and Gateway each insert the routes instantly upon learning them and then convey these preferences to the Orchestrator.
DCC resolves an issue seen in large scale deployments where relying solely on the Orchestrator can prevent timely route preference updates either because it could not be reached by an Edge or Gateway to receive updated routing preferences, or because the Orchestrator could not deliver route updates quickly when it is calculating a large number of them at one time. Distributing the responsibilities for route preference calculation to the Edges and Gateways ensures fast and reliable route updates.
How Distributed Cost Calculation Preference is Done
| Edge | Partner Gateway / Hosted Gateway |
|---|---|
| NSD E BGP | NSD E/I BGP |
| NSD I BGP | E/I BGP |
| NSD Uplink BGP | |
| OSPF O | |
| OSPF IA | |
| E BGP | |
| I BGP | |
| OSPF OE1 | |
| OSPF OE2 | |
| Uplink BGP |
| O = OSPF Intra area |
| IA = OSPF Inter area |
| OE1 = OSPF External Type-1 |
| OE2 = OSPF External Type-2 |
| E BGP = External BGP |
| I BGP = Internal BGP |
| NSD = Non SD-WAN Destination |
| Device | Route Type | Default Preference |
|---|---|---|
| Edge | NSD E BGP | 997 |
| Edge | NSD I BGP | 998 |
| Gateway | NSD E/I BGP | 999 |
| Edge | NSD Uplink BGP | 1000 |
| Edge | OSPF O | 1001 |
| Edge | OSPF IA | 1002 |
| Edge | E BGP | 1003 |
| Edge | I BGP | 1004 |
| Partner Gateway | E/I BGP | 1005 |
| Hub | OSFP OE1 | 1001006 |
| Hub | OSPF OE2 | 1001007 |
| Hub | BGP Uplink | 1001008 |
Dynamic Route Workflow
- The Edge or Gateway learns a dynamic route.
- SD-WAN internally identifies what type of route it is and its default preference value.
- SD-WAN assigns the correct preference value and installs the route in the routing information base (RIB) and forwarding information base (FIB).
- SD-WAN considers the default advertising action configured for this route. Based on the advertising action, SD-WAN either advertises the route across the customer enterprise (advertised) or takes no action apart from adding the route locally into the RIB and FIB (not advertised).
- SD-WAN then synchronizes this route to the Orchestrator which displays it on the Orchestrator.
Preferred VPN Exit Points
This section covers Preferred VPN Exit Points: what they are, what routes can fall into which categories, and using route pinning to override default values.
In the SD-WAN service of the Enterprise portal, when navigating to , you can see a section titled Preferred VPN Exits. This section displays default priorities and marks some route categories to be preferred over others.
- Edge: Any internal route that can be learned either on a Hub or Spoke Edge falls under this category and is marked with the highest priority. An internal route cannot be an OSPF OE 1 / 2 or BGP Uplink type route.
- Hub: Any external site that is learned on an Edge falls into the Hub category and typically has a lower priority. Hub routes include OSPF OE1/2 and BGP Uplink.
- Partner Gateway: Any route learned on a Partner Gateway.
- Router: Router represents any route prefix learned by an Edge with BGP or OSPF and determines the preference that is assigned to a dynamic route. Typically, all exit points above Router in the VPN Exit are assigned a low preference value and are thus more preferred, while all exit points below Router are assigned a higher preference value and are thus less preferred.
- For example: When DCC is activated, all routes that belong to VPN Exit Points (Edge, Partner Gateway, or Hub) that are above Router get a preference value of less than 1,000,000, and the routes that are below Router get a preference value greater than 1,000,000.
- In the example below, the VPN Exit Points above Router, which are NSD, Edge, and Partner Gateway will get a preference value less than 1,000,000 and Hub will get a preference value greater than 1,000,000.
Pinning a Route to Override a Default Preference Value
- A user pins a route on the Overlay Flow Control page by either:
- On the Routes List, select one or more routes and then click the Pin Learned Route Preference option.
- Modifying the order of the Preferred VPN Exits by clicking Edit under the table.
- The Orchestrator sends this routing event to the relevant Edges in the customer enterprise.
- The Edges override the previous preference value to match the pinned order.
- The preference values that get assigned to pinned routes start from 1, 2, 3, and so on (the lowest values and thus the highest preferences), and this matches the order of the routes on the Overlay Flow Control page.
Note: For more information on pinning a route, consult Configure Subnets.
Tie-Breaking Scenarios for Dynamic Routes
What happens when an Edge receives the same prefix for two or more sources/neighbors?
A potential scenario in SD-WAN deployments is for the same prefix to be advertised from two different Edges or Partner Gateways. With VMware SD-WAN, if the subnets are within the same category (Edge, Hub, or Partner Gateway) and have the same preference value, the BGP attributes or OSPF metrics are first considered for route sorting.
If there is still a tie, SD-WAN uses the logical ID (which is derived from the Edge or Gateway's universally unique identifier (UUID)) of the next hop device to break the tie. The next hop device can be a Gateway or a Hub Edge depending on the type of Branch to Branch VPN being used. If the customer enterprise is using Branch to Branch via Gateway, the next hop is a Gateway, while a customer using Branch to Hub would have the next hop be a Hub Edge.
There is a final tie-breaker if multiple Gateways advertise the same exact route type and preference. This final tie-breaker prefers the oldest route learned. To ensure the routing outcome you want, you can either pin certain routes or configure the BGP attributes and costs to favor some routes over others.
