Configure BGP Over IPsec from Edge to Non SD-WAN Neighbors

The Non SD-WAN BGP Neighbors configuration is not applicable at Profile level. You can configure the NSD Neighbors only at the Edge level.

About this task:

BGP is used to establish the BGP neighborship over the IPSec tunnels to the Non SD-WAN Sites. Direct IPSec tunnels are used for establishing a secure communication between the SD-WAN Edge and the Non SD-WAN Destination (NSD). In previous releases, VMware supported NSD tunnels from the SD-WAN Edge with the ability to add NVS static routes. In the 4.3 release, this functionality is extended to support BGP over IPSec to the NSD endpoint for a route-based VPN.

VMware SD-WAN supports 4-Byte ASN BGP. See Configure BGP, for more information.

Note: The Azure vWAN Automation from Edge feature is not compatible with BGP over IPSec. This is because only static routes are supported when automating connectivity from an Edge to an Azure vWAN.

Use Cases

Use Case 1: BGP Over IPSec from an Edge to an Azure VPN

Each Azure VPN gateway allocates one set of public Virtual Public IPs (VIP) for a branch Edge to form IPSec tunnels. Similarly, Azure also allocates one internal private subnet and assigns one internal IP per VIP. This internal tunnel-ip (peer tunnel-ip) will be used for creating BGP peering with the Azure Gateway.

Azure has a restriction that the BGP peer IP (Edge's local tunnel-ip) shouldn't be in the same connected subnet or 169.x.x.x subnet, and therefore we need to support multi-hop BGP on the Edge. In BGP terminology, the local tunnel-ip maps to BGP source address and peer tunnel-ip maps to neighbor/peer address. We need to form a mesh of BGP connections - one per NSD tunnel so that the return traffic from the NVS could be load-balanced (flow-based) - design on the Azure Gateway side. In the below diagram for the physical Edge, we have two public WAN links and so four tunnels to an Azure Gateway. Each tunnel is associated with one BGP connection uniquely identified by the local tunnel_ip and remote peer tunnel_ip. On the Virtual Edge, the only difference is that we have one public WAN link and a maximum of two tunnels and two BGP sessions to the Azure Gateway.

Note: When an SD-WAN Edge is connected to the same Azure end-point using multiple WAN links, there is a maximum of two NSD-BGP neighbors that could be configured (since remote end has only two public_ips and two NSD-BGP peer_ips). Both NSD-BGP neighbors can be configured on the same link (primary/secondary tunnel), or tunnels on different links. If a customer attempts to configure more than two NSD-BGP neighbors and configure the same NSD-BGP peer_ip on more than one tunnel, the last configured BGP nbr_ip + local_ip would be on the SD-WAN Edge and Free Range Routing (FRR).

Use Case 2: BGP Over IPSec from Edge to AWS VPN/Transit Gateway

Unlike Azure, AWS VPN Gateway allocates one set of public VIPs per link to a branch Edge. The total sets of public IPs allocated to a branch Edge from an AWS Gateway will be equal to the number of Edge public WAN links that will connect to the AWS VPN Gateway. Similarly, a /30 internal/private subnet would be allocated per tunnel, which are used for BGP peering on that tunnel. These IPs could be manually overridden in AWS Gateway configuration to ensure they are unique across different availability zones.

Similar to the Azure use-case, the Edge will form a mesh of BGP connections - one per tunnel to the AWS gateway. This will allow load-balancing of the return traffic from the AWS VPN Gateway - design on the AWS side. In the diagram below, for the physical Edge, the AWS Gateway allocates one set of public IPs and one set of tunnel-ips (/30) for each Edge WAN link. There are a total of four tunnels, but terminate in different public IPs on the AWS Gateway and four BGP connections.

Use Case 3: Edge Connecting to Both AWS and Azure VPN Gateways (Hybrid Cloud)

One branch Edge could be connected to both Azure Gateway and AWS Gateway for redundancy purposes or some workloads/apps hosted in one cloud provider while other workloads/apps hosted in a different cloud provider. Regardless of the use-case, the Edge always establishes one BGP session per tunnel and propagates the routes between SD-WAN and IaaS. The diagram below is an example of one branch Edge connected to both Azure and AWS clouds.

Use Case 4: Hub Cluster Connecting to Azure/AWS Transit Gateways

The Hub cluster members can form IPSec tunnels to the Azure/AWS transit Gateways and leverage the transit Gateways as Layer 3 for routing traffic between different VPCs. Without the native BGP over IPSec functionality on Hub, the Hub needs to connect to an L3 router (Cisco CSR widely used here) using native BGP and the L3 router forming a mesh of BGP over IPSec tunnels with different VPCs. L3 router serves as a transit end-point between different VPCs. Usecase-1 (left diagram below): Use Hub as a transit node between different VPCs in different Availability Zones (AZ) so that one VPC can talk to another VPC. Usecase-2 (right diagram below): Connect all Hubs in the cluster directly to a cloud transit gateway and can use the cloud gateway as a PE(L3) router for routes distribution between cluster members. In both use-cases, without the support for BGP over IPSec on Hub, hub connects to an L3 router like CSR using native BGP and CSR peers with transit/VPC gateway using BGP over IPSec.

Use Case 5: Support Transit Functionality in Cloud Providers without Native Support

Some cloud providers like Google Cloud and AliCloud do not have native support for transit functionality (no transit Gateways), and with the support for BGP over IPSec, can rely on SD-WAN Edge/Hub deployed in the cloud to achieve the transit functionality between different VPCs/VNETs. Without the BGP over IPSec support, you must use an L3 router like CSR (solution (2)) to achieve the transit functionality.

Note: Prior to the 4.3 release, for customers who have reachability to the same NVS-Static destination via NVS-From-Gateway and NVS-From-Edge, the traffic from other branch SD-WAN Edges will prefer the path via NVS-Gateway. When customers upgrade their network to the 4.3 release or later, this traffic path from other branch- SD-WAN Edges will prefer the path via the NVS-Edge. Therefore, customers must update the NVS-Static-Destination’s metric of the NSD-Edge and the NSD-Gateway as per their traffic path preference.

Prerequisites:

Ensure that you have configured Configure Tunnel Between Branch and Non SD-WAN Destinations via Edge to configure BGP with NSD Neighbors.
The Local IP address from the Edge is required to configure BGP with NSD Neighbors.

Procedure

To enable BGP with Non SD-WAN neighbors:

In the SD-WAN service of the Enterprise Portal, click Configure.
From the left menu, select Edges. The Edges page displays.
Click an Edge from the list of available Edges.
Go to the Routing & NAT section in the UI and click the arrow next to BGP.
In the BGP area, check the Override check box and toggle the radio button from Off to On.
In the BGP Editor window, configure the following settings:
Enter the local Autonomous System Number (ASN) and then configure the following in the BGP Settings section.

Configure the BGP Settings, as described in the table below.


Option	Description
Router ID	Enter the global BGP router ID. If you do not specify any value, the ID is automatically assigned. If you have configured a loopback Interface for the Edge, the IP address of the loopback Interface will be assigned as the router ID.
Keep Alive	Enter the keep alive timer in seconds, which is the duration between the keep alive messages that are sent to the peer. The range is from 0 to 65535 seconds. The default value is 60 seconds.
Hold Timer	Enter the hold timer in seconds. When the keep alive message is not received for the specified time, the peer is considered as down. The range is from 0 to 65535 seconds. The default value is 180 seconds.
Uplink Community	Enter the community string to be treated as uplink routes. Uplink refers to link connected to the Provider Edge(PE). Inbound routes towards the Edge matching the specified community value will be treated as Uplink routes. The Hub/Edge is not considered as the owner for these routes. Enter the value in number format ranging from 1 to 4294967295 or in AA:NN format.
Enable Graceful Restart check box	Please note when selecting this check box: The local router does not support forwarding during the routing plane restart. This feature supports preserving forwarding and routing in case of peer restart.

Click +Add in the Filter List area to create one or more filters. These filters are applied to the neighbor to deny or change the attributes of the route. The same filter can be used for multiple neighbors.

In the appropriate text fields, set the rules for the filter, as described in the table below.


Option	Description
Filter Name	Enter a descriptive name for the BGP filter.
Match Type and Value	Choose the type of the routes to be matched with the filter: Prefix for IPv4 or IPv6: Choose to match with a prefix for IPv4 or IPv6 address and enter the corresponding prefix IP address in the Value field. Community: Choose to match with a community and enter the community string in the Value field.
Exact Match	The filter action is performed only when the BGP routes match exactly with the specified prefix or community string. By default, this option is enabled.
Action Type	Choose the action to be performed when the BGP routes match with the specified prefix or the community string. You can either permit or deny the traffic.
Action Set	When the BGP routes match the specified criteria, you can set to route the traffic to a network based on the attributes of the path. Select one of the following options from the drop-down list: None: The attributes of the matching routes remain the same. Local Preference: The matching traffic is routed to the path with the specified local preference. Community: The matching routes are filtered by the specified community string. You can also select the Community Additive check box to enable the additive option, which appends the community value to existing communities. Metric: The matching traffic is routed to the path with the specified metric value. AS-Path-Prepend: Allows pre-pending multiple entries of Autonomous System (AS) to a BGP route.

To add more matching rules to the filter, click the Plus (+) icon.
Click OK to create the filter.
The configured filters are displayed in the BGP Editor window.
Configure Underlay Neighbors for IPv4 and IPv6 addresses, as required. For more information, see Configure BGP from Edge to Underlay Neighbors with New Orchestrator UI.
Note: The maximum number of supported BGPv4 Match/Set rules is 512 (256 inbound, 256 outbound). Exceeding 512 total Match/Set rules is not supported and may cause performance issues, resulting in disruptions to the enterprise network.

In the NSD Neighbors section, configure the following settings, as described in the table below.


Option	Description
NSD Name	Select the NSD Name from the drop-down list. The NSDs already configured in the Branch to Non SD-WAN Destination via Edge area of the SASE Orchestrator are displayed in the drop-down list.
Link Name	Choose the name of the WAN link associated with the NSD neighbor.
Tunnel Type	Choose the tunnel type of the Peer as Primary or Secondary.
Neighbor IP	Enter the IP address of the NSD neighbor.
ASN	Enter the ASN for the NSD neighbor.
Inbound Filter	Select an Inbound filer from the drop-down list.
Outbound Filter	Select an Outbound filer from the drop-down list.
Additional Options – Click the view all link to configure the following additional settings:
Uplink	Used to flag the neighbor type to Uplink. Select this flag option if it is used as the WAN overlay towards MPLS. It will be used as the flag to determine whether the site will become a transit site (e.g. SD-WAN Hub), by propagating routes leant over a SD-WAN overlay to a WAN link toward MPLS. If you need to make it a transit site, select the Overlay Prefix Over Uplink check box in the Advanced Settings.
Local IP	Local IP is mandatory for configuring Non SD-WAN Neighbors. Local IP address is the equivalent of a loopback IP address. Enter an IP address that the BGP neighborships can use as the source IP address for the outgoing packets.
Max-hop	Enter the number of maximum hops to enable multi-hop for the BGP peers. For the 5.1 release and later, the range is from 2 to 255 and the default value is 2. Note: When upgrading to the 5.1 release, any max-hop value of 1 will automatically be updated to a max-hop value of 2. Note: This field is available only for eBGP neighbors, when the local ASN and the neighboring ASN are different. With iBGP, when both ASNs are the same, multi-hop is deactivated by default and this field is not configurable.
Allow AS	Select the check box to allow the BGP routes to be received and processed even if the Edge detects its own ASN in the AS-Path.
Default Route	The Default Route adds a network statement in the BGP configuration to advertise the default route to the neighbor.
Enable BFD	Enables subscription to existing BFD session for the BGP neighbor. Note: Single-hop BFD session is not supported for BGP over IPSec with NSD Neighbors. However, multi-hop BFD is supported. Local IP is mandatory for NSD-BGP sessions on the SD-WAN Edge. The SD-WAN Edge handles only the connected Interface IPs as a single-hop BFD.
Keep Alive	Enter the keep alive timer in seconds, which is the duration between the keep alive messages that are sent to the peer. The range is from 0 to 65535 seconds. The default value is 60 seconds.
Hold Timer	Enter the hold timer in seconds. When the keep alive message is not received for the specified time, the peer is considered as down. The range is from 0 to 65535 seconds. The default value is 180 seconds.
Connect	Enter the time interval to try a new TCP connection with the peer if it detects the TCP session is not passive. The default value is 120 seconds.
MD5 Auth	Select the check box to enable BGP MD5 authentication. This option is used in a legacy network or federal network, and it is common that BGP MD5 is used as a security guard for BGP peering.
MD5 Password	Enter a password for MD5 authentication. Note: Starting from the 4.5 release, the use of the special character "<" in the password is no longer supported. In cases where users have already used "<" in their passwords in previous releases, they must remove it to save any changes on the page.

Note: Over Multi-hop BGP, the system might learn routes that require recursive lookup. These routes have a next-hop IP which is not in a connected subnet, and do not have a valid exit interface. In this case, the routes must have the next-hop IP resolved using another route in the routing table that has an exit interface. When there is traffic for a destination that needs these routes to be looked up, routes requiring recursive lookup will get resolved to a connected Next Hop IP address and interface. Until the recursive resolution happens, the recursive routes point to an intermediate interface. For more information about Multi-hop BGP Routes, see the "Remote Diagnostic Tests on Edges" section in the VMware SD-WAN Troubleshooting Guide published at https://docs.vmware.com/en/VMware-SD-WAN/index.html.

Click Advanced to configure the following settings, as described in the table below.

Note: Advanced Settings are shared across both the underlay BGP neighbors and NSD BGP neighbors.


Option	Description
Overlay Prefix	Select the check box to redistribute the prefixes learned from the overlay.
Turn off AS-Path carry over	By default, this should be left unchecked. Select the check box to turn off AS-PATH Carry Over. In certain topologies, turning off AS-PATH Carry Over will influence the outbound AS-PATH to make the L3 routers prefer a path towards an Edge or a Hub. Warning: When the AS-PATH Carry Over is turned off, tune your network to avoid routing loops.
Connected Routes	Select the check box to redistribute all the connected Interface subnets.
OSPF	Select the check box to enable OSPF redistribute into BGP.
Set Metric	When you enable OSPF, enter the BGP metric for the redistributed OSPF routes. The default value is 20.
Default Route	Select the check box to redistribute the default route only when Edge learns the BGP routes through overlay or underlay. When you select the Default Route option, the Advertise option is available as Conditional.
Overlay Prefixes over Uplink	Select the check box to propagate routes learned from overlay to the neighbor with uplink flag.
Networks	Enter the network address that BGP will be advertising to the peers. Click the Plus (+) Icon to add more network addresses.

When you enable the Default Route option, the BGP routes are advertised based on the Default Route selection globally and per BGP neighbor, as shown in the following table.


Default Route Selection		Advertising Options
Global	Per BGP Neighbor	Advertising Options
Yes	Yes	The per BGP neighbor configuration overrides the global configuration and hence default route is always advertised to the BGP peer.
Yes	No	BGP redistributes the default route to its neighbor only when the Edge learns an explicit default route through the overlay or underlay network.
No	Yes	Default route is always advertised to the BGP peer.
No	No	The default route is not advertised to the BGP peer.

Click OK to save the configured filters and NSD Neighbors.
The BGP Settings section displays the configured settings.
Route Summarization
The Route Summarization feature is available in the 5.2 release, for an overview and use case of this functionality, see Route Summarization. For configuration details, follow the steps below.
Click +Add in the Route Summarization area. A new row is added to the Route Summarization area. See image below.
Under the Subnet column, enter the network range that you want to summarize in the A.B.C.D/M format and the IP subnet.
Under the AS Set column, click the Yes check box if applicable.
Under the Summary Only column, click the Yes check box to allow only the summarized route to be sent.
Add additional routes, if necessary, by clicking +Add. To Clone or Delete a route summarization, use the appropriate buttons, located next to +Add.
The BGP Settings section displays the BGP configuration settings.
Click Save Changes when complete to save the configuration.

You can also configure BGP from Edge to underlay neighbors. For more information, see Configure BGP from Edge to Underlay Neighbors with New Orchestrator UI.