You can configure the BGP per segment for a Profile or an Edge. This section provides steps on how to configure BGP with Non SD-WAN neighbors, which is supported from the 4.3 release.

About this task:

BGP is used to establish the BGP neighborship over the IPsec tunnels to the Non SD-WAN Sites. Direct IPsec tunnels are used for establishing a secure communication between the SD-WAN Edge and the Non SD-WAN Destination (NSD). In previous releases, VMware supported NSD tunnels from the SD-WAN Edge with the ability to add NVS static routes. In the 4.3 release, this functionality is extended to support BGP over IPSec to the NSD endpoint for a route-based VPN.

VMware also supports 4-Byte ASN BGP. See section titled, Configure BGP, for more information.

Note: For the 4.3.0 release, the Azure vWAN Automation from Edge feature is not compatible with BGP over IPsec. This is because only static routes are supported when automating connectivity from an Edge to an Azure vWAN.

Use Cases

Use Case 1: BGP Over IPsec from an Edge to an Azure VPN

Each Azure VPN gateway allocates one set of public Virtual Public IPs (VIP) for a branch Edge to form IPsec tunnels. Similarly, Azure also allocates one internal private subnet and assigns one internal IP per VIP. This internal tunnel-ip (peer tunnel-ip) will be used for creating BGP peering with the Azure Gateway.

Azure has a restriction that the BGP peer IP (Edge's local tunnel-ip) shouldn't be in the same connected subnet or 169.x.x.x subnet, and therefore we need to support multi-hop BGP on the Edge. In BGP terminology, the local tunnel-ip maps to BGP source address and peer tunnel-ip maps to neighbor/peer address. We need to form a mesh of BGP connections - one per NSD tunnel so that the return traffic from the NVS could be load-balanced (flow-based) - design on the Azure Gateway side. In the below diagram for the physical Edge, we have two public WAN links and so four tunnels to an Azure Gateway. Each tunnel is associated with one BGP connection uniquely identified by the local tunnel_ip and remote peer tunnel_ip. On the Virtual Edge, the only difference is that we have one public WAN link and a maximum of two tunnels and two BGP sessions to the Azure Gateway.

Note: When an SD-WAN Edge is connected to the same Azure end-point using multiple WAN links, there is a maximum of two NSD-BGP neighbors that could be configured (since remote end has only two public_ips and two NSD-BGP peer_ips). Both NSD-BGP neighbors can be configured on the same link (primary/secondary tunnel), or tunnels on different links. If a customer attempts to configure more than two NSD-BGP neighbors and configure the same NSD-BGP peer_ip on more than one tunnel, the last configured BGP nbr_ip + local_ip would be on the SD-WAN Edge and Free Range Routing (FRR).

Use Case 2: BGP Over IPsec from Edge to AWS VPN/Transit Gateway

Unlike Azure, AWS VPN Gateway allocates one set of public VIPs per link to a branch Edge. The total sets of public IPs allocated to a branch Edge from an AWS Gateway will be equal to the number of Edge public WAN links that will connect to the AWS VPN Gateway. Similarly, a /30 internal/private subnet would be allocated per tunnel, which are used for BGP peering on that tunnel. These IPs could be manually overridden in AWS Gateway configuration to ensure they are unique across different availability zones.

Similar to the Azure use-case, the Edge will form a mesh of BGP connections - one per tunnel to the AWS gateway. This will allow load-balancing of the return traffic from the AWS VPN Gateway - design on the AWS side. In the diagram below, for the physical Edge, the AWS Gateway allocates one set of public IPs and one set of tunnel-ips (/30) for each Edge WAN link. There are a total of four tunnels, but terminate in different public IPs on the AWS Gateway and four BGP connections.

Use Case 3: Edge Connecting to Both AWS and Azure VPN Gateways (Hybrid Cloud)

One branch Edge could be connected to both Azure Gateway and AWS Gateway for redundancy purposes or some workloads/apps hosted in one cloud provider while other workloads/apps hosted in a different cloud provider. Regardless of the use-case, the Edge always establishes one BGP session per tunnel and propagates the routes between SD-WAN and IaaS. The diagram below is an example of one branch Edge connected to both Azure and AWS clouds.

Use Case 4: Hub Cluster Connecting to Azure/AWS Transit Gateways

The Hub cluster members can form IPsec tunnels to the Azure/AWS transit Gateways and leverage the transit Gateways as Layer 3 for routing traffic between different VPCs. Without the native BGP over IPsec functionality on Hub, the Hub needs to connect to an L3 router (Cisco CSR widely used here) using native BGP and the L3 router forming a mesh of BGP over IPsec tunnels with different VPCs. L3 router serves as a transit end-point between different VPCs. Usecase-1 (left diagram below): Use Hub as a transit node between different VPCs in different Availability Zones (AZ) so that one VPC can talk to another VPC. Usecase-2 (right diagram below): Connect all Hubs in the cluster directly to a cloud transit gateway and can use the cloud gateway as a PE(L3) router for routes distribution between cluster members. In both use-cases, without the support for BGP over IPsec on Hub, hub connects to an L3 router like CSR using native BGP and CSR peers with transit/VPC gateway using BGP over IPsec.

Use Case 5: Support Transit Functionality in Cloud Providers without Native Support

Some cloud providers like Google Cloud and AliCloud do not have native support for transit functionality (no transit Gateways), and with the support for BGP over IPsec, can rely on SD-WAN Edge/Hub deployed in the cloud to achieve the transit functionality between different VPCs/VNETs. Without the BGP over IPsec support, you must use an L3 router like CSR (solution (2)) to achieve the transit functionality.

Note: Prior to the 4.3 release, for customers who have reachability to the same NVS-Static destination via NVS-From-Gateway and NVS-From-Edge, the traffic from other branch SD-WAN Edges will prefer the path via NVS-Gateway. When customers upgrade their network to the 4.3 release or later, this traffic path from other branch- SD-WAN Edges will prefer the path via the NVS-Edge. Therefore, customers must update the NVS-Static-Destination’s metric of the NSD-Edge and the NSD-Gateway as per their traffic path preference.

Prerequisites:

Procedure

To enable BGP with Non SD-WAN neighbors:

  1. In the Enterprise portal, click Configure Edge and select an SD-WAN Edge to configure.
  2. Click the Device tab.
  3. In the Device tab, scroll down to the BGP Settings section, click the slider to the ON position, and then click the Edit button.

  4. In the BGP Editor dialog, add filters.
    1. Click the Add Filter button to create one or more filters. Filters are applied to the neighbor to deny or change the attributes of the route. The same filter can be used for multiple neighbors, including both Neighbors and NSD Neighbors.

      The Create BGP Filter dialog displays.

    2. In the Create BGP Filter area, set the rules for the filter (see image below). See the table below for a description of the fields in the Create BGP Filter dialog.

      Option Description
      Filter Name Enter a descriptive name for the BGP filter.
      Match Type and Value Choose the type of the routes to be matched with the filter:
      • Prefix: Choose to match with a prefix and enter the prefix IP address in the Value field.
      • Community: Choose to match with a community and enter the community string in the Value field.
      Exact Match The filter action is performed only when the BGP routes match exactly with the specified prefix or community string. By default, this option is enabled.
      Action Type Choose the action to be performed when the BGP routes match with the specified prefix or the community string. You can either permit or deny the traffic.
      Set When the BGP routes match the specified criteria, you can set to route the traffic to a network based on the attributes of the path. Select one of the following options from the drop-down list:
      • None: The attributes of the matching routes remain the same.
      • Local Preference: The matching traffic is routed to the path with the specified local preference.
      • Community: The matching routes are filtered by the specified community string. You can also select the Community Additive checkbox to enable the additive option, which appends the community value to existing communities.
      • Metric: The matching traffic is routed to the path with the specified metric value.
      • AS-Path-Prepend: Allows prepending multiple entries of Autonomous System (AS) to a BGP route.
    3. To add more matching rules to the filter, click the Plus (+) icon.
    4. Click OK to create the filter.

      The configured filters are displayed in the BGP Editor window.

  5. In the BGP Editor dialog, indicate the Local ASN in the appropriate text box for the device to be configured.
  6. Configure Underlay Neighbors.
    Note: The steps to configure Underlay Neighbors for the 4.3 release are the same from previous releases. If you would like specific information about this step, see section titled, Configure BGP with Underlay Neighbors.
  7. Configure NSD Neighbors.
    Note: The 4.3 release supports Non SD-WAN (NSD) neighbors. All global settings will be shared by both neighbors, and the filter list can also be used for both types of neighbors. See the Prerequisites section above before configuring NSD Neighbors. See the steps below to configure NSD Neighbors, or see the table below for a description of the BGP Editor fields for underlay neighbors and NSD neighbors.

    1. Choose an NSD Neighbor from the NSD Name drop-down menu. This must have been configured earlier in the Branch to Non SD-WAN Destination via Edge area of the SD-WAN Orchestrator to display in the drop-down menu.
    2. In the Link Name drop-down menu, choose the link associated with the selected NSD Neighbor.
    3. Choose the Tunnel type (Primary or Secondary) from the Tunnel Type drop-down menu. If a Secondary-IP has not been configured for the NSD Neighbor, the drop-down menu will only display the Primary option and will not display the Secondary option.
    4. Enter the ASN for the selected NSD Neighbor.
    5. Select an Inbound filer from the drop-down list. (This feature is optional).
    6. Select an Outbound filer from the drop-down list. (This feature is optional).

      See the table below for a description of the fields for NSD Neighbors.

      Option Description
      Local ASN Enter the local Autonomous System Number (ASN)
      Neighbor IP Enter the IP address of the BGP neighbor
      NSD Name Choose the NSD Name that was configured in the Branch to Non SD-WAN Destination via Edge area of the SD-WAN Orchestrator.
      Link Name Choose the name of the WAN link which is associated with the NSD neighbor.
      ASN Enter the ASN for the NSD Neighbor.
      Tunnel Type Choose the tunnel type (Primary or Secondary) of the Peer.
      Inbound Filter Select an Inbound filer from the drop-down list.
      Outbound Filter Select an Outbound filer from the drop-down list.
      Additional Options – Click the view all link to configure the following additional settings:
      Uplink Used to flag the neighbor type to Uplink. Select this flag option if it is used as the WAN overlay towards MPLS. It will be used as the flag to determine whether the site will become a transit site (e.g. SD-WAN Hub), by propagating routes leant over a SD-WAN overlay to a WAN link toward MPLS. If you need to make it a transit site, also check "Overlay Prefix Over Uplink" in the Advanced Settings area.
      Local IP

      Local IP is mandatory for configuring Non SD-WAN Neighbors.

      Local IP address is the equivalent of a loopback IP address. Enter an IP address that the BGP neighborships can use as the source IP address for the outgoing packets.
      Max-hop Enter the number of maximum hops to enable multi-hop for the BGP peers. The range is from 1 to 255 and the default value is 1.
      Note: This field is available only for eBGP neighbors, when the local ASN and the neighboring ASN are different. With iBGP, when both ASNs are the same, multi-hop is disabled by default and this field is not configurable.
      Allow AS Select the checkbox to allow the BGP routes to be received and processed even if the Edge detects its own ASN in the AS-Path.
      Default Route The Default Route adds a network statement in the BGP configuration to advertise the default route to the neighbor.
      Enable BFD Enables subscription to existing BFD session for the BGP neighbor.
      Note: Single-hop BFD session is not supported for BGP over IPsec with NSD Neighbors. However, multi-hop BFD is supported. Local_ip is mandatory for NSD-BGP sessions on the SD-WAN Edge. The SD-WAN Edge handles only the connected interface IPs as a single-hop BFD.
      Keep Alive Enter the keepalive timer in seconds, which is the duration between the keepalive messages that are sent to the peer. The range is from 0 to 65535 seconds. The default value is 60 seconds.
      Hold Timer Enter the hold timer in seconds. When the keepalive message is not received for the specified time, the peer is considered as down. The range is from 0 to 65535 seconds. The default value is 180 seconds.
      Connect Enter the time interval to try a new TCP connection with the peer if it detects the TCP session is not passive. The default value is 120 seconds.
      MD5 Auth Select the checkbox to enable BGP MD5 authentication. This option is used in a legacy network or federal network, and it is common that BGP MD5 is used as a security guard for BGP peering.
      MD5 Password Enter a password for MD5 authentication. The password should not contain the character $ followed by numbers. For example $1, $123, password$123 are invalid inputs.
      Note: The MD5 authentication fails if the password has the character $ followed by numbers.
      Note: Over Multi-hop BGP, the system might learn routes that require recursive lookup. These routes have a next-hop IP which is not in a connected subnet, and do not have a valid exit interface. In this case, the routes must have the next-hop IP resolved using another route in the routing table that has an exit interface. When there is traffic for a destination that needs these routes to be looked up, routes requiring recursive lookup will get resolved to a connected Next Hop IP address and interface. Until the recursive resolution happens, the recursive routes point to an intermediate interface. For more information, see Multi-hop BGP Routes.
    7. To configure additional options for NSD Neighbors, click the Advanced Settings button.
      Note: Advanced Settings are shared across both regular underlay BGP neighbors and NSD-BGP neighbors. See the table below for a description of all the fields in the Advanced Settings area for NSD Neighbors. There are no changes in the 4.3 release for these settings for NSD Neighbors.
    8. Additional NSD Neighbors can be created or cloned by clicking the plus icon. To delete a created or cloned NSD Neighbors, click the minus icon.
      Option Description
      Router ID Enter the global BGP router ID. If you do not specify any value, the ID is automatically assigned.
      Keep Alive Enter the keepalive timer in seconds, which is the duration between the keepalive messages that are sent to the peer. The range is from 0 to 65535 seconds. The default value is 60 seconds.
      Hold Timer Enter the hold timer in seconds. When the keepalive message is not received for the specified time, the peer is considered as down. The range is from 0 to 65535 seconds. The default value is 180 seconds.
      Uplink Community

      Enter the community string to be treated as uplink routes.

      Uplink refers to link connected to the Provider Edge (PE). Inbound routes towards the Edge matching the specified community value will be treated as Uplink routes. The Hub/Edge is not considered as the owner for these routes.

      Enter the value in number format ranging from 1 to 4294967295 or in AA:NN format.
      Overlay Prefix Select the checkbox to redistribute the prefixes learned from the overlay.
      Disable AS-Path carry over By default, this should be left unchecked. Select the checkbox to disable AS-PATH Carry Over. In certain topologies, disabling AS-PATH Carry Over will influence the outbound AS-PATH to make the L3 routers prefer a path towards an Edge or a Hub.
      Warning: When the AS-PATH Carry Over is disabled, tune your network to avoid routing loops.
      Connected Routes Select the checkbox to redistribute all the connected Interface subnets.
      OSPF Select the checkbox to enable OSPF redistribute into BGP.
      Set Metric When you enable OSPF, enter the BGP metric for the redistributed OSPF routes. The default value is 20.
      Default Route

      Select the checkbox to redistribute the default route only when Edge learns the BGP routes through overlay or underlay.

      When you select the Default Route option, the Advertise option is available as Conditional.
      Overlay Prefixes over Uplink Select the checkbox to propagate routes learned from overlay to the neighbor with uplink flag.
      Networks Enter the network address that BGP will be advertising to the peers. Click the Plus (+) Icon to add more network addresses.
      When you enable the Default Route option, the BGP routes are advertised based on the Default Route selection globally and per BGP neighbor, as shown in the following table.
      Default Route Selection Advertising Options
      Global Per BGP Neighbor
      Yes Yes The per BGP neighbor configuration overrides the global configuration and hence default route is always advertised to the BGP peer.
      Yes No BGP redistributes the default route to its neighbor only when the Edge learns an explicit default route through the overlay or underlay network.
      No Yes Default route is always advertised to the BGP peer.
      No No The default route is not advertised to the BGP peer.
  8. Click OK to save the configured filters and NSD Neighbors and close the BGP Editor.

    The BGP Settings section displays the BGP configuration settings.

  9. Click Save Changes in the Device screen to save the configuration.

When you configure BGP settings for a profile, Non SD-WAN BGP Neighbors configuration is not applicable at Profile level. It can be configured only at Edge-override level. If required, you can override the configuration for a specific Edge. See section titled, Configure BGP for more information.