BGP Routing Design for VMware Cloud Foundation

Determine the number, networking, and high-availability configuration of the Tier-0 and Tier-1 gateways in NSX for VMware Cloud Foundation workload domains. Identify the BGP configuration for a single availability zone and two availability zones in the environment.

Table 1. Routing Direction Definitions
Routing Direction	Description
North-south	Traffic leaving or entering the NSX domain, for example, a virtual machine on an overlay network communicating with an end-user device on the corporate network.
East-west	Traffic that remains in the NSX domain, for example, two virtual machines on the same or different segments communicating with each other.

North-South Routing

The routing design considers different levels of routing in the environment, such as number and type of gateways in NSX, dynamic routing protocol, and others.

The following models for north-south traffic exist:

Table 2. Considerations for the Operating Model for North-South Service Routers
North-South Service Router Operating Model	Description	Benefits	Drawbacks
Active-Active	Bandwidth independent of the Tier-0 gateway failover model. Configured in active-active equal-cost multi-path (ECMP) mode. Failover takes approximately 2 seconds for virtual edges and is sub-second for bare-metal edges.	The active-active mode can support up to 8 NSX Edge nodes per northbound service router (SR). Availability can be as high as N+7, with up to 8 active-active NSX Edge nodes. Supports ECMP north-south routing on all nodes in the NSX Edge cluster.	Cannot provide stateful services, such as NAT.
Active-Standby	Bandwidth independent of the Tier-0 gateway failover model. Failover takes approximately 2 seconds for virtual edges and is sub-second for bare-metal edges.	Can provide stateful services such as NAT.	The active-standby mode is limited to a single node. Availability limited to N+1.

BGP North-South Routing for a Single or Multiple Availability Zones

For multiple availability zones, plan for failover of the NSX Edge nodes by configuring BGP so that traffic from the top of rack switches is directed to the first availability zone unless a failure in this zone occurs.

The two-node NSX Edge cluster manages the Tier-0 and Tier-1 gateways. The routing protocol between Tier-0 gateway and the ToRs is BGP with ECMP. — Figure 1. BGP North-South Routing for VMware Cloud Foundation Instances with a Single Availability Zone

Two availability zones, the NSX two-node edge cluster contains Tier-0 and Tier-1 gateways. The routing protocol between the Tier-0 gateway and ToRs in each zone is BGP with ECMP. — Figure 2. BGP North-South Routing for VMware Cloud Foundation Instances with Multiple Availability Zones

BGP North-South Routing Design for NSX Federation

In a routing design for an environment with VMware Cloud Foundation instances that use NSX Federation, you identify the instances that an SDN network must span and at which physical location ingress and egress traffic should occur.

Local egress allows traffic to leave any location which the network spans. The use of local-egress would require controlling local-ingress to prevent asymmetrical routing. This design does not use local-egress. Instead, this design uses a preferred and failover VMware Cloud Foundation instances for all networks.

Each VCF instance can have local segments connected to a local Tier-1 gateway that are only available in that instance as a result. One or more cross-instance segments can be connected to a stretched Tier-1 gateway allows that segment to be available in both instances. All Tier-1 gateways are connected to a stretched Tier-0 gateway. — Figure 3. BGP North-South Routing for VMware Cloud Foundation Instances with NSX Federation

Tier-0 Gateways with NSX Federation

In NSX Federation, a Tier-0 gateway can span multiple VMware Cloud Foundation instances.

Each VMware Cloud Foundation instance that is in the scope of a Tier-0 gateway can be configured as primary or secondary. A primary instance passes traffic for any other SDN service such as Tier-0 logical segments or Tier-1 gateways. A secondary instance routes traffic locally but does not egress traffic outside the SDN or advertise networks in the data center.

When deploying an additional VMware Cloud Foundation instance, the Tier-0 gateway in the first instance is extended to the new instance.

In this design, the Tier-0 gateway in each VMware Cloud Foundation instance is configured as primary. Although the Tier-0 gateway technically supports local-egress, the design does not recommend the use of local-egress. Ingress and egress traffic is controlled at the Tier-1 gateway level.

Each VMware Cloud Foundation instance has its own NSX Edge cluster with associated uplink VLANs for north-south traffic flow for that instance. The Tier-0 gateway in each instance peers with the top of rack switches over eBGP.

The stretched Tier-0 spanning both edge clusters has edges located in each instance peering with the ToR swicthes in that instance — Figure 4. BGP Peering to Top of Rack Switches for VMware Cloud Foundation Instances with NSX Federation

Tier-1 Gateways with NSX Federation

A Tier-1 gateway can span several VMware Cloud Foundation instances. As with a Tier-0 gateway, you can configure an instance's location as primary or secondary for the Tier-1 gateway. The gateway then passes ingress and egress traffic for the logical segments connected to it.

Any logical segments connected to the Tier-1 gateway follow the span of the Tier-1 gateway. If the Tier-1 gateway spans several VMware Cloud Foundation instances, any segments connected to that gateway become available in both instances.

Using a Tier-1 gateway enables more granular control on logical segments in the first and second VVMware Cloud Foundation instances. You use three Tier-1 gateways - one in each VMware Cloud Foundation instance for segments that are local to the instance, and one for segments which span the two instances.

Table 3. Location Configuration of the Tier-1 Gateways for Multiple VMware Cloud Foundation Instances
Tier-1 Gateway	First VMware Cloud Foundation Instance	Second VMware Cloud Foundation Instance	Ingress and Egress Traffic
Connected to both VMware Cloud Foundation instances	Primary	Secondary	First VMware Cloud Foundation instance Second VMware Cloud Foundation instance
Local to the firstVMware Cloud Foundation instance	Primary	-	First VMware Cloud Foundation instance only
Local to the second VMware Cloud Foundation instance	-	Primary	Second VMware Cloud Foundation instance only

The Tier-1 gateway advertises its networks to the connected local-instance unit of the Tier-0 gateway. In the case of primary-secondary location configuration, the Tier-1 gateway advertises its networks only to the Tier-0 gateway unit in the location where the Tier-1 gateway is primary. The Tier-0 gateway unit then re-advertises those networks to the data center in the sites where that Tier-1 gateway is primary. During failover of the components in the first VMware Cloud Foundation instance, an administrator must manually set the Tier-1 gateway in the second VMware Cloud Foundation instance as primary. Then, networks become advertised through the Tier-1 gateway unit in the second instance.

In a Multiple Instance-Multiple Availability Zone topology, the same Tier-0 and Tier-1 gateway architecture applies. The ESXi transport nodes from the second availability zone are also attached to the Tier-1 gateway as per the BGP North-South Routing for VMware Cloud Foundation Instances with Multiple Availability Zones design.