Determine the number, networking, and high-availability configuration of the Tier-0 and Tier-1 gateways in NSX for VMware Cloud Foundation workload domains. Identify the BGP configuration for a single availability zone and two availability zones in the environment.

Table 1. Routing Direction Definitions

Routing Direction

Description

North-south

Traffic leaving or entering the NSX domain, for example, a virtual machine on an overlay network communicating with an end-user device on the corporate network.

East-west

Traffic that remains in the NSX domain, for example, two virtual machines on the same or different segments communicating with each other.

North-South Routing

The routing design considers different levels of routing in the environment, such as number and type of gateways in NSX, dynamic routing protocol, and others.

The following models for north-south traffic exist:

Table 2. Considerations for the Operating Model for North-South Service Routers

North-South Service Router Operating Model

Description

Benefits

Drawbacks

Active-Active

  • Bandwidth independent of the Tier-0 gateway failover model.

  • Configured in active-active equal-cost multi-path (ECMP) mode.

  • Failover takes approximately 2 seconds for virtual edges and is sub-second for bare-metal edges.

  • The active-active mode can support up to 8 NSX Edge nodes per northbound service router (SR).

  • Availability can be as high as N+7, with up to 8 active-active NSX Edge nodes.

  • Supports ECMP north-south routing on all nodes in the NSX Edge cluster.

  • Cannot provide stateful services, such as NAT.

Active-Standby

  • Bandwidth independent of the Tier-0 gateway failover model.

  • Failover takes approximately 2 seconds for virtual edges and is sub-second for bare-metal edges.

  • Can provide stateful services such as NAT.

  • The active-standby mode is limited to a single node.

  • Availability limited to N+1.

BGP North-South Routing for a Single or Multiple Availability Zones

For multiple availability zones, plan for failover of the NSX Edge nodes by configuring BGP so that traffic from the top of rack switches is directed to the first availability zone unless a failure in this zone occurs.

Figure 1. BGP North-South Routing for VMware Cloud Foundation Instances with a Single Availability Zone

The two-node NSX Edge cluster manages the Tier-0 and Tier-1 gateways. The routing protocol between Tier-0 gateway and the ToRs is BGP with ECMP.
Figure 2. BGP North-South Routing for VMware Cloud Foundation Instances with Multiple Availability Zones

Two availability zones, the NSX two-node edge cluster contains Tier-0 and Tier-1 gateways. The routing protocol between the Tier-0 gateway and ToRs in each zone is BGP with ECMP.

BGP North-South Routing Design for NSX Federation

In a routing design for an environment with VMware Cloud Foundation instances that use NSX Federation, you identify the instances that an SDN network must span and at which physical location ingress and egress traffic should occur.

Local egress allows traffic to leave any location which the network spans. The use of local-egress would require controlling local-ingress to prevent asymmetrical routing. This design does not use local-egress. Instead, this design uses a preferred and failover VMware Cloud Foundation instances for all networks.

Figure 3. BGP North-South Routing for VMware Cloud Foundation Instances with NSX Federation
Each VCF instance can have local segments connected to a local Tier-1 gateway that are only available in that instance as a result. One or more cross-instance segments can be connected to a stretched Tier-1 gateway allows that segment to be available in both instances. All Tier-1 gateways are connected to a stretched Tier-0 gateway.

Tier-0 Gateways with NSX Federation

In NSX Federation, a Tier-0 gateway can span multiple VMware Cloud Foundation instances.

Each VMware Cloud Foundation instance that is in the scope of a Tier-0 gateway can be configured as primary or secondary. A primary instance passes traffic for any other SDN service such as Tier-0 logical segments or Tier-1 gateways. A secondary instance routes traffic locally but does not egress traffic outside the SDN or advertise networks in the data center.

When deploying an additional VMware Cloud Foundation instance, the Tier-0 gateway in the first instance is extended to the new instance.

In this design, the Tier-0 gateway in each VMware Cloud Foundation instance is configured as primary. Although the Tier-0 gateway technically supports local-egress, the design does not recommend the use of local-egress. Ingress and egress traffic is controlled at the Tier-1 gateway level.

Each VMware Cloud Foundation instance has its own NSX Edge cluster with associated uplink VLANs for north-south traffic flow for that instance. The Tier-0 gateway in each instance peers with the top of rack switches over eBGP.

Figure 4. BGP Peering to Top of Rack Switches for VMware Cloud Foundation Instances with NSX Federation
The stretched Tier-0 spanning both edge clusters has edges located in each instance peering with the ToR swicthes in that instance

Tier-1 Gateways with NSX Federation

A Tier-1 gateway can span several VMware Cloud Foundation instances. As with a Tier-0 gateway, you can configure an instance's location as primary or secondary for the Tier-1 gateway. The gateway then passes ingress and egress traffic for the logical segments connected to it.

Any logical segments connected to the Tier-1 gateway follow the span of the Tier-1 gateway. If the Tier-1 gateway spans several VMware Cloud Foundation instances, any segments connected to that gateway become available in both instances.

Using a Tier-1 gateway enables more granular control on logical segments in the first and second VVMware Cloud Foundation instances. You use three Tier-1 gateways - one in each VMware Cloud Foundation instance for segments that are local to the instance, and one for segments which span the two instances.

Table 3. Location Configuration of the Tier-1 Gateways for Multiple VMware Cloud Foundation Instances

Tier-1 Gateway

First VMware Cloud Foundation Instance

Second VMware Cloud Foundation Instance

Ingress and Egress Traffic

Connected to both VMware Cloud Foundation instances

Primary

Secondary

First VMware Cloud Foundation instance

Second VMware Cloud Foundation instance

Local to the firstVMware Cloud Foundation instance

Primary

-

First VMware Cloud Foundation instance only

Local to the second VMware Cloud Foundation instance

-

Primary

Second VMware Cloud Foundation instance only

The Tier-1 gateway advertises its networks to the connected local-instance unit of the Tier-0 gateway. In the case of primary-secondary location configuration, the Tier-1 gateway advertises its networks only to the Tier-0 gateway unit in the location where the Tier-1 gateway is primary. The Tier-0 gateway unit then re-advertises those networks to the data center in the sites where that Tier-1 gateway is primary. During failover of the components in the first VMware Cloud Foundation instance, an administrator must manually set the Tier-1 gateway in the second VMware Cloud Foundation instance as primary. Then, networks become advertised through the Tier-1 gateway unit in the second instance.

In a Multiple Instance-Multiple Availability Zone topology, the same Tier-0 and Tier-1 gateway architecture applies. The ESXi transport nodes from the second availability zone are also attached to the Tier-1 gateway as per the BGP North-South Routing for VMware Cloud Foundation Instances with Multiple Availability Zones design.