Routing for the Management Domain for Multiple VMware Cloud Foundation Instances

Design the routing configuration in NSX-T Data Center for multiple VMware Cloud Foundation instances to support network span for management applications that require resilient connectivity at multiple locations and to enable granular control of traffic from and to each VMware Cloud Foundation instance.

North-South Routing

In a routing design for an environment with multiple VMware Cloud Foundation instances, you identify VMware Cloud Foundation instances that an SDN network must span and whose physical location must let ingress and egress traffic.

Network traffic that is entering or leaving the SDN networks with a preference to the physical location of a VMware Cloud Foundation instance and failover is a key design choice for a multi-site deployment. This design does not use local egress, that is, traffic leaving and entering any location which the network spans. Instead, this design uses a preferred and failover VMware Cloud Foundation instance for all networks. The complexity of local-egress, that is, controlling local-ingress to prevent asymmetrical routing, is not necessary for this design.

In this design, an NSX component can be primary for one or more VMware Cloud Foundation instances. During a failure at the location of a VMware Cloud Foundation instance, you must sett a network component as primary in another VMware Cloud Foundation instance manually.

Figure 1. North-South Routing for Multiple VMware Cloud Foundation Instances

Tier-0 Gateways

In NSX Federation, a Tier-0 gateway can span multiple VMware Cloud Foundation instances. Each VMware Cloud Foundation instance contains a logical unit of the Tier-0 gateway which is assigned to the edge cluster in that instance and is configured to interface with the top of rack switches in the data center.

Each VMware Cloud Foundation instance that is in the scope of a Tier-0 gateway can be configured as primary or secondary. A primary instance passes traffic for any other SDN service such as Tier-0 logical segments or Tier-1 gateways. A secondary instance routes traffic locally but does not egress traffic outside the SDN or advertise networks in the data center.

When deploying an additional VMware Cloud Foundation instance, the Tier-0 gateway in the first instance is extended to the new instance.

In this design, the Tier-0 gateway in each VMware Cloud Foundation instance is configured as primary. Although the Tier-0 gateway technically supports local-egress, the design does not recommend the use of local-egress. Ingress and egress traffic is controlled at the Tier-1 gateway level.

Table 1. Design Decisions on the Tier-0 Gateway Configuration for Multiple VMware Cloud Foundation Instances

Decision ID	Design Decision	Design Justification	Design Implication
VCF-MGMT-NSX-SDN-FED-001	Extend the management domain active-active Tier-0 gateway to the second VMware Cloud Foundation instance.	Supports ECMP north-south routing on all nodes in the NSX Edge cluster. Enables support for cross-instance Tier-1 gateways and cross-instance network segments.	Active-active Tier-0 gateways cannot provide stateful services such as NAT.
VCF-MGMT-NSX-SDN-FED-002	Set the Tier-0 gateway as primary for all VMware Cloud Foundation instances.	In NSX Federation, a Tier-0 gateway lets egress traffic from connected Tier-1 gateways only in its primary locations. Local ingress and egress traffic is controlled independently at the Tier-1 level. No segments are provisioned directly to the Tier-0 gateway. A mixture of network spans (local to a VMware Cloud Foundation instance or spanning multiple instances) is enabled without requiring additional Tier-0 gateways and hence edge nodes. If a failure in a VMware Cloud Foundation instance occurs, the local-instance networking in the other instance remains available without manual intervention.	None.

Each VMware Cloud Foundation instance has its own NSX Edge cluster with associated uplink VLANs for north-south traffic flow for that instance. The Tier-0 gateway unit in each instance peers with the top of rack switches over eBGP.

Figure 2. BGP Peering to Top of Rack Switches

Table 2. Design Decisions on Dynamic Routing for Multiple VMware Cloud Foundation Instances

Decision ID	Design Decision	Design Justification	Design Implication
VCF-MGMT-NSX-SDN-FED-003	From the global Tier-0 gateway, establish BGP neighbor peering to the ToR switches connected to the second VMware Cloud Foundation instance.	Enables the learning and advertising of routes in the second VMware Cloud Foundation instance. Facilitates a potential automated failover of networks from the first to the second VMware Cloud Foundation instance.	None.

Tier-1 Gateways

A Tier-1 gateway can span several VMware Cloud Foundation instances. As with a Tier-0 gateway, you can configure an instance's location as primary or secondary for the Tier-1 gateway. The gateway then passes ingress and egress traffic for the logical segments connected to it.

Any logical segments connected to the Tier-1 gateway follow the span of the Tier-1 gateway. If the Tier-1 gateway spans several VMware Cloud Foundation instances, any segments connected to that gateway become available in both instances.

Using a Tier-1 gateway enables more granular control on logical segments in the first and second VMware Cloud Foundation instances. You use three Tier-1 gateways - one in each VMware Cloud Foundation instance for segments that are local to the instance, and one for segments which span the two instances.

Table 3. Location Configuration of the Tier-1 Gateways for Multiple VMware Cloud Foundation Instances

Tier-1 Gateway	First VMware Cloud Foundation Instance	Second VMware Cloud Foundation Instance	Ingress and Egress Traffic
Connected to both VMware Cloud Foundation instances	Primary	Secondary	Primary. First VMware Cloud Foundation instance Failover. Second VMware Cloud Foundation instance
Local to the first VMware Cloud Foundation instance	Primary	-	First VMware Cloud Foundation instance only
Local to the second VMware Cloud Foundation instance	-	Primary	Second VMware Cloud Foundation instance only

The Tier-1 gateway advertises its networks to the connected local-instance unit of the Tier-0 gateway. In the case of primary-secondary location configuration, the Tier-1 gateway advertises its networks only to the Tier-0 gateway unit in the the location where the Tier-1 gateway is primary. The Tier-0 gateway unit then re-advertises those networks to the data center in the sites where that Tier-1 gateway is primary. During failover of the components in the first VMware Cloud Foundation instance, the IT administrator must manually set the Tier-1 gateway in the second VMware Cloud Foundation instance as primary. Then, networks become advertised through the Tier-1 gateway unit in the second instance.

Table 4. Design Decisions on the Tier-1 Gateway Configuration for Multiple VMware Cloud Foundation Instances

Decision ID	Design Decision	Design Justification	Design Implication
VCF-MGMT-NSX-SDN-FED-004	Use Tier-1 gateways to control the span of networks and ingress and egress traffic in the VMware Cloud Foundation instances.	Enables a mixture of network spans (isolated to a VMware Cloud Foundation instance or spanning multiple instances) without requiring additional Tier-0 gateways and hence edge nodes.	To control location span, a Tier-1 gateway must be assigned to an edge cluster and hence has the Tier-1 SR component. East-west traffic between Tier-1 gateways with SRs need to physically traverse an edge node.
VCF-MGMT-NSX-SDN-FED-005	Use a global cross-instance Tier-1 gateway and connect it to the Tier-0 gateway for cross-instance networking.	Enables network span between the VMware Cloud Foundation instances because NSX network segments follow the span of the gateway they are attached to. Creates a two-tier routing architecture.	None.
VCF-MGMT-NSX-SDN-FED-006	Assign the NSX Edge cluster in each VMware Cloud Foundation instance to the global cross-instance Tier-1 gateway. Set the first VMware Cloud Foundation instance as primary and the second instance as secondary.	Enables cross-instance network span between the first and second VMware Cloud Foundation instances. Enables deterministic ingress and egress traffic for the cross-instance network. If a VMware Cloud Foundation instance failure occurs, enables deterministic failover of the Tier-1 traffic flow. During the recovery of the inaccessible VMware Cloud Foundation instance, enables deterministic failback of the Tier-1 traffic flow, preventing unintended asymmetrical routing. Eliminates the need to use BGP attributes in the first and second VMware Cloud Foundation instances to influence location preference and failover.	You must manually fail over and fail back the cross-instance network from the standby NSX Global Manager.
VCF-MGMT-NSX-SDN-FED-007	Allocate a Tier-1 gateway in each instance for instance-specific networks and connect it to the cross-instance Tier-0 gateway.	Creates a two-tier routing architecture. Enables local-instance networks that are not to span between the VMware Cloud Foundation instances. Guarantees that local-instance networks remain available if a failure occurs in another VMware Cloud Foundation instance.	None.
VCF-MGMT-NSX-SDN-FED-008	Assign the NSX Edge cluster in each VMware Cloud Foundation instance to the instance-specific Tier-1 gateway for that VMware Cloud Foundation instance.	Enables instance-specific networks to be isolated to their specific instances. Enables deterministic flow of ingress and egress traffic for the instance-specific networks.	You can use the service router that is created for the Tier-1 gateway for networking services. However, such configuration is not required for network connectivity.
VCF-MGMT-NSX-SDN-FED-009	Set each local-instance Tier-1 gateway only as primary in the home instance. Avoid setting the gateway as secondary in the other instances.	Prevents the need to use BGP attributes in primary and secondary instances to influence the instance ingress-egress preference.	None.