Use this list of requirements and recommendations for reference related to the configuration of NSX in an environment with a single or multiple VMware Cloud Foundation instances. The design also considers if an instance contains a single or multiple availability zones.

For full design details, see NSX Design for VMware Cloud Foundation.

NSX Manager Design Elements

Table 1. NSX Manager Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-LM-REQD-CFG-001

Place the appliances of the NSX Manager cluster on the VM management network in the management domain.

  • Simplifies IP addressing for management VMs by using the same VLAN and subnet.

  • Provides simplified secure access to management VMs in the same VLAN network.

None.

VCF-NSX-LM-REQD-CFG-002

Deploy three NSX Manager nodes in the default vSphere cluster in the management domain for configuring and managing the network services for the workload domain.

Supports high availability of the NSX manager cluster.

You must have sufficient resources in the default cluster of the management domain to run three NSX Manager nodes.
Table 2. NSX Manager Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-LM-RCMD-CFG-001

Deploy appropriately sized nodes in the NSX Manager cluster for the workload domain.

Ensures resource availability and usage efficiency per workload domain.

The default size for a management domain is Medium, and for VI workload domains is Large.

VCF-NSX-LM-RCMD-CFG-002

Create a virtual IP (VIP) address for the NSX Manager cluster for the workload domain.

Provides high availability of the user interface and API of NSX Manager.

  • The VIP address feature provides high availability only. It does not load-balance requests across the cluster.

  • When using the VIP address feature, all NSX Manager nodes must be deployed on the same Layer 2 network.

VCF-NSX-LM-RCMD-CFG-003

Apply VM-VM anti-affinity rules in vSphere Distributed Resource Scheduler (vSphere DRS) to the NSX Manager appliances.

Keeps the NSX Manager appliances running on different ESXi hosts for high availability.

You must allocate at least four physical hosts so that the three NSX Manager appliances continue running if an ESXi host failure occurs.

VCF-NSX-LM-RCMD-CFG-004

In vSphere HA, set the restart priority policy for each NSX Manager appliance to high.

  • NSX Manager implements the control plane for virtual network segments. vSphere HA restarts the NSX Manager appliances first so that other virtual machines that are being powered on or migrated by using vSphere vMotion while the control plane is offline lose connectivity only until the control plane quorum is re-established.

  • Setting the restart priority to high reserves the highest priority for flexibility for adding services that must be started before NSX Manager.

If the restart priority for another management appliance is set to highest, the connectivity delay for management appliances will be longer.
Table 3. NSX Manager Design Recommendations for Stretched Clusters in VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-LM-RCMD-CFG-006

Add the NSX Manager appliances to the virtual machine group for the first availability zone.

Ensures that, by default, the NSX Manager appliances are powered on a host in the primary availability zone.

None.

NSX Global Manager Design Elements

You must perform configuration tasks manually for the design requirements and recommendations for NSX Global Manager.

Table 4. NSX Global Manager Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-GM-REQD-CFG-001

Place the appliances of the NSX Global Manager cluster on the Management VM network in each VMware Cloud Foundation instance.

  • Simplifies IP addressing for management VMs.

  • Provides simplified secure access to all management VMs in the same VLAN network.

None.
Table 5. NSX Global Manager Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-GM-RCMD-CFG-001

Deploy three NSX Global Manager nodes for the workload domain to support NSX Federation across VMware Cloud Foundation instances.

Provides high availability for the NSX Global Manager cluster.

You must have sufficient resources in the default cluster of the management domain to run three NSX Global Manager nodes.

VCF-NSX-GM-RCMD-CFG-002

Deploy appropriately sized nodes in the NSX Global Manager cluster for the workload domain.

Check VMware Configuration Maximums to select the right NSX Global Managers form factor for your scale needs.

Ensures resource availability and usage efficiency per workload domain.

Incorrectly sized NSX Global Managers might impact the ability to scale according to the requirements.

VCF-NSX-GM-RCMD-CFG-003

Create a virtual IP (VIP) address for the NSX Global Manager cluster for the workload domain.

Provides high availability of the user interface and API of NSX Global Manager.

  • The VIP address feature provides high availability only. It does not load-balance requests across the cluster.

  • When using the VIP address feature, all NSX Global Manager nodes must be deployed on the same Layer 2 network.

VCF-NSX-GM-RCMD-CFG-004

Apply VM-VM anti-affinity rules in vSphere DRS to the NSX Global Manager appliances.

Keeps the NSX Global Manager appliances running on different ESXi hosts for high availability.

You must allocate at least four physical hosts so that the three NSX Manager appliances continue running if an ESXi host failure occurs.

VCF-NSX-GM-RCMD-CFG-005

In vSphere HA, set the restart priority policy for each NSX Global Manager appliance to medium.

  • NSX Global Manager implements the management plane for global segments and firewalls.

    NSX Global Manager is not required for control plane and data plane connectivity.

  • Setting the restart priority to medium reserves the high priority for services that impact the NSX control or data planes.

  • Management of NSX global components will be unavailable until the NSX Global Manager virtual machines restart.

  • The NSX Global Manager cluster is deployed in the management domain, where the total number of virtual machines is limited and where it competes with other management components for restart priority.

VCF-NSX-GM-RCMD-CFG-006

Deploy an additional NSX Global Manager Cluster in the second VMware Cloud Foundation instance.

Enables recoverability of NSX Global Manager in the second VMware Cloud Foundation instance if a failure in the first VMware Cloud Foundation instance occurs.

Requires additional NSX Global Manager nodes in the second VMware Cloud Foundation instance.

VCF-NSX-GM-RCMD-CFG-007
Set the NSX Global Manager cluster in the second VMware Cloud Foundation instance as standby for the workload domain.

Enables recoverability of NSX Global Manager in the second VMware Cloud Foundation instance if a failure in the first instance occurs.
Must be done manually.

VCF-NSX-GM-RCMD-SEC-001

Establish an operational practice to capture and update the thumbprint of the NSX Local Manager certificate on NSX Global Manager every time the certificate is updated by using SDDC Manager.

Ensures secured connectivity between the NSX Manager instances.

Each certificate has its own unique thumbprint. NSX Global Manager stores the unique thumbprint of the NSX Local Manager instances for enhanced security.

If an authentication failure between NSX Global Manager and NSX Local Manager occurs, objects that are created from NSX Global Manager will not be propagated on to the SDN.

The administrator must establish and follow an operational practice by using a runbook or automated process to ensure that the thumbprint is up-to-date.
Table 6. NSX Global Manager Design Recommendations for Stretched Clusters in VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-GM-RCMD-CFG-008

Add the NSX Global Manager appliances to the virtual machine group for the first availability zone.

Ensures that, by default, the NSX Global Manager appliances are powered on a host in the primary availability zone.
Done automatically by VMware Cloud Foundation when stretching a cluster.

NSX Edge Design Elements

Table 7. NSX Edge Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-EDGE-REQD-CFG-001

Connect the management interface of each NSX Edge node to the VM management network.

Provides connection from the NSX Manager cluster to the NSX Edge.

None.

VCF-NSX-EDGE-REQD-CFG-002

  • Use two physical NICs as uplinks for the NSX Edge appliances, for example, vmnic0 and vmnic1.

  • Connect the fp-eth0 interface of each NSX Edge appliance to a VLAN trunk port group pinned to the one physical NIC (vmnic0) of the host, with the ability to failover to another physical NIC (vmnic1).

  • Connect the fp-eth1 interface of each NSX Edge appliance to a VLAN trunk port group pinned to a physical NIC of the host (vmnic1), with the ability to failover to first physical NIC (vmnic0).

  • Leave the additional fp-eth2 and fp-eth3 interfaces of each NSX Edge appliance unused.

  • Because VLAN trunk port groups pass traffic for all VLANs, VLAN tagging can occur in the NSX Edge node itself for easy post-deployment configuration.

  • By using two separate VLAN trunk port groups, you can direct traffic from the edge node to a particular host network interface and top of rack switch as needed.

  • In the event of failure of the top of rack switch, the VLAN trunk port group will failover to the other physical NIC and to ensure both fp-eth0 and fp-eth1 are available.

None.

VCF-NSX-EDGE-REQD-CFG-003

Use a dedicated VLAN for edge overlay that is different from the host overlay VLAN.

A dedicated edge overlay network provides support for edge mobility in support of advanced deployments such as multiple availability zones or multi-rack clusters.

  • You must have routing between the VLANs for edge overlay and host overlay.

  • You must allocate another VLAN in the data center infrastructure for edge overlay.

VCF-NSX-EDGE-REQD-CFG-004

Create one uplink profile for the edge nodes with three teaming policies.

  • Default teaming policy of load balance source with both active uplinks uplink1 and uplink2.

  • Named teaming policy of failover order with a single active uplink uplink1 without standby uplinks.

  • Named teaming policy of failover order with a single active uplink uplink2 without standby uplinks.

  • An NSX Edge node that uses a single N-VDS can have only one uplink profile.

  • For increased resiliency and performance, supports the concurrent use of both edge uplinks through both physical NICs on the ESXi hosts.

  • The default teaming policy increases overlay performance and availability by using multiple TEPs, and balancing of overlay traffic.

  • By using named teaming policies, you can connect an edge uplink to a specific host uplink and from there to a specific top of rack switch in the data center.

  • Enables ECMP because the NSX Edge nodes can uplink to the physical network over two different VLANs.

None.
Table 8. NSX Multi-Rack Edge Availability Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-MR-EDGE-REQD-ENV-001

Place at least one edge VM per edge cluster onto each vSphere cluster, that is in a separate rack.

Provides availability if a failure of a rack or a vSphere cluster occurs.

Additional compute resources required.

VCF-NSX-MR-EDGE-REQD-CFG-001

Connect the management interface of each NSX Edge node in each rack to the VM management network that rack.

Provides connection from the NSX Manager cluster to the NSX Edge.

None.

VCF-NSX-MR-EDGE-REQD-CFG-002

Use a dedicated IP pool for the edge overlay network for each rack used for the deployment of a multi-rack NSX Edge cluster.

A dedicated edge overlay network and subnet per rack provides support for a Layer 3 leaf-and-spine physical network design with the Layer 2 boundary at the leaf nodes.

You must manage additional IP pools for each additional rack.

VCF-NSX-MR-EDGE-REQD-CFG-003

Create an edge uplink profile for each rack.

Enables having a dedicated edge overlay VLAN per rack to provide support for a Layer 3 leaf-and-spine physical network design with the Layer 2 boundary at the leaf nodes.

 None.
Table 9. NSX Edge Design Requirements for NSX Federation in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-EDGE-REQD-CFG-005

Allocate a separate VLAN for edge RTEP overlay that is different from the edge overlay VLAN.

The RTEP network must be on a VLAN that is different from the edge overlay VLAN. This is an NSX requirement that provides support for configuring different MTU size per network.

You must allocate another VLAN in the data center infrastructure.
Table 10. NSX Edge Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implications

VCF-NSX-EDGE-RCMD-CFG-001

Use appropriately sized NSX Edge virtual appliances.

Ensures resource availability and usage efficiency per workload domain.

You must provide sufficient compute resources to support the chosen appliance size.

VCF-NSX-EDGE-RCMD-CFG-002

Deploy the NSX Edge virtual appliances to the default vSphere cluster of the workload domain, sharing the cluster between the workloads and the edge appliances.

Simplifies the configuration and minimizes the number of ESXi hosts required for initial deployment.

Workloads and NSX Edges share the same compute resources.

VCF-NSX-EDGE-RCMD-CFG-003

Deploy two NSX Edge appliances in an edge cluster in the default vSphere cluster of the workload domain.

Creates the minimum size NSX Edge cluster while satisfying the requirements for availability.

For a VI workload domain, additional edge appliances might be required to satisfy increased bandwidth requirements.

VCF-NSX-EDGE-RCMD-CFG-004

Apply VM-VM anti-affinity rules for vSphere DRS to the virtual machines of the NSX Edge cluster.

Keeps the NSX Edge nodes running on different ESXi hosts for high availability.

None.

VCF-NSX-EDGE-RCMD-CFG-005

In vSphere HA, set the restart priority policy for each NSX Edge appliance to high.

  • The NSX Edge nodes are part of the north-south data path for overlay segments. vSphere HA restarts the NSX Edge appliances first to minimise the time an edge VM is offline.

  • Setting the restart priority to high reserves highest for future needs.

If the restart priority for another VM in the cluster is set to highest, the connectivity delays for edge appliances will be longer.
VCF-NSX-EDGE-RCMD-CFG-006

Create an NSX Edge cluster with the default Bidirectional Forwarding Detection (BFD) configuration between the NSX Edge nodes in the cluster.

  • Satisfies the availability requirements by default.

  • Edge nodes must remain available to create services such as NAT, routing to physical networks, and load balancing.

None.

VCF-NSX-EDGE-RCMD-CFG-007

Use a single N-VDS in the NSX Edge nodes.

  • Simplifies deployment of the edge nodes.

  • The same N-VDS switch design can be used regardless of edge form factor.

  • Supports multiple TEP interfaces in the edge node.

  • vSphere Distributed Switch is not supported in the edge node.

None.
Table 11. NSX Edge Design Recommendations for Stretched Clusters in VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implications

VCF-NSX-EDGE-RCMD-CFG-008

Add the NSX Edge appliances to the virtual machine group for the first availability zone.

Ensures that, by default, the NSX Edge appliances are powered on upon a host in the primary availability zone.

None.

BGP Routing Design Elements for VMware Cloud Foundation

Table 12. BGP Routing Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-BGP-REQD-CFG-001

To enable ECMP between the Tier-0 gateway and the Layer 3 devices (ToR switches or upstream devices), create two VLANs.

The ToR switches or upstream Layer 3 devices have an SVI on one of the two VLANS, and each Edge node in the cluster has an interface on each VLAN.

Supports multiple equal-cost routes on the Tier-0 gateway and provides more resiliency and better bandwidth use in the network.

Additional VLANs are required.

VCF-NSX-BGP-REQD-CFG-002

Assign a named teaming policy to the VLAN segments to the Layer 3 device pair.

Pins the VLAN traffic on each segment to its target edge node interface. From there, the traffic is directed to the host physical NIC that is connected to the target top of rack switch.

None.

VCF-NSX-BGP-REQD-CFG-003

Create a VLAN transport zone for edge uplink traffic.

Enables the configuration of VLAN segments on the N-VDS in the edge nodes.

Additional VLAN transport zones might be required if the edge nodes are not connected to the same top of rack switch pair.

VCF-NSX-BGP-REQD-CFG-004

Deploy a Tier-1 gateway and connect it to the Tier-0 gateway.

Creates a two-tier routing architecture.

Abstracts the NSX logical components which interact with the physical data center from the logical components which provide SDN services.

A Tier-1 gateway can only be connected to a single Tier-0 gateway.

In cases where multiple Tier-0 gateways are required, you must create multiple Tier-1 gateways.

VCF-NSX-BGP-REQD-CFG-005

Deploy a Tier-1 gateway to the NSX Edge cluster.

Enables stateful services, such as load balancers and NAT, for SDDC management components.

Because a Tier-1 gateway always works in active-standby mode, the gateway supports stateful services.

None.
Table 13. BGP Routing Design Requirements for NSX Multi-Rack Edge Availability for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-MRE-BGP-REQD-CFG-001

To enable ECMP between the Tier-0 gateway and the Layer 3 devices, such as leaf switches or upstream devices, create two separate uplink VLANs for the edge nodes in each rack.

The leaf switches or Layer 3 upstream devices have an SVI on one of the two VLANS for each rack, and each edge node in the rack has an interface on each VLAN.

Supports multiple equal-cost routes on the Tier-0 gateway and improves resiliency and bandwidth use in the network across multiple racks with a pair of leaf switches in each rack.

Additional VLANs are required.

VCF-NSX-MRE-BGP-REQD-CFG-002

Assign a named teaming policy to the VLAN segments to the Layer 3 device pair for each rack.

Pins the VLAN traffic on each segment to its target edge node interface. From there, the traffic is directed to the host physical NIC that is connected to the target leaf switch in the rack.

None.
Table 14. BGP Routing Design Requirements for Stretched Clusters in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-BGP-REQD-CFG-006

Extend the uplink VLANs to the top of rack switches so that the VLANs are stretched between both availability zones.

Because the NSX Edge nodes will fail over between the availability zones, ensures uplink connectivity to the top of rack switches in both availability zones regardless of the zone the NSX Edge nodes are presently in.

You must configure a stretched Layer 2 network between the availability zones by using physical network infrastructure.

VCF-NSX-BGP-REQD-CFG-007

Provide this SVI configuration on the top of the rack switches.

  • In the second availability zone, configure the top of rack switches or upstream Layer 3 devices with an SVI on each of the two uplink VLANs.

  • Make the top of rack switch SVI in both availability zones part of a common stretched Layer 2 network between the availability zones.

Enables the communication of the NSX Edge nodes to the top of rack switches in both availability zones over the same uplink VLANs.

You must configure a stretched Layer 2 network between the availability zones by using the physical network infrastructure.

VCF-NSX-BGP-REQD-CFG-008

Provide this VLAN configuration:

  • Use two VLANs to enable ECMP between the Tier-0 gateway and the Layer 3 devices (top of rack switches or Leaf switches).

  • The ToR switches or upstream Layer 3 devices have an SVI to one of the two VLANS and each NSX Edge node has an interface to each VLAN.

Supports multiple equal-cost routes on the Tier-0 gateway, and provides more resiliency and better bandwidth use in the network.

  • Extra VLANs are required.

  • Requires stretching uplink VLANs between availability zones

VCF-NSX-BGP-REQD-CFG-009

Create an IP prefix list that permits access to route advertisement by any network instead of using the default IP prefix list.

Used in a route map to prepend a path to one or more autonomous system (AS-path prepend) for BGP neighbors in the second availability zone.

You must manually create an IP prefix list that is identical to the default one.

VCF-NSX-BGP-REQD-CFG-010

Create a route map-out that contains the custom IP prefix list and an AS-path prepend value set to the Tier-0 local AS added twice.

  • Used for configuring neighbor relationships with the Layer 3 devices in the second availability zone.

  • Ensures that all ingress traffic passes through the first availability zone.

You must manually create the route map.

The two NSX Edge nodes will route north-south traffic through the second availability zone only if the connection to their BGP neighbors in the first availability zone is lost, for example, if a failure of the top of the rack switch pair or in the availability zone occurs.

VCF-NSX-BGP-REQD-CFG-011

Create an IP prefix list that permits access to route advertisement by network 0.0.0.0/0 instead of using the default IP prefix list.

Used in a route map to configure local-reference on learned default-route for BGP neighbors in the second availability zone.

You must manually create an IP prefix list that is identical to the default one.

VCF-NSX-BGP-REQD-CFG-012

Apply a route map-in that contains the IP prefix list for the default route 0.0.0.0/0 and assign a lower local-preference , for example, 80, to the learned default route and a lower local-preference, for example, 90 any routes learned.

  • Used for configuring neighbor relationships with the Layer 3 devices in the second availability zone.

  • Ensures that all egress traffic passes through the first availability zone.

You must manually create the route map.

The two NSX Edge nodes will route north-south traffic through the second availability zone only if the connection to their BGP neighbors in the first availability zone is lost, for example, if a failure of the top of the rack switch pair or in the availability zone occurs.

VCF-NSX-BGP-REQD-CFG-013

Configure the neighbors of the second availability zone to use the route maps as In and Out filters respectively.

Makes the path in and out of the second availability zone less preferred because the AS path is longer and the local preference is lower. As a result, all traffic passes through the first zone.

The two NSX Edge nodes will route north-south traffic through the second availability zone only if the connection to their BGP neighbors in the first availability zone is lost, for example, if a failure of the top of the rack switch pair or in the availability zone occurs.
Table 15. BGP Routing Design Requirements for NSX Federation in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-BGP-REQD-CFG-014

Extend the Tier-0 gateway to the second VMware Cloud Foundation instance.

  • Supports ECMP north-south routing on all nodes in the NSX Edge cluster.

  • Enables support for cross-instance Tier-1 gateways and cross-instance network segments.

The Tier-0 gateway deployed in the second instance is removed.

VCF-NSX-BGP-REQD-CFG-015

Set the Tier-0 gateway as primary for all VMware Cloud Foundation instances.

  • In NSX Federation, a Tier-0 gateway lets egress traffic from connected Tier-1 gateways only in its primary locations.

  • Local ingress and egress traffic is controlled independently at the Tier-1 level. No segments are provisioned directly to the Tier-0 gateway.

  • A mixture of network spans (local to a VMware Cloud Foundation instance or spanning multiple instances) is enabled without requiring additional Tier-0 gateways and hence edge nodes.

  • If a failure in a VMware Cloud Foundation instance occurs, the local-instance networking in the other instance remains available without manual intervention.

None.

VCF-NSX-BGP-REQD-CFG-016

From the global Tier-0 gateway, establish BGP neighbor peering to the ToR switches connected to the second VMware Cloud Foundation instance.

  • Enables the learning and advertising of routes in the second VMware Cloud Foundation instance.

  • Facilitates a potential automated failover of networks from the first to the second VMware Cloud Foundation instance.

None.

VCF-NSX-BGP-REQD-CFG-017

Use a stretched Tier-1 gateway and connect it to the Tier-0 gateway for cross-instance networking.

  • Enables network span between the VMware Cloud Foundation instances because NSX network segments follow the span of the gateway they are attached to.

  • Creates a two-tier routing architecture.

None.

VCF-NSX-BGP-REQD-CFG-018

Assign the NSX Edge cluster in each VMware Cloud Foundation instance to the stretched Tier-1 gateway. Set the first VMware Cloud Foundation instance as primary and the second instance as secondary.

  • Enables cross-instance network span between the first and second VMware Cloud Foundation instances.

  • Enables deterministic ingress and egress traffic for the cross-instance network.

  • If a VMware Cloud Foundation instance failure occurs, enables deterministic failover of the Tier-1 traffic flow.

  • During the recovery of the inaccessible VMware Cloud Foundation instance, enables deterministic failback of the Tier-1 traffic flow, preventing unintended asymmetrical routing.

  • Eliminates the need to use BGP attributes in the first and second VMware Cloud Foundation instances to influence location preference and failover.

You must manually fail over and fail back the cross-instance network from the standby NSX Global Manager.

VCF-NSX-BGP-REQD-CFG-019

Assign the NSX Edge cluster in each VMware Cloud Foundation instance to the local Tier-1 gateway for that VMware Cloud Foundation instance.

  • Enables instance-specific networks to be isolated to their specific instances.

  • Enables deterministic flow of ingress and egress traffic for the instance-specific networks.

You can use the service router that is created for the Tier-1 gateway for networking services. However, such configuration is not required for network connectivity.

VCF-NSX-BGP-REQD-CFG-020

Set each local Tier-1 gateway only as primary in that instance. Avoid setting the gateway as secondary in the other instances.

Prevents the need to use BGP attributes in primary and secondary instances to influence the instance ingress-egress preference.

None.
Table 16. BGP Routing Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Recommendation Justification

Recommendation Implication

VCF-NSX-BGP-RCMD-CFG-001

Deploy an active-active Tier-0 gateway.

Supports ECMP north-south routing on all Edge nodes in the NSX Edge cluster.

Active-active Tier-0 gateways cannot provide stateful services such as NAT.

VCF-NSX-BGP-RCMD-CFG-002

Configure the BGP Keep Alive Timer to 4 and Hold Down Timer to 12 or lower between the top of tack switches and the Tier-0 gateway.

Provides a balance between failure detection between the top of rack switches and the Tier-0 gateway, and overburdening the top of rack switches with keep-alive traffic.

By using longer timers to detect if a router is not responding, the data about such a router remains in the routing table longer. As a result, the active router continues to send traffic to a router that is down.

These timers must be aligned with the data center fabric design of your organization.

VCF-NSX-BGP-RCMD-CFG-003

Do not enable Graceful Restart between BGP neighbors.

Avoids loss of traffic.

On the Tier-0 gateway, BGP peers from all the gateways are always active. On a failover, the Graceful Restart capability increases the time a remote neighbor takes to select an alternate Tier-0 gateway. As a result, BFD-based convergence is delayed.

None.

VCF-NSX-BGP-RCMD-CFG-004

Enable helper mode for Graceful Restart mode between BGP neighbors.

Avoids loss of traffic.

During a router restart, helper mode works with the graceful restart capability of upstream routers to maintain the forwarding table which in turn will forward packets to a down neighbor even after the BGP timers have expired causing loss of traffic.

None.

VCF-NSX-BGP-RCMD-CFG-005

Enable Inter-SR iBGP routing.

In the event that an edge node has all of its northbound eBGP sessions down, north-south traffic will continue to flow by routing traffic to a different edge node.

None.

VCF-NSX-BGP-RCMD-CFG-006

Deploy a Tier-1 gateway in non-preemptive failover mode.

Ensures that after a failed NSX Edge transport node is back online, it does not take over the gateway services thus preventing a short service outage.

None.

VCF-NSX-BGP-RCMD-CFG-007

Enable standby relocation of the Tier-1 gateway.

Ensures that if an edge failure occurs, a standby Tier-1 gateway is created on another edge node.

None.
Table 17. BGP Routing Design Recommendations for NSX Federation in VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-BGP-RCMD-CFG-008

Use Tier-1 gateways to control the span of networks and ingress and egress traffic in the VMware Cloud Foundation instances.

Enables a mixture of network spans (isolated to a VMware Cloud Foundation instance or spanning multiple instances) without requiring additional Tier-0 gateways and hence edge nodes.

To control location span, a Tier-1 gateway must be assigned to an edge cluster and hence has the Tier-1 SR component. East-west traffic between Tier-1 gateways with SRs need to physically traverse an edge node.

VCF-NSX-BGP-RCMD-CFG-009

Allocate a Tier-1 gateway in each instance for instance-specific networks and connect it to the stretched Tier-0 gateway.

  • Creates a two-tier routing architecture.

  • Enables local-instance networks that are not to span between the VMware Cloud Foundation instances.

  • Guarantees that local-instance networks remain available if a failure occurs in another VMware Cloud Foundation instance.

None.

Overlay Design Elements for VMware Cloud Foundation

Table 18. Overlay Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-OVERLAY-REQD-CFG-001

Configure all ESXi hosts in the workload domain as transport nodes in NSX.

Enables distributed routing, logical segments, and distributed firewall.

None.

VCF-NSX-OVERLAY-REQD-CFG-002

Configure each ESXi host as a transport node using transport node profiles.

  • Enables the participation of ESXi hosts and the virtual machines running on them in NSX overlay and VLAN networks.

  • Transport node profiles can only be applied at the cluster level.

None.

VCF-NSX-OVERLAY-REQD-CFG-003

To provide virtualized network capabilities to workloads, use overlay networks with NSX Edge nodes and distributed routing.

  • Creates isolated, multi-tenant broadcast domains across data center fabrics to deploy elastic, logical networks that span physical network boundaries.

  • Enables advanced deployment topologies by introducing Layer 2 abstraction from the data center networks.

Requires configuring transport networks with an MTU size of at least 1,600 bytes.

VCF-NSX-OVERLAY-REQD-CFG-004

Create a single overlay transport zone in the NSX instance for all overlay traffic across the host and NSX Edge transport nodes of the workload domain or multiple workload domains using a shared NSX instance.

  • Ensures that overlay segments are connected to an NSX Edge node for services and north-south routing.

  • Ensures that all segments are available to all ESXi hosts and NSX Edge nodes configured as transport nodes.

All clusters in all workload domains that share the same NSX Manager share the same overlay transport zone.

VCF-NSX-OVERLAY-REQD-CFG-005

Create an uplink profile with a load balance source teaming policy with two active uplinks for ESXi hosts.

For increased resiliency and performance, supports the concurrent use of both physical NICs on the ESXi hosts that are configured as transport nodes.

None.
Table 19. Overlay Design Requirements for a Multi-Rack Compute VI Workload Domain Cluster for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-L3MR-OVERLAY-REQD-CFG-001

Create an uplink profile for each rack with a separate host TEP transport VLAN ID per rack.

Enables a Layer 3 boundary at the top-of-rack switches for host TEP traffic.

Requires additional host TEP VLAN ID per rack.

VCF-NSX-L3MR-OVERLAY-REQD-CFG-002

Create a host TEP IP pool for each rack with a subnet allocated per rack and a gateway for the subnet.

  • Provides the host TEP IP addressing for each rack

  • Enables a Layer 3 boundary at the top-of-rack switches for host TEP traffic.

  • Provides routing capabilities for the host TEPs between racks.

Requires additional subnet per rack.

VCF-NSX-L3MR-OVERLAY-REQD-CFG-003

Create NSX host sub transport node profiles for additional racks.

  • Enables NSX host transport node configuration per rack.

None
Table 20. Overlay Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-OVERLAY-RCMD-CFG-001

Use static IP pools to assign IP addresses to the host TEP interfaces.

  • Removes the need for an external DHCP server for the host overlay VLANs.

  • You can use NSX Manager to verify static IP pool configurations.

None.

VCF-NSX-OVERLAY-RCMD-CFG-002

Use hierarchical two-tier replication on all overlay segments.

Hierarchical two-tier replication is more efficient because it reduced the number of ESXi hosts the source ESXi host must replicate traffic to if hosts have different TEP subnets. This is typically the case with more than one cluster and will improve performance in that scenario.

None.
Table 21. Overlay Design Recommendations for Stretched Clusters inVMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-OVERLAY-RCMD-CFG-003

Configure an NSX sub-transport node profile.

  • You can use static IP pools for the host TEPs in each availability zone.

  • The NSX transport node profile can remain attached when using two separate VLANs for host TEPs at each availability zone as required for clusters that are based on vSphere Lifecycle Manager images.

  • Using an external DHCP server for the host overlay VLANs in both availability zones is not required.

Changes to the host transport node configuration are done at the vSphere cluster level.

Application Virtual Network Design Elements for VMware Cloud Foundation

Table 22. Application Virtual Network Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-AVN-REQD-CFG-001

Create one cross-instance NSX segment for the components of a VMware Aria Suite application or another solution that requires mobility between VMware Cloud Foundation instances.

Prepares the environment for the deployment of solutions on top of VMware Cloud Foundation, such as VMware Aria Suite, without a complex physical network configuration.

The components of the VMware Aria Suite application must be easily portable between VMware Cloud Foundation instances without requiring reconfiguration.

Each NSX segment requires a unique IP address space.

VCF-NSX-AVN-REQD-CFG-002

Create one or more local-instance NSX segments for the components of a VMware Aria Suite application or another solution that are assigned to a specific VMware Cloud Foundation instance.

Prepares the environment for the deployment of solutions on top of VMware Cloud Foundation, such as VMware Aria Suite, without a complex physical network configuration.

Each NSX segment requires a unique IP address space.
Table 23. Application Virtual Network Design Requirements for NSX Federation in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-AVN-REQD-CFG-003

Extend the cross-instance NSX segment to the second VMware Cloud Foundation instance.

Enables workload mobility without a complex physical network configuration.

The components of a VMware Aria Suite application must be easily portable between VMware Cloud Foundation instances without requiring reconfiguration.

Each NSX segment requires a unique IP address space.

VCF-NSX-AVN-REQD-CFG-004

In each VMware Cloud Foundation instance, create additional local-instance NSX segments.

Enables workload mobility within a VMware Cloud Foundation instance without complex physical network configuration.

Each VMware Cloud Foundation instance should have network segments to support workloads which are isolated to that VMware Cloud Foundation instance.

Each NSX segment requires a unique IP address space.

VCF-NSX-AVN-REQD-CFG-005

In each VMware Cloud Foundation instance, connect or migrate the local-instance NSX segments to the corresponding local-instance Tier-1 gateway.

Configures local-instance NSX segments at required sites only.

Requires an individual Tier-1 gateway for local-instance segments.
Table 24. Application Virtual Network Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-NSX-AVN-RCMD-CFG-001

Use overlay-backed NSX segments.

  • Supports expansion to deployment topologies for multiple VMware Cloud Foundation instances.

  • Limits the number of VLANs required for the data center fabric.

Using overlay-backed NSX segments requires routing, eBGP recommended, between the data center fabric and edge nodes.

Load Balancing Design Elements for VMware Cloud Foundation

Table 25. Load Balancing Design Requirements for VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-LB-REQD-CFG-001

Deploy a standalone Tier-1 gateway to support advanced stateful services such as load balancing for other management components.

Provides independence between north-south Tier-1 gateways to support advanced deployment scenarios.

You must add a separate Tier-1 gateway.

VCF-NSX-LB-REQD-CFG-002

When creating load balancing services for Application Virtual Networks, connect the standalone Tier-1 gateway to the cross-instance NSX segments.

Provides load balancing to applications connected to the cross-instance network.

You must connect the gateway to each network that requires load balancing.

VCF-NSX-LB-REQD-CFG-003

Configure a default static route on the standalone Tier-1 gateway with a next hop the Tier-1 gateway for the segment to provide connectivity to the load balancer.

Because the Tier-1 gateway is standalone, it does not auto-configure its routes.

None.
Table 26. Load Balancing Design Requirements for NSX Federation in VMware Cloud Foundation

Requirement ID

Design Requirement

Justification

Implication

VCF-NSX-LB-REQD-CFG-004

Deploy a standalone Tier-1 gateway in the second VMware Cloud Foundation instance.

Provides a cold-standby non-global service router instance for the second VMware Cloud Foundation instance to support services on the cross-instance network which require advanced services not currently supported as NSX global objects.

  • You must add a separate Tier-1 gateway.

  • You must manually configure any services and synchronize them between the non-global service router instances in the first and second VMware Cloud Foundation instances.

  • To avoid a network conflict between the two VMware Cloud Foundation instances, make sure that the primary and standby networking services are not both active at the same time.

VCF-NSX-LB-REQD-CFG-005

Connect the standalone Tier-1 gateway in the second VMware Cloud Foundationinstance to the cross-instance NSX segment.

Provides load balancing to applications connected to the cross-instance network in the second VMware Cloud Foundation instance.

You must connect the gateway to each network that requires load balancing.

VCF-NSX-LB-REQD-CFG-006

Configure a default static route on the standalone Tier-1 gateway in the second VMware Cloud Foundation instance with a next hop as the Tier-1 gateway for the segment it connects with to provide connectivity to the load balancers.

Because the Tier-1 gateway is standalone, it does not autoconfigure its routes.

None.

VCF-NSX-LB-REQD-CFG-007

Establish a process to ensure any changes made on to the load balancer instance in the first VMware Cloud Foundationinstance are manually applied to the disconnected load balancer in the second instance.

Keeps the network service in the failover load balancer instance ready for activation if a failure in the first VMware Cloud Foundation instance occurs.

Because network services are not supported as global objects, you must configure them manually in each VMware Cloud Foundation instance. The load balancer service in one instance must be connected and active, while the service in the other instance must be disconnected and inactive.

  • Because of incorrect configuration between the VMware Cloud Foundation instances, the load balancer service in the second instance might come online with an invalid or incomplete configuration.

  • If both VMware Cloud Foundation instances are online and active at the same time, a conflict between services could occur resulting in a potential outage.

  • The administrator must establish and follow an operational practice by using a runbook or automated process to ensure that configuration changes are reproduced in each VMware Cloud Foundation instance.