VMware Cloud Foundation uses vSphere Distributed Switch for virtual networking.

Logical vSphere Networking Design for VMware Cloud Foundation

When you design vSphere networking, consider the configuration of the vSphere Distributed Switches, distributed port groups, and VMkernel adapters in the VMware Cloud Foundation environment.

vSphere Distributed Switch Design

The default cluster in a workload domain uses a single vSphere Distributed Switch with a configuration for system traffic types, NIC teaming, and MTU size.

VMware Cloud Foundation supports NSX Overlay traffic over a single vSphere Distributed Switch per cluster. Additional distributed switches are supported for other traffic types.

When using vSAN ReadyNodes, you must define the number of vSphere Distributed Switches at workload domain deployment time. You cannot add additional vSphere Distributed Switches post deployment.

Table 1. Configuration Options for vSphere Distributed Switch for VMware Cloud Foundation

vSphere Distributed Switch Configuration

Management Domain Options

VI Workload Domain Options

Benefits

Drawbacks

Single vSphere Distributed Switch for hosts with two physical NICs

  • One vSphere Distributed Switch for each cluster with all traffic using two uplinks.

  • One vSphere Distributed Switch for each cluster with all traffic using two uplinks.

Requires the least number of physical NICs and switch ports.

All traffic shares the same two uplinks.

Single vSphere Distributed Switch for hosts with four or six physical NICs

  • One vSphere Distributed Switch for each cluster with four uplinks by using the predefined profiles in the Deployment Parameters Workbook in VMware Cloud Builder to deploy the default management cluster.

  • One vSphere Distributed Switch for each cluster with four or six uplinks by using the VMware Cloud Builder API to deploy the default management cluster.

  • One vSphere Distributed Switch for each cluster with four or six uplinks.

  • Provides support for traffic separation across different uplinks.

  • You must provide additional physical NICs and switch ports.

Multiple vSphere Distributed Switches

  • Maximum two vSphere Distributed Switches by using the predefined profiles in the Deployment Parameters Workbook in VMware Cloud Builder to deploy the default management cluster.

  • Maximum 16 vSphere Distributed Switches per cluster. You use the VMware Cloud Builder API to deploy the default management cluster using combinations of vSphere Distributed Switches and physical NIC configurations that are not available as predefined profiles in the Deployment Parameters Workbook

  • You can use only one of the vSphere Distributed Switches for NSX overlay traffic.

  • Maximum 16 vSphere Distributed Switches per cluster.

  • You can use only one of the vSphere Distributed Switches for NSX overlay traffic.

  • Provides support for traffic separation across different uplinks or vSphere Distributed Switches.

  • Provides support for traffic separation onto different physical network fabrics.

  • You must provide additional physical NICs and switch ports.

  • More complex with additional configuration and management overhead.

Distributed Port Group Design

VMware Cloud Foundation requires several port groups on the vSphere Distributed Switch for a workload domain. The VMkernel adapters for the host TEPs are connected to the host overlay network, but does not require a dedicated port group on the distributed switch. The VMkernel network adapter for host TEP is automatically created VMware Cloud Foundation configures the ESXi host as a transport node.

Table 2. Distributed Port Group Configuration for VMware Cloud Foundation

Function

Teaming Policy

Management Domain

VI Workload Domain

  • Management

  • vSphere vMotion

  • vSAN

Route based on physical NIC load

Required.

Recommended.

  • Failover Detection: Link status only

  • Failback: Yes

    Occurs only on saturation of the active uplink.

  • Notify Switches: Yes

Recommended.

Recommended.

  • Host Overlay

Not applicable.

Not applicable.

Not applicable.

  • Edge Uplinks and Overlay

Use explicit failover order.

Required.

Required.

  • Edge RTEP (NSX Federation Only)

Not applicable.

Not applicable.

Not applicable.

VMkernel Network Adapter Design

The VMkernel networking layer provides connectivity to hosts and handles the system traffic for management, vSphere vMotion, vSphere HA, vSAN, and others.

Table 3. Default VMkernel Adapters for a Workload Domain per Availability Zone

VMkernel Adapter Service

Connected Port Group

Activated Services

Recommended MTU Size (Bytes)

Management

Management Port Group

Management Traffic

1500 (Default)

vMotion

vMotion Port Group

vMotion Traffic

9000

vSAN

vSAN Port Group

vSAN

9000

Host TEPs

Not applicable

Not applicable

9000

vSphere Networking Design Recommendations for VMware Cloud Foundation

Consider the recommendations for vSphere networking in VMware Cloud Foundation, such as MTU size, port binding, teaming policy and traffic-specific network shares.

Table 4. vSphere Networking Design Recommendations for VMware Cloud Foundation

Recommendation ID

Design Recommendation

Justification

Implication

VCF-VDS-RCMD-CFG-001

Use a single vSphere Distributed Switch per cluster.

  • Reduces the complexity of the network design.

  • Reduces the size of the fault domain.

Increases the number of vSphere Distributed Switches that must be managed.

VCF-VDS-RCMD-CFG-002

Configure the MTU size of the vSphere Distributed Switch to 9000 for jumbo frames.

  • Supports the MTU size required by system traffic types.

  • Improves traffic throughput.

When adjusting the MTU packet size, you must also configure the entire network path (VMkernel ports, virtual switches, physical switches, and routers) to support the same MTU packet size.

VCF-VDS-RCMD-DPG-001

Use ephemeral port binding for the Management VM port group.

Using ephemeral port binding provides the option for recovery of the vCenter Server instance that is managing the distributed switch.

Port-level permissions and controls are lost across power cycles, and no historical context is saved.

VCF-VDS-RCMD-DPG-002

Use static port binding for all non-management port groups.

Static binding ensures a virtual machine connects to the same port on the vSphere Distributed Switch. This allows for historical data and port level monitoring.

None.

VCF-VDS-RCMD-DPG-003

Use the Route based on physical NIC load teaming algorithm for the VM management port group.

Reduces the complexity of the network design, increases resiliency, and can adjust to fluctuating workloads.

None.

VCF-VDS-RCMD-DPG-004

Use the Route based on physical NIC load teaming algorithm for the ESXi management port group.

Reduces the complexity of the network design, increases resiliency, and can adjust to fluctuating workloads.

None.

VCF-VDS-RCMD-DPG-005

Use the Route based on physical NIC load teaming algorithm for the vSphere vMotion port group.

Reduces the complexity of the network design, increases resiliency, and can adjust to fluctuating workloads.

None.

VCF-VDS-RCMD-DPG-006

Use the Route based on physical NIC load teaming algorithm for the vSAN port group.

Reduces the complexity of the network design, increases resiliency, and can adjust to fluctuating workloads.

None.

VCF-VDS-RCMD-NIO-001

Enable Network I/O Control on vSphere Distributed Switch of the management domain cluster.

Increases resiliency and performance of the network.

Network I/O Control might impact network performance for critical traffic types if misconfigured.

VCF-VDS-RCMD-NIO-002

Set the share value for management traffic to Normal.

By keeping the default setting of Normal, management traffic is prioritized higher than vSphere vMotion but lower than vSAN traffic. Management traffic is important because it ensures that the hosts can still be managed during times of network contention.

None.

VCF-VDS-RCMD-NIO-003

Set the share value for vSphere vMotion traffic to Low.

During times of network contention, vSphere vMotion traffic is not as important as virtual machine or storage traffic.

During times of network contention, vMotion takes longer than usual to complete.

VCF-VDS-RCMD-NIO-004

Set the share value for virtual machines to High.

Virtual machines are the most important asset in the SDDC. Leaving the default setting of High ensures that they always have access to the network resources they need.

None.

VCF-VDS-RCMD-NIO-005

Set the share value for vSAN traffic to High.

During times of network contention, vSAN traffic needs guaranteed bandwidth to support virtual machine performance.

None.

VCF-VDS-RCMD-NIO-006

Set the share value for other traffic types to Low.

By default, VVMware Cloud Foundation does not use other traffic types, like vSphere FT traffic. Hence, these traffic types can be set the lowest priority.

None.