When performing network configuration, you have to consider the overall traffic and decide how to isolate vSAN traffic.

vSAN Network Considerations

  • Consider how much replication and communication traffic is running between ESXi hosts. With vSAN, the amount of traffic depends on the number of VMs that are running in the cluster, and on how write-intensive the I/O is for the applications running in the VMs. 

  • Isolate vSAN traffic on its own Layer 2 network segment. You can do this using dedicated switches or ports, or by using a VLAN. 

The vSAN VMkernel port group is created as part of cluster creation. Configure this port group on all ESXi hosts in a cluster, even for ESXi hosts that are not contributing storage resources to the cluster. 

Figure 1. VMware vSAN Conceptual Network with a Single Availability Zone



Availability Zones Network Considerations

When using two availability zones the management VLAN that vCenter Server and other VLAN backed management virtual machines utilize must be stretched across both availability zones. The technology used to stretch the VLAN is out of scope and will vary based on the customers’ existing infrastructure.

The connectivity between Availability Zones must support jumbo frames, and ensure that latency is less than 5 ms.

Figure 2. VMware vSAN Conceptual Network with two Availability Zones



Network Bandwidth Requirements

For solutions use a 10-Gb Ethernet connection for use with vSAN to ensure the best and most predictable performance (IOPS) for the environment. Without it, a significant decrease in array performance results.

Table 1. Network Speed Selection

Design Quality

1Gb

10Gb

Comments

Availability

o

o

Neither design option impacts availability.

Manageability

o

o

Neither design option impacts manageability.

Performance

Faster network speeds increase vSAN performance (especially in I/O intensive situations).

Recoverability

Faster network speeds increase the performance of rebuilds and synchronizations in the environment. This ensures that VMs are properly protected from failures.

Security

o

o

Neither design option impacts security.

 Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Note:

10 GbE Ethernet connection also provides support for future use of vSAN all-flash configurations.

Table 2. Design Decisions on Network Bandwidth for vSAN

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-001

Use only 10 GbE for vSAN traffic.

Performance with 10 GbE is optimal. Without it, a significant decrease in array performance results.

The physical network must support 10 Gb networking between every ESXi host in the vSAN clusters.

VMware vSAN Virtual Switch Type

vSAN supports the use of vSphere Standard Switch or vSphere Distributed Switch. The benefit of using vSphere Distributed Switch is that it supports Network I/O Control which allows for prioritization of bandwidth in case of contention in an environment.

This design uses a vSphere Distributed Switch for the vSAN port group to ensure that priority can be assigned using Network I/O Control to separate and guarantee the bandwidth for vSAN traffic.

Virtual Switch Design Background

Virtual switch type affects performance and security of the environment.

Table 3. Virtual Switch Types

Design Quality

vSphere Standard Switch

vSphere Distributed Switch

Comments

Availability

o

o

Neither design option impacts availability.

Manageability

The vSphere Distributed Switch is centrally managed across all ESXi hosts, unlike the standard switch which is managed on each ESXi host individually.

Performance

The vSphere Distributed Switch has added controls, such as Network I/O Control, which you can use to guarantee performance for vSAN traffic.

Recoverability

The vSphere Distributed Switch configuration can be backed up and restored, the standard switch does not have this functionality.

Security

The vSphere Distributed Switch has added built-in security controls to help protect traffic.

Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Table 4. Design Decisions on Virtual Switch Configuration for vSAN

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-002

Use the existing vSphere Distributed Switch instances in the management cluster in each region.

Provides guaranteed performance for vSAN traffic, if there is network contention, by using existing networking components. 

All traffic paths are shared over common uplinks.

Jumbo Frames

VMware vSAN supports jumbo frames for vSAN traffic. 

A VMware vSAN design should use jumbo frames only if the physical environment is already configured to support them, they are part of the existing design, or if the underlying configuration does not create a significant amount of added complexity to the design.

Table 5. Design Decisions on Jumbo Frames for vSAN

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-003

Configure jumbo frames on the VLAN dedicated to vSAN traffic.

Jumbo frames are already used to improve performance of vSphere vMotion and NFS storage traffic.

Every device in the network must support jumbo frames.

VLANs

VMware recommends isolating VMware vSAN traffic on its own VLAN. When a design uses multiple vSAN clusters, each cluster should use a dedicated VLAN or segment for its traffic. This approach prevents interference between clusters and helps with troubleshooting cluster configuration.

Table 6.  Design Decisions on vSAN VLAN

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-004

When using a single availability zone, configure a dedicated VLAN for vSAN traffic for each vSAN enabled cluster.

VLANs provide traffic isolation.

 VLANs span only a single cluster.

Enough VLANs are available in each cluster and are to be used for traffic segregation.

SDDC-VI-Storage-SDS-005

When using two availability zones, configure a dedicated VLAN in each availability zone for each vSAN enabled cluster.

VLANs provide traffic isolation.

vSAN traffic between availability zones is routed. An additional stretched VLAN is not required.

Enough VLANs are available within each cluster and are to be used for traffic segregation.

Static routes on the ESXi hosts are required.

vSAN Witness

When using vSAN in a stretched cluster configuration, you must configure a vSAN stretched cluster witness host. This ESXi host must be configured in a third location that is not local to ESXi hosts on either side of a stretched cluster.

This vSAN witness can be configured as a physical ESXi host or make use of the vSAN Witness Appliance.

Table 7. Design Decisions on the vSAN Witness Appliance for Multiple Availability Zones

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-VI-Storage-SDS-006

Use a vSAN witness appliance located in the management cluster of Region B.

Region B is isolated from both availability zones in Region A and can function as an appropriate quorum location.

A third physically separate location is required when implementing a vSAN stretched cluster between two locations.