vSAN is network-dependent. Understanding and configuring the right vSAN network settings is key to avoiding performance and stability issues.

A reliable and robust vSAN network has the following characteristics:

Unicast

vSAN 6.6 and later releases support unicast communication. Unicast traffic is a one-to-one transmission of IP packets from one point in the network to another point. Unicast transmits the heartbeat sent from the primary host to all other hosts each second. This ensures that the hosts are active and indicates the participation of hosts in the vSAN cluster. You can design a simple unicast network for vSAN. For more information on the unicast communication, see Using Unicast in vSAN Network.

Multicast

Releases earlier than vSAN 6.6 use IP multicast communication as a discovery protocol to identify the nodes trying to join a vSAN cluster.
Note: If possible, always use the latest version of vSAN.
IP multicast relies on communication protocols used by hosts, clients, and network devices to participate in multicast-based communications. For more information on the multicast communication, see Using Multicast in vSAN Network.

Layer 2 and Layer 3 Network

All hosts in the vSAN cluster must be connected through a Layer 2 or Layer 3 network. vSAN releases earlier than vSAN 6.0 support only Layer 2 networking, whereas subsequent releases include support for both Layer 2 and Layer 3 protocols. Use a Layer 2 or Layer 3 network to provide communication between the data sites and the witness site. For more information on Layer 2 and Layer 3 network topologies, see Standard Deployments.

VMkernel Network

Each ESXi host in a vSAN cluster must have a network adapter for vSAN communication. All the intra-cluster node communication happens through the vSAN VMkernel port. VMkernel ports provide Layer 2 and Layer 3 services to each vSAN host and hosted virtual machines.

vSAN Network Traffic

Several different traffic types are available in the vSAN network, such as the storage traffic and the unicast traffic. The compute and storage of a virtual machine can be on the same host or on different hosts in the cluster. A VM that is not configured to tolerate a failure might be running on one host, and accessing a VM object or component that resides on a different host. This implies that all I/O from the VM passes through the network. The storage traffic constitutes most of the traffic in a vSAN cluster.

The cluster-related communication between all the ESXi hosts creates traffic in the vSAN cluster. This unicast traffic also contributes to the vSAN network traffic.

Virtual Switch

vSAN supports the following types of virtual switches:
  • The Standard Virtual Switch provides connectivity from VMs and VMkernel ports to external networks. This switch is local to each ESXi host.
  • A vSphere Distributed Switch provides central control of the virtual switch administration across multiple ESXi hosts. A distributed switch also provides networking features such as Network I/O Control (NIOC) that can help you set Quality of Service (QoS) levels on vSphere or virtual network. vSAN includes vSphere Distributed Switch irrespective of the vCenter Server version.

Bandwidth

vSAN traffic can share 10 GbE, 25 GbE, 40 GbE, and 100 GbE physical network adapters with other system traffic types, such as vSphere vMotion traffic, vSphere HA traffic, and virtual machine traffic. It also provides more bandwidth for shared network configurations where vSAN, vSphere management, vSphere vMotion traffic, and so on, are on the same physical network. To guarantee the amount of bandwidth required for vSAN, use vSphere Network I/O Control in the distributed switch.

In vSphere Network I/O Control, you can configure reservation and shares for the vSAN outgoing traffic:
  • Set a reservation so that Network I/O Control guarantees that a minimum bandwidth is available on the physical adapter for vSAN.
  • Set the share value to 100 so that when the physical adapter assigned for vSAN becomes saturated, certain bandwidth is available to vSAN. For example, the physical adapter might become saturated when another physical adapter in the team fails and all traffic in the port group is transferred to the other adapters in the team.

For information about using Network I/O Control to configure bandwidth allocation for vSAN traffic, see the vSphere Networking documentation.