Unicast traffic refers to a one-to-one transmission from one point in the network to another. vSAN version 6.6 and later uses unicast to simplify network design and deployment.
All ESXi hosts use the unicast traffic, and the vCenter Server becomes the source for the cluster membership. The vSAN nodes are automatically updated with the latest host membership list that vCenter provides. vSAN communicates using unicast for CMMDS updates.
Releases earlier than vSAN version 6.6 rely on multicast to enable the heartbeat and to exchange metadata between hosts in the cluster. If some hosts in your vSAN cluster are running earlier versions of the software, a multicast network is still required. The switch to unicast network from multicast provides better performance and network support. For more information on multicast, see Using Multicast in vSAN Network.
Pre-Version 5 Disk Group Behavior
The availability of a single version 5 disk group in vSAN version 6.6 disk group triggers the cluster to communicate permanently in the unicast mode.
- All cluster hosts are running vSAN version 6.5 or lower.
- All disk groups are using on-disk version 3 or earlier.
- A non-vSAN 6.6 host such as vSAN 6.2 or vSAN 6.5 is added to the cluster.
For example, if a host running vSAN 6.5 or earlier is added to an existing vSAN 6.6 cluster, the cluster reverts to multicast mode and includes the 6.5 host as a valid node. To avoid this behavior, use the latest version for both ESXi hosts and on-disk format. To ensure that vSAN cluster continues communicating in unicast mode and does not revert to multicast, upgrade the disk groups on the vSAN 6.6 hosts to on-disk version 5.0.
Version 5 Disk Group Behavior
The presence of a single version 5 disk group in a vSAN version 6.6 cluster triggers the cluster to communicate permanently in unicast mode.
In an environment where a vSAN 6.6 cluster is already using an on-disk version 5 and a vSAN 6.5 node is added to the cluster, the following events occur:
- The vSAN 6.5 node forms its own network partition.
- The vSAN 6.5 node continues to communicate in multicast mode but is unable to communicate with vSAN 6.6 nodes as they use unicast mode.
A cluster summary warning appears on the on-disk format showing that one node is at an earlier version. You can upgrade the node to the latest version. You cannot upgrade disk format versions when a cluster is in a mixed mode.
DHCP Support on Unicast Network
vCenter Server deployed on a vSAN 6.6 cluster can use IP addresses from Dynamic Host Configuration Protocol (DHCP) without reservations.
You can use DHCP with reservations as the assigned IP addresses are tied to the MAC addresses of VMkernel ports.
IPv6 Support on Unicast Network
vSAN 6.6 supports IPv6 with unicast communications.
With IPv6, the link-local address is automatically configured on any interface using the link-local prefix. By default, vSAN does not add the link local address of a node to other neighboring cluster nodes. As a result, vSAN 6.6 does not support IPv6 link local addresses for unicast communications.
Query Unicast with ESXCLI
You can run ESXCLI commands to determine the unicast configuration.
View the Communication Modes
Using esxcli vsan cluster get command, you can view the CMMDS mode (unicast or multicast) of the vSAN cluster node.
Procedure
- ♦ Run the
esxcli vsan cluster getcommand.
Results
Cluster Information
Enabled: true
Current Local Time: 2020-04-09T18:19:52Z
Local Node UUID: 5e8e3dc3-43ab-5452-795b-a03d6f88f022
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5e8e3d3f-3015-9075-49b6-a03d6f88d426
Sub-Cluster Backup UUID: 5e8e3daf-e5e0-ddb6-a523-a03d6f88dd4a
Sub-Cluster UUID: 5282f9f3-d892-3748-de48-e2408dc34f72
Sub-Cluster Membership Entry Revision: 11
Sub_cluster Member Count: 5
Sub-Cluster Member UUIDs: 5e8e3d3f-3015-9075-49b6-a03d6f88d426, 5e8e3daf-e5e0-ddb6-a523-a03d6f88dd4a,
5e8e3d73-6d1c-0b81-1305-a03d6f888d22, 5e8e3d33-5825-ee5c-013c-a03d6f88ea4c, 5e8e3dc3-43ab-5452-795b-a03d6f88f022
Sub-Cluster Member HostNames: testbed-1.vmware.com, testbed2.vmware.com,
testbed3.vmware.com, testbed4.vmware.com, testbed5.vmware.com
Sub-Cluster Membership UUID: 0f438e5e-d400-1bb2-f4d1-a03d6f88d426
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: ed845022-5c08-48d0-aa1d-6b62c0022222 7 2020-04-08T22:44:14.889
Verify the vSAN Cluster Hosts
Use the esxcli vsan cluster unicastagent list command to verify whether the vSAN cluster hosts are operating in unicast mode.
Procedure
- ♦ Run the
esxcli vsan cluster unicastagent listcommand.
Results
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid ------------------------------------ --------- ---------------- ---------- ----- ---------- 5e8e3d73-6d1c-0b81-1305-a03d6f888d22 0 true 10.198.95.10 12321 43:80:B7:A1:3F:D1:64:07:8C:58:01:2B:CE:A2:F5:DE:D6:B1:41:AB 5e8e3daf-e5e0-ddb6-a523-a03d6f88dd4a 0 true 10.198.94.240 12321 FE:39:D7:A5:EF:80:D6:41:CD:13:70:BD:88:2D:38:6C:A0:1D:36:69 5e8e3d3f-3015-9075-49b6-a03d6f88d426 0 true 10.198.94.244 12321 72:A3:80:36:F7:5D:8F:CE:B0:26:02:96:00:23:7D:8E:C5:8C:0B:E1 5e8e3d33-5825-ee5c-013c-a03d6f88ea4c 0 true 10.198.95.11 12321 5A:55:74:E8:5F:40:2F:2B:09:B5:42:29:FF:1C:95:41:AB:28:E0:57
The output includes the vSAN node UUID, IPv4 address, IPv6 address, UDP port with which vSAN node communicates, and whether the node is a data host (0) or a witness host (1). You can use this output to identify the vSAN cluster nodes that are operating in unicast mode and view the other hosts in the cluster. vCenter Server maintains the output list.
View the vSAN Network Information
Use the esxcli vsan network list command to view the vSAN network information such as the VMkernel interface that vSAN uses for communication, the unicast port (12321), and the traffic type (vSAN or witness) associated with the vSAN interface.
Procedure
- ♦ Run the
esxcli vsan network listcommand.
Results
Interface
VmkNic Name: vmk1
IP Protocol: IP
Interface UUID: e290be58-15fe-61e5-1043-246e962c24d0
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Multicast TTL: 5
Traffic Type: vsan
This output also displays the multicast information.
Intra-Cluster Traffic
In unicast mode, the primary node addresses all the cluster nodes as it sends the same message to all the vSAN nodes in a cluster.
For example, if N is the number of vSAN nodes, then the primary node sends the messages N number of times. This results in a slight increase of vSAN CMMDS traffic. You might not notice this slight increase of traffic during normal, steady-state operations.
Intra-Cluster Traffic in a Single Rack
If all the nodes in a vSAN cluster are connected to the same top of the rack (TOR) switch, then the total increase in traffic is only between the primary node and the switch.
If a vSAN cluster spans more than one TOR switch, traffic between the switch expands. If a cluster spans many racks, multiple TORs form Fault Domains (FD) for the rack awareness. The primary node sends N messages to the racks or fault domains, where N is the number of hosts in each fault domain.
Intra-Cluster Traffic in a vSAN Stretched Cluster
In a vSAN stretched cluster, the primary node is located at the preferred site.
In a fault domain, CMMDS data must be communicated from the secondary site to the preferred site. To calculate the traffic in a vSAN stretched cluster, you must multiply the number of nodes in a secondary site with the CMMDS node size (in MB) to the number of nodes in the secondary site.
Traffic in a vSAN stretched cluster = number of nodes in the secondary site * CMMDS node size (in MB) * number of nodes in the secondary site.
With the unicast traffic, there is no change in the witness site traffic requirements.
