vSAN supports stretched cluster deployments that span two locations.
In vSAN 6.5 and earlier, vSAN traffic between data sites is multicast for metadata and unicast for I/O.
In vSAN 6.6 and later, all traffic is unicast. In all versions of vSAN, the witness traffic between a data site and the witness host is unicast.
Layer 2 Everywhere
You can configure a vSAN stretched cluster in a Layer 2 network, but this configuration is not recommended.
Consider a design where the vSAN stretched cluster is configured in one large Layer 2 design. Data Site 1 and Site 2 are where the virtual machines are deployed. Site 3 contains the witness host.
To demonstrate Layer 2 everywhere as simply as possible, we use switches (and not routers) in the topologies.
Layer 2 networks cannot have any loops (multiple paths), so features such as Spanning Tree Protocol (STP) are needed to block one of the connections between Site 1 and Site 2. Now consider a situation where the link between Site 2 and Site 3 is broken (the link between Site 1 and Site 2). Network traffic can be switched from Site 1 to Site 2 through the witness host at Site 3. As VMware supports a much lower bandwidth and higher latency for the witness host, you see a significant decrease in performance if data network traffic passes through a lower specification witness site.
If switching traffic between data sites through the witness site does not impact latency of applications, and bandwidth is acceptable, a stretched L2 configuration between sites is possible. In most cases, such a configuration is not feasible, and adds complexity to the networking requirements.
With vSAN 6.5 or earlier, which uses multicast traffic, you must configure IGMP snooping on the switches. This is not necessary with vSAN 6.6 and later. PIM is not necessary because there is no routing of multicast traffic.
Supported vSAN Stretched Cluster Configurations
vSAN supports stretched cluster configurations.
The following configuration prevent traffic from Site 1 being routed to Site 2 through the witness host, in the event of a failure on either of the data sites' network. This configuration avoids performance degradation. To ensure that data traffic is not switched through the witness host, use the following network topology.
Between Site 1 and Site 2, implement a stretched Layer 2 switched configuration or a Layer 3 routed configuration. Both configurations are supported.
Between Site 1 and the witness host, implement a Layer 3 routed configuration.
Between Site 2 and the witness host, implement a Layer 3 routed configuration.
These configurations (L2+L3, and L3 everywhere) are described with considerations given to multicast in vSAN 6.5 and earlier, and unicast only, which is available in vSAN 6.6. Multicast traffic introduces additional configuration steps for IGMP snooping, and PIM for routing multicast traffic.
We shall examine a stretched Layer 2 network between the data sites and a Layer 3 routed network to the witness site. To demonstrate a combination of Layer 2 and Layer 3 as simply as possible, use a combination of switches and routers in the topologies.
Stretched Layer 2 Between Data Sites, Layer 3 to Witness Host
vSAN supports stretched Layer 2 configurations between data sites.
The only traffic that is routed in this case is the witness traffic. With vSAN 6.5 and earlier, which uses multicast, use IGMP snooping for the multicast traffic on the stretched L2 vSAN between data sites. However, since the witness traffic is unicast, there is no need to implement PIM on the Layer 3 segments.
With vSAN 6.6, which uses unicast, there is no requirement to consider IGMP snooping or PIM.
Layer 3 Everywhere
In this vSAN stretched cluster configuration, the data traffic is routed between the data sites and the witness host.
To implement Layer 3 everywhere as simply as possible, use routers or routing switches in the topologies.
For example, consider an environment with vSAN 6.5 or earlier, which uses multicast traffic. In this case, configure IGMP snooping on the data site switches to manage the amount of multicast traffic on the network. This is unnecessary at the witness host since witness traffic is unicast. The multicast traffic is routed between the data sites, so configure PIM to allow multicast routing.
With vSAN 6.6 and later, neither IGMP snooping nor PIM are needed because all the routed traffic is unicast.
Separating Witness Traffic on vSAN Stretched Clusters
vSAN supports separating witness traffic on stretched clusters.
Inn vSAN 6.5 and later releases, you can separate witness traffic from vSAN traffic in two-node configurations. This means that the two vSAN hosts can be directly connected without a 10 Gb switch.
This witness traffic separation is only supported on two-node deployments in vSAN 6.6. Separating the witness traffic on vSAN stretched clusters is supported in vSAN 6.7 and later.
Using vSAN Stretched Cluster to Achieve Rack Awareness
With vSAN stretched clusters, vSAN provides rack awareness in a single site.
If you have two racks of vSAN hosts, you can continue to run your vSAN cluster after a complete rack failure. In this case, availability of the VM workloads is provided by the remaining rack and a remote witness host.
In this example, if rack 1 fails, rack 2 and the witness host provide VM availability. This configuration is a pre-vSAN 6.6 environment, and needs multicast configured on the network. The witness host must be on the vSAN network. Witness traffic is unicast. In vSAN 6.6 and later, all traffic is unicast.
This topology is also supported over L3. Place the vSAN VMkernel ports on different subnets or VLANs, and use a separate subnet or VLAN for each rack.
This topology supports deployments with two racks to achieve rack awareness (fault domains) with a vSAN stretched cluster. This solution uses a witness host that is external to the cluster.
