All three sites in a stretched cluster communicate across the management network and across the Virtual SAN network. The VMs in both data sites communicate across a common virtual machine network.

A Virtual SAN stretched cluster must meet certain basic networking requirements.
  • Management network requires connectivity across all three sites, using a Layer 2 stretched network or a Layer 3 network.
  • Virtual SAN network requires connectivity across all three sites. VMware recommends using a Layer 2 stretched network between the two data sites and a Layer 3 network between the data sites and the witness host.
  • VM network requires connectivity between the data sites, but not the witness host. VMware recommends using a Layer 2 stretched network between the data sites. In the event of a failure, the VMs do not require a new IP address to work on the remote site.
  • vMotion network requires connectivity between the data sites, but not the witness host. VMware supports using a Layer 2 stretched or a Layer 3 network between data sites.

Using Static Routes on ESXi Hosts

If you use a single default gateway on ESXi hosts, note that each ESXi host contains a default TCP/IP stack that has a single default gateway. The default route is typically associated with the management network TCP/IP stack.

The management network and the Virtual SAN network might be isolated from one another. For example, the management network might use vmk0 on physical NIC 0, while the Virtual SAN network uses vmk2 on physical NIC 1 (separate network adapters for two distinct TCP/IP stacks). This configuration implies that the Virtual SAN network has no default gateway.

Consider a Virtual SAN network that is stretched over two data sites on a Layer 2 broadcast domain (for example, 172.10.0.0) and the witness host is on another broadcast domain (for example, 172.30.0.0). If the VMkernel adapters on a data site try to connect to the Virtual SAN network on the witness host, the connection will fail because the default gateway on the ESXi host is associated with the management network and there is no route from the management network to the Virtual SAN network.

You can use static routes to resolve this issue. Define a new routing entry that indicates which path to follow to reach a particular network. For a Virtual SAN network on a stretched cluster, you can add static routes to ensure proper communication across all hosts.

For example, you can add a static route to the hosts on each data site, so requests to reach the 172.30.0.0 witness network are routed through the 172.10.0.0 interface. Also add a static route to the witness host so that requests to reach the 172.10.0.0 network for the data sites are routed through the 172.30.0.0 interface.

Note: If you use static routes, you must manually add the static routes for new ESXi hosts added to either site before those hosts can communicate across the cluster. If you replace the witness host, you must update the static route configuration.

Use the esxcli network ip route command to add static routes.