From release 3.9.1 onwards, VMware Cloud Foundation supports stretching a cluster across two availability zones within a region for both Layer 2 and Layer 3 networks.

For more information on the availability zones, see About Availability Zones and Regions.
Note:
  • Command such as --show-free-hosts operations is not applicable for the Dell EMC VxRail environment. If you run these commands, 'Operation is not applicable for this platform!!' warning is thrown.
  • The Import VxRail Host to cluster operation is disabled for a stretched cluster.
  • For L2 networks in NSX-V, the management VLAN, vSAN VLAN, and vMotion VLAN between the two availability zones have to be stretched.
  • For L3 networks in NSX-V, the management VLAN has to be stretched.
The details for the Cloud Foundation networks for Layer 2 are as follows:
Network Name Connectivity to AZ2 Minimum MTU Maximum MTU
vSAN L2 1500 9000
vMotion L2 1500 9000
VXLAN (VTEP) L2 1600 9000
Management L2 1500 9000
Witness Management L3 1500 9000
Witness vSAN L3 1500 9000
The details for the Cloud Foundation networks for Layer 3 are as follows:
Network Name Connectivity to AZ2 Minimum MTU Maximum MTU
vSAN L3 1500 9000
vMotion L3 1500 9000
VXLAN (VTEP) L3 1600 9000
Management L2 1500 9000
Witness Management L3 1500 9000
Witness vSAN L3 1500 9000

To stretch a cluster for VMware Cloud Foundation on Dell EMC VxRail, perform the following steps:

Procedure

  1. Using SSH, log in to the SDDC Manager VM with the user name vcf and the password you specified in the deployment parameter sheet.
  2. Prepare the workflow. Use the SoS commands to prepare the cluster. See SoS Utility Options for vSAN Stretched Clusters in SoS Utility Options.
    /opt/vmware/sddc-support/sos --prepare-stretch --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched>
    Once the workflow is triggered, track the task status in the SDDC Manager UI.
  3. Power on the hosts that are rack mounted in Availability Zone 2 (or Region B) and log in to VxRail Manager. Perform the cluster expansion. For information on cluster expansion, refer the Dell EMC VxRail documentation.
  4. You have to deploy the witness in a different site. Add the witness host or the appliance to the management or the workload domain vCenter. Follow the steps listed below as described in Deploying a VSAN Witness Appliance to add a vSAN witness.
    1. Deploy and Configure the vSAN Witness Host in Region B.
    2. Add Static Routes for both Availability Zones and the vSAN Witness Host.

      If the default gateway in the vSAN network provided for the network pool does not provide routing between the two availability zones and the witness host, perform all the steps in this procedure.

    3. Check the connectivity between the vSAN VM kernel adapters in the two availability zones and the witness host by following the instructions in KB article 1003728. Resolve errors, if any, before proceeding to the next step.
    4. Configure vSAN Stretched Cluster for the Management Cluster in Region A.

      In step 5 of the section "Update the vSphere High Availability Settings of the Management Cluster in Region A", set Host failures cluster tolerates to the number of hosts in AZ1.

      Note: Skip the Update Host Profiles section to capture the vSAN stretched cluster configuration.
  5. You can stretch the cluster on either Layer 2 (L2) networks or on Layer 3 (L3) networks.
    1. To stretch the cluster for Layer 2 (L2) networks, run the following command:
      /opt/vmware/sddc-support/sos --l2-stretch --stretch-vsan --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn <witness host/appliance IP or fqdn> --witness-vsan-ip <witness vsan IP address> --witness-vsan-cidr <witness-vsan-network-IP-address-with-mask>
      Note: --witness-host-fqdn accepts either an IP address or an FQDN. If you deployed the witness host in Step 4 using an FQDN, enter the FQDN. If you deployed the witness host using an IP address, enter the IP address.
      Enter the inputs for the following:
      • esxi host passwords
      • vsan gateway IP
      • vSAN CIDR

      For example:

      "/opt/vmware/sddc-support/sos --l2-stretch --stretch-vsan --sc-domain MGMT --sc-cluster VxRail-Virtual-SAN-Cluster-4fb50f56-953b-4acb-b9aa-de3f664126f6 --sc-hosts dr27b-011.rainpole.local --witness-host-fqdn 172.27.160.106 --witness-vsan-ip 172.27.164.106 --witness-vsan-cidr 172.27.164.0/22
      Welcome to Supportability and Serviceability(SoS) utility!
      Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-10-22-07-24-05-36774
      Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-10-22-07-24-05-36774/sos.log
      Starting vSAN stretched cluster operations..
      Initiating L2 vSAN stretch operation
      [**IMPORTANT**] Please make sure passwords are correct for each esxi host!!
      * Please provide root user password for host dr27b-011.rainpole.local :
      * Please confirm root user password for host dr27b-011.rainpole.local :
      Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.18.94.1
      Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.18.94.0/24
      Api Response:{"id":"6b168b3f-2df8-4c84-a431-0a8c7a9ecc6d","link":null,"taskId":"6b168b3f-2df8-4c84-a431-0a8c7a9ecc6d","resourceId":"330c0c4a-8b4c-475d-8873-9d8db4329754","resourceType":"ESXI","state":"IN_PROGRESS","description":"Extends VxRail vSAN cluster from a single data site to two sites","errors":null,"timestamp":1571729075072}
      Workflow triggered, please track the task status in SDDC Manager UI"
    2. To stretch the cluster for Layer 3 (L3) networks, run the following command:

      /opt/vmware/sddc-support/sos --l3-stretch --stretch-vsan --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn <witness host/appliance IP or fqdn> --witness-vsan-ip <witness vsan IP address> --witness-vsan-cidr <witness-vsan-network-IP-address-with-mask>

      Enter the inputs for the following:
      • esxi host passwords
      • vsan gateway IP for the preferred(primary) and non-preferred(secondary) site
      • vSAN CIDR for the preferred(primary) and non-preferred(secondary) site
      • nsx vlan id

      For example:

      root@wdc1sddc-1 [ /home/.feature ]# /opt/vmware/sddc-support/sos --l3-stretch --stretch-vsan --sc-domain wld-1 --sc-cluster VxRail-Virtual-SAN-Cluster --sc-hosts wdc1-005.vxrail.local,wdc1-006.vxrail.local --witness-host-fqdn 172.16.10.125 --witness-vsan-ip 172.16.11.222 --witness-vsan-cidr 172.16.11.0/24
      Welcome to Supportability and Serviceability(SoS) utility!
      Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-06-12-18-02-65677
      Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-06-12-18-02-65677/sos.log
      Starting vSAN stretched cluster operations..
      Initiating L3 vSAN stretch operation
      [**IMPORTANT**] Please make sure passwords are correct for each esxi host!!
      * Please provide root user password for host wdc1-005.vxrail.local :
      * Please confirm root user password for host wdc1-005.vxrail.local :
      * Please provide root user password for host wdc1-006.vxrail.local :
      * Please confirm root user password for host wdc1-006.vxrail.local :
      ** Please enter Preferred(Primary) site network information
      Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.43.253
      Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.43.0/24
      ** Please enter Non-Preferred(secondary) site network information
      Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.11.253
      Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.11.0/24
      Please enter Preferred site Nsx Vlan Id? (ex: 800): 2057
      Api Response:{"id":"7fb4a115-e519-4eda-a3b7-5a7a928fc2da","link":null,"taskId":"7fb4a115-e519-4eda-a3b7-5a7a928fc2da","resourceId":"741cd932-a01d-4fd8-af2a-33c0a5b0317e","resourceType":"ESXI","state":"IN_PROGRESS","description":"Extends VxRail vSAN cluster from a single data site to two sites","errors":null,"timestamp":1573042747188}
      Workflow triggered, please track the task status in SDDC Manager UI
    Note:
    • Ensure that the passwords are correct for each host.
    • For --sc-hosts <valid host names>, ensure that the multiple host names are separated by commas.
    • Ensure that the witness host ip or fqdn should match to how it is managed in vCenter. For example, if the witness host is managed using IP address in the vCenter Server, then the IP address should be provided and if the witness host is managed using FQDN in the vCenter Server, then FQDN should be provided.
  6. Once the workflow is triggered, the task is tracked in the SDDC Manager UI.
  7. Monitor the progress of the AZ2 hosts being added to the cluster.
    1. On the SDDC Manager Dashboard, click View All Tasks.
    2. Refresh the window to monitor the status.
  8. Validate that stretched cluster operations are working correctly by logging in to the vSphere Web Client.
    1. Verify the vSAN Health page.
      1. On the home page, click Host and Clusters and then select the stretched cluster (SDDC-Cluster1 in our example).
      2. Click Monitor > vSAN > Health.
      3. Click Retest.
      4. Fix errors, if any.
    2. Verify the vSAN Storage Policy page.
      1. On the home page, click Policies and Profiles > VM Storage Policies > vSAN Default Storage Policies .
      2. Select the policy associated with the vCenter Server for the stretched cluster.
      3. Click Monitor > VMs and Virtual Disks.
      4. Click Refresh.
      5. Click Trigger VM storage policy compliance check
      6. Verify the Compliance Status column for each VM component.
      7. Fix errors, if any.
  9. You can expand the stretched cluster either on Layer 2 (L2) networks or on Layer 3 (L3) networks.
    1. Use the VxRail vCenter plugin to add the additional hosts in AZ1 or AZ2 to the cluster by performing the VxRail Manager cluster expansion work flow. Refer to the Dell EMC VxRail documentation for more details.
    2. Log in to SDDC Manager and run the SoS tool to trigger the workflow to import the newly added hosts in the SDDC Manager inventory.
      In the SoS tool, provide the root credential and the fault domain to which the host to be added for each host.
    3. To expand the stretched cluster for Layer 2 (L2) networks, run the following SoS command :
      /opt/vmware/sddc-support/sos --l2-stretch --expand-stretch-cluster --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn < witness host/appliance IP or fqdn> --witness-vsan-ip <witness-vsan-network-IP-address-with-mask> --witness-vsan-cidr <IP address with mask> --vsan-gateway-ip <host-vsan-gateway-ip-address>
      Note: --witness-host-fqdn accepts either an IP address or an FQDN. If you deployed the witness host in Step 4 using an FQDN, enter the FQDN. If you deployed the witness host using an IP address, enter the IP address.

      For both stretch and expand workflows, once the SoS command triggers, it prompts for passwords for the hosts given as inputs so you have to keep them ready in advance. In case of the expand workflow, you have to provide the fault domain information as an input for hosts. So keep the fault domain information ready.

      Enter the inputs for the following:
      • esxi host passwords
      • fault domain for the hosts
      • vSAN CIDR

      Note:
      • Ensure that you have the fault domain information (preferred fault domain information) for the hosts.
      • Ensure that the passwords are correct for each host.
      • For --sc-hosts <valid host names>, ensure that the multiple host names are separated by commas.
      • Ensure that the witness host ip or fqdn should match to how it is managed in vCenter. For example, if the witness host is managed using IP address in the vCenter Server, then the IP address should be provided and if the witness host is managed using FQDN in the vCenter Server, then FQDN should be provided.
    4. To expand the stretched cluster for Layer 3 (L3) networks, run the following SoS command :
      /opt/vmware/sddc-support/sos --l3-stretch --expand-stretch-cluster --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn < witness host/appliance IP or fqdn> --witness-vsan-ip <witness-vsan-network-IP-address-with-mask> --witness-vsan-cidr <IP address with mask> --vsan-gateway-ip <host-vsan-gateway-ip-address>

      For both stretch and expand workflows, once the SoS command triggers, it prompts for passwords for the hosts given as inputs so you have to keep them ready in advance. In case of the expand workflow, you have to provide the fault domain information as an input for hosts. So keep the fault domain information ready.

      Enter the inputs for the following:
      • esxi host passwords
      • vsan gateway IP for the preferred(primary) and non-preferred(secondary) site
      • vSAN CIDR for the preferred(primary) and non-preferred(secondary) site
      • nsx vlan id

      For example:

      root@wdc1sddc-1 [ /home/vcf ]# /opt/vmware/sddc-support/sos --l3-stretch --expand-stretch-cluster --sc-domain wld-1 --sc-cluster VxRail-Virtual-SAN-Cluster --sc-hosts wdc1-010.vxrail.local,wdc3-008.vxrail.local --witness-host-fqdn 172.16.10.125 --witness-vsan-ip 172.16.11.222 --witness-vsan-cidr 172.16.11.0/24
      Welcome to Supportability and Serviceability(SoS) utility!
      Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147
      Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147/sos.log
      Starting vSAN stretched cluster operations..
      Initiating L3 expand vSAN stretch operation
      [**IMPORTANT**]
              * Please make sure passwords are correct for each esxi host!!
              * Please keep fault-domain info handy for hosts!!
      * Please provide root user password for host wdc1-010.vxrail.local :
      * Please confirm root user password for host wdc1-010.vxrail.local :
      * Please provide fault domain for host wdc1-010.vxrail.local :VxRail-Virtual-SAN-Cluster_az2-faultdomain
      * Please provide root user password for host wdc3-008.vxrail.local :
      * Please confirm root user password for host wdc3-008.vxrail.local :
      * Please provide fault domain for host wdc3-008.vxrail.local :VxRail-Virtual-SAN-Cluster_az1-faultdomain
      ** Please enter Preferred(Primary) site network information
      Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.43.253
      Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.43.0/24
      ** Please enter Non-Preferred(secondary) site network information
      Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.11.253
      Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.11.0/24
      Api Response:{"id":"c63358c5-b811-4394-b08e-ad4a42c06c19","link":null,"taskId":"c63358c5-b811-4394-b08e-ad4a42c06c19","resourceId":"2271600f-aee2-4df1-85e3-1e65adc075fa","resourceType":"ESXI","state":"IN_PROGRESS","description":"Expands VxRail vSAN stetch cluster","errors":null,"timestamp":1573556282386}
      Workflow triggered, please track the task status in SDDC Manager UI
      Note:
      • Ensure that you have the fault domain information (preferred fault domain information) for the hosts.
      • Ensure that the passwords are correct for each host.
      • For --sc-hosts <valid host names>, ensure that the multiple host names are separated by commas.
      • Ensure that the witness host ip or fqdn should match to how it is managed in vCenter. For example, if the witness host is managed using IP address in the vCenter Server, then the IP address should be provided and if the witness host is managed using FQDN in the vCenter Server, then FQDN should be provided.
    5. Once the workflow is triggered, track the task status in the SDDC Manager UI.

What to do next

Deploy two ECMP-enabled NSX Edge devices and enable North-South routing in Availability Zone 2. See Configure NSX Dynamic Routing for Availability Zone 2 in Region A.