From Release 3.9 onwards, VMware Cloud Foundation on VxRail enable L3 awareness, but the management VLAN must still be stretched between AZs.

For more information on the availability zones, see About Availability Zones and Regions.
Note:
  • Command such as --show-free-hosts operations is not applicable for the Dell EMC VxRail environment. If you run these commands, 'Operation is not applicable for this platform!!' warning is thrown.
  • The Import VxRail Host for cluster operation is unavailable for a stretched cluster.
The details for the Cloud Foundation networks for Layer 3 are as follows:
Network Name Connectivity to AZ2 Minimum MTU Maximum MTU
vSAN L3 1500 9000
vMotion L3 1500 9000
Tunnel End Points (TEP) L3 1600 9000
Management L2 1500 9000
Witness Management L3 1500 9000
Witness vSAN L3 1500 9000

To stretch a cluster for VMware Cloud Foundation on Dell EMC VxRail, perform the following steps:

Procedure

  1. Using SSH, log in to the SDDC Manager VM with the user name vcf and the password you specified in the deployment parameter sheet.
  2. Prepare the workflow. Use the SoS commands to prepare the cluster. See SoS Utility Options for vSAN Stretched Clusters in SoS Utility Options.
    /opt/vmware/sddc-support/sos --prepare-stretch --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched>
    Once the workflow is triggered, track the task status in the SDDC Manager UI.
  3. You have to deploy the witness in a different site. Add the witness host or the appliance to the management or the workload domain vCenter. Follow the steps listed below as described in Deploying a VSAN Witness Appliance to add a vSAN witness.
    1. Deploy and Configure the vSAN Witness Host in Region B.
    2. Add Static Routes for both Availability Zones and the vSAN Witness Host.

      If the default gateway in the vSAN network provided for the network pool does not provide routing between the two availability zones and the witness host, perform all the steps in this procedure.

    3. Check the connectivity between the vSAN VM kernel adapters in the two availability zones and the witness host by following the instructions in KB article 1003728. Resolve errors, if any, before proceeding to the next step.
    4. " Configure vSAN Stretched Cluster for the Management Cluster in Region A"

      In step 5 of the section "Update the vSphere High Availability Settings of the Management Cluster in Region A", set Host failures cluster tolerates to the number of hosts in AZ1.

      Note: Skip the section Update Host Profiles to Capture the vSAN Stretched Cluster Configuration.
  4. Run the following command to stretch the cluster for Layer 3 (L3) networks:

    /opt/vmware/sddc-support/sos --l3-stretch --stretch-vsan --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn <witness host/appliance IP or fqdn> --witness-vsan-ip <witness vsan IP address> --witness-vsan-cidr <witness-vsan-network-IP-address-with-mask>

    Enter the inputs for the following:
    • esxi host passwords
    • vsan gateway IP for the preferred(primary) and non-preferred(secondary) site
    • vSAN CIDR for the preferred(primary) and non-preferred(secondary) site
    • nsx vlan id

    For example: 

    root@wdc1sddc-1 [ /home/vcf ]# /opt/vmware/sddc-support/sos --l3-stretch --expand-stretch-cluster --sc-domain wld-1 --sc-cluster VxRail-Virtual-SAN-Cluster --sc-hosts wdc1-010.vxrail.local,wdc3-008.vxrail.local --witness-host-fqdn 172.16.10.125 --witness-vsan-ip 172.16.11.222 --witness-vsan-cidr 172.16.11.0/24
Welcome to Supportability and Serviceability(SoS) utility!
Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147
Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147/sos.log
Starting vSAN stretched cluster operations..
Initiating L3 expand vSAN stretch operation
[**IMPORTANT**]
        * Please make sure passwords are correct for each esxi host!!
        * Please keep fault-domain info handy for hosts!!
* Please provide root user password for host wdc1-010.vxrail.local :
* Please confirm root user password for host wdc1-010.vxrail.local :
* Please provide fault domain for host wdc1-010.vxrail.local :VxRail-Virtual-SAN-Cluster_az2-faultdomain
* Please provide root user password for host wdc3-008.vxrail.local :
* Please confirm root user password for host wdc3-008.vxrail.local :
* Please provide fault domain for host wdc3-008.vxrail.local :VxRail-Virtual-SAN-Cluster_az1-faultdomain
** Please enter Preferred(Primary) site network information
Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.43.253
Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.43.0/24
** Please enter Non-Preferred(secondary) site network information
Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.11.253
Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.11.0/24
Api Response:{"id":"c63358c5-b811-4394-b08e-ad4a42c06c19","link":null,"taskId":"c63358c5-b811-4394-b08e-ad4a42c06c19","resourceId":"2271600f-aee2-4df1-85e3-1e65adc075fa","resourceType":"ESXI","state":"IN_PROGRESS","description":"Expands VxRail vSAN stetch cluster","errors":null,"timestamp":1573556282386}
Workflow triggered, please track the task status in SDDC Manager UI
  5. Expand cluster with Availability Zone 2 hosts in vCenter. Reimage and power on the hosts that are rack mounted in Availability Zone 2 (or Region B). See Cluster Spanning for VMware Cloud Foundation on VxRail for more information on expanding clusters that need cluster spanning.
    1. Use the VxRail vCenter plugin to add the additional hosts in AZ1 or AZ2 to the cluster by performing the VxRail Manager cluster expansion work flow. Refer to the Dell EMC VxRail documentation for more details.
    2. Log in to SDDC Manager and run the SoS tool to trigger the workflow to import the newly added hosts in the SDDC Manager inventory.
      In the SoS tool, provide the root credential and the fault domain to which the host to be added for each host.
    3. Run the following SoS command to expand the stretched cluster for Layer 3 (L3) networks:
      /opt/vmware/sddc-support/sos --l3-stretch --expand-stretch-cluster --sc-domain <SDDC-valid-domain-name> --sc-cluster <valid cluster name which is a part of the domain to be stretched> --sc-hosts <valid host names> --witness-host-fqdn < witness host/appliance IP or fqdn> --witness-vsan-ip <witness-vsan-network-IP-address-with-mask> --witness-vsan-cidr <IP address with mask> --vsan-gateway-ip <host-vsan-gateway-ip-address>

      For both stretch and expand workflows, once the SoS command triggers, it prompts for passwords for the hosts given as inputs so you have to keep them ready in advance. In case of the expand workflow, you have to provide the fault domain information as an input for hosts. So keep the fault domain information ready.

      Enter the inputs for the following:
      • esxi host passwords
      • vsan gateway IP for the preferred(primary) and non-preferred(secondary) site
      • vSAN CIDR for the preferred(primary) and non-preferred(secondary) site
      • nsx vlan id

      For example:

      root@wdc1sddc-1 [ /home/vcf ]# /opt/vmware/sddc-support/sos --l3-stretch --expand-stretch-cluster --sc-domain wld-1 --sc-cluster VxRail-Virtual-SAN-Cluster --sc-hosts wdc1-010.vxrail.local,wdc3-008.vxrail.local --witness-host-fqdn 172.16.10.125 --witness-vsan-ip 172.16.11.222 --witness-vsan-cidr 172.16.11.0/24
Welcome to Supportability and Serviceability(SoS) utility!
Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147
Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-11-12-10-56-26-88147/sos.log
Starting vSAN stretched cluster operations..
Initiating L3 expand vSAN stretch operation
[**IMPORTANT**]
        * Please make sure passwords are correct for each esxi host!!
        * Please keep fault-domain info handy for hosts!!
* Please provide root user password for host wdc1-010.vxrail.local :
* Please confirm root user password for host wdc1-010.vxrail.local :
* Please provide fault domain for host wdc1-010.vxrail.local :VxRail-Virtual-SAN-Cluster_az2-faultdomain
* Please provide root user password for host wdc3-008.vxrail.local :
* Please confirm root user password for host wdc3-008.vxrail.local :
* Please provide fault domain for host wdc3-008.vxrail.local :VxRail-Virtual-SAN-Cluster_az1-faultdomain
** Please enter Preferred(Primary) site network information
Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.43.253
Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.43.0/24
** Please enter Non-Preferred(secondary) site network information
Please enter vSAN Gateway IP? (ex: 172.18.93.1): 172.16.11.253
Please enter vSAN CIDR? (ex: 172.18.93.0/24): 172.16.11.0/24
Api Response:{"id":"c63358c5-b811-4394-b08e-ad4a42c06c19","link":null,"taskId":"c63358c5-b811-4394-b08e-ad4a42c06c19","resourceId":"2271600f-aee2-4df1-85e3-1e65adc075fa","resourceType":"ESXI","state":"IN_PROGRESS","description":"Expands VxRail vSAN stetch cluster","errors":null,"timestamp":1573556282386}
Workflow triggered, please track the task status in SDDC Manager UI
      Note:
      • Ensure that you have the fault domain information (preferred fault domain information) for the hosts.
      • Ensure that the passwords are correct for each host.
      • For --sc-hosts <valid host names>, ensure that the multiple host names are separated by commas.
    4. Once the workflow is triggered, track the task status in the SDDC Manager UI.
  6. Monitor the progress of the AZ2 hosts being added to the cluster.
    1. On the SDDC Manager Dashboard, click View All Tasks.
    2. Refresh the window to monitor the status.
  7. Validate that stretched cluster operations are working correctly by logging in to the vSphere Web Client.
    1. Verify the vSAN Health page.
      1. On the home page, click Host and Clusters and then select the stretched cluster (SDDC-Cluster1 in our example).
      2. Click Monitor > vSAN > Health.
      3. Click Retest.
      4. Fix errors, if any.
    2. Verify the vSAN Storage Policy page.
      1. On the home page, click Policies and Profiles > VM Storage Policies > vSAN Default Storage Policies .
      2. Select the policy associated with the vCenter Server for the stretched cluster.
      3. Click Monitor > VMs and Virtual Disks.
      4. Click Refresh.
      5. Click Trigger VM storage policy compliance check
      6. Check the Compliance Status column for each VM component.
      7. Fix errors, if any.
  8. You can shrink the stretched cluster by removing the nodes from the cluster.
    1. Switch the VxRail hosts that are part of Availability Zone 2 into the maintenance mode.
    2. Remove the hosts that are part of Availability Zone 2 from the host group.
    3. Remove the host group created for Availability Zone 2.
    4. Remove every removed host that is part of Availability Zone 2 from SDDC Manager.
    5. Remove every removed host that is part of Availability Zone 2 from vCenter VxRail UI.
    6. Log in to each ESXi server that is a part of Availability Zone 1 and remove the static routes to reach Witness vSAN host/appliance.
      See the following knowledge base article to configure the static route on the ESXi host:

      https://kb.vmware.com/s/article/2001426

    7. In vCenter, remove the vSphere Availability settings that were made while stretching the cluster.

What to do next

Deploy two ECMP-enabled NSX Edge devices and enable North-South routing in Availability Zone 2. See Configure NSX Dynamic Routing for Availability Zone 2 in Region A.