If a host or host component in a stretched cluster fails, it is recommended that you replace the host with a new host.

Prerequisites

  • Image the replacement host with the same ESXi version as the other hosts in the cluster.
  • Check the health of the cluster.

    See "Check vSAN Health" in Administering VMware vSAN.

Procedure

  1. Remove the failed host from the cluster.
    See Remove a Host from a Cluster in a Workload Domain
  2. Decommission the host.
    See Decommission Hosts.
  3. Commission the replacement host to the same network pool as the removed host.
    See Commission Hosts .
  4. SSH in to the SDDC Manager VM using the vcf administrative user account.
  5. Enter su to switch to the root user and navigate to the /opt/vmware/sddc-support directory.
  6. Enter the following command: 
    ./sos --expand-stretch-cluster --sc-domain <DOMAIN NAME> --sc-cluster <CLUSTER NAME> --sc-hosts <REPLACEMENT HOSTFQDN> --esxi-license-key <LICENSE KEY>
    Example input and response: 
    root@sddc-manager [ /opt/vmware/sddc-support ]# ./sos --expand-stretch-cluster --sc-domain MGMT --sc-cluster SDDC-Cluster1 --sc-hosts esxi-11.vrack.vsphere.local --esxi-license-key AAAAA-BBBBB-CCCCC-DDDDD-EEEEE
Welcome to Supportability and Serviceability(SoS) utility!
Logs : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-02-20-10-04-32-123007
Stretch Cluster operation log : /var/log/vmware/vcf/sddc-support/stretchCluster-2019-02-20-10-04-32-123007/sos.log
Starting vSAN stretched cluster operations..
expand vSAN stretched cluster operation started
Api Response:{
  "taskId": "6e4b13d9-eead-408b-a595-4e89ef885a3e",
  "resourceId": "0c518498-b302-40ae-abc4-10addead7bc2",
  "resourceType": "ESXI",
  "state": "IN_PROGRESS",
  "description": "Prepare vSAN Cluster - SDDC-Cluster1 for Stretch in VMware Cloud Foundation",
  "timestamp": 1550657073615,
  "id": "6e4b13d9-eead-408b-a595-4e89ef885a3e"
}
  7. Monitor the state of the task in the SDDC Manager Dashboard.
    Wait until the task completes successfully.
  8. If required, SSH in to the newly added host and add a static route to the vSAN network of the witness host. Add static routes in the witness if it could not reach the vSAN network of the newly added host.
  9. In the vSphere Web Client, move the host to the appropriate availability zone.
    1. On the SDDC Manager Dashboard, click Inventory > Workload Domains and then click View Details.
    2. Click the name of the domain containing the stretched cluster, for example, MGMT.
    3. Click the Services tab and click the vCenter Server launch icon and log in to the vSphere Web Client.
    4. In the vSphere Web Client, select the stretched cluster. Then select Configure > vSAN > Fault Domains & Stretched Cluster.
    5. Select the newly added host and drag it to the appropriate availability zone.
  10. If the host belongs to AZ1, add the host to the AZ1 VMHost rule. If the host belongs to AZ2, no operation is required.
    1. In the vSphere Web Client, select Hosts and Clusters and then select the stretched cluster.
    2. Select Configure > VM/Host Rules.
    3. Select the appropriate rule and click Add.
    4. Select the newly added host and click OK.

Results

vSAN automatically rebuilds the stretch cluster.