Recover from a Primary Database Failure in a High Availability Cluster

If the primary cell is not running properly, to recover the vCloud Director database, you can promote one of the standby cells to become the new primary cell. After that, you must deploy a new standby cell.

You can use this workflow to reuse the IP addresses and hostname of the failed primary when you deploy the new standby.

Prerequisites

• Verify that the primary cell is in the not reachable or failed state.
• Verify that the two standby cells are in the running state.
• Familiarize yourself with the procedure to remove a failed appliance from the vCloud Director server group and the repmgr high availability cluster. See Delete a Cloud Cell and Unregister a Failed Primary Cell in a Database High Availability Cluster.

See View the Status of the Cells in a Database High Availability Cluster.

Procedure

1. If possible, use the cell management tool to shut down the vCloud Director process. From the failed primary cell, run the following command.

   /opt/vmware/vcloud-director/bin/cell-management-tool -u <admin username> cell --shutdown

2. Power off the failed primary VM.
3. Log in as root to the appliance management user interface of a running standby cell, https://standby_ip_address:5480.
4. In the Role column for the standby cell that you want to become the new primary cell, click Promote.
   The cell becomes the new primary cell in running state. The other standby cell is following the newly promoted primary cell.
5. Remove the failed primary appliance from both the vCloud Director server group and the repmgr high availability cluster.
6. If you want to reuse the IP address and hostname of the failed primary, ensure that the failed primary appliance remains powered off or delete it.
7. Deploy a new standby appliance. You can deploy the appliance by using the vSphere Client or deploy the appliance by using the VMware OVF Tool.

   After deploying the new standby appliance, the cluster health must be Healthy.

   Important: The OVA version of the newly deployed standby must be the same as the existing appliances in the cluster. To view the version of running appliances, see the About information in the appliance management UI. The appliance is distributed with a name of the form VMware Cloud Director-v.v.v.v- nnnnnn_ OVF10.ova, where v. v. v. v represents the product version and nnnnnn the build number. For example: VMware Cloud Director-10.0.0.0-9229800_OVA10.ova.