Three Node Appliance Database Automatic Failover Scenarios

There are several appliance database high availability failover scenarios, and vRealize Automation behavior varies depending on appliance database configuration and the number of nodes that fail.

Single Node Failure Scenarios

If one of the three nodes fails, vRealize Automation will initiate an auto failover. No additional auto failover operations can occur until all three nodes are restored.

The following table describes behavior and actions related to a master node failure in a high availability deployment.

Table 1. The Master Node Fails

Expected Behavior	The configured sync replica node becomes the master and automatically picks up appliance database functionality. The potential sync replica becomes the sync standby node. The vRealize Automation deployment functions in read only mode until the automatic failover completes.
Further Action	When the former master is recovered, it will be reset as replica automatically by the failover agent repair logic. No manual action is required. If the former master cannot be recovered, manually set the appliance database to asynchronous mode.

The following table describes behavior and actions related to a sync replica node failure in a high availability deployment.

Table 2. The Sync Replica Fails

Expected Behavior	The vRealize Automation deployment experiences no downtime. There will be a delay of a couple of seconds for database requests until the potential replica becomes the new sync replica. The appliance database performs this action automatically.
Further Action	When the former synch replica comes online, it will become a potential replica automatically. No manual action is required. If the former sync replica cannot be repaired, manually set the appliance database to asynchronous mode.

The following table describes behavior and actions related to a master node failure in a high availability deployment.

Table 3. The Potential Replica Fails

Expected Behavior	No deployment downtime.
Further Action	When the former potential replica comes online, it becomes a potential replica automatically. No manual action is required. If the former potential replica cannot be repaired, set the appliance database to asynchronous mode.

Two Node Failure Scenarios

If two out of the three nodes fail simultaneously, vRealize Automation switches to read only mode until a manual repair is performed.

The following table describes behavior and actions related to a master node and potential replica node failure in a high availability deployment.

Table 4. The Master Node and Potential Replica Fail

Expected Behavior	The sync replica is not promoted to master automatically. vRealize Automation will switch to read only mode until a manual promotion is performed.
Further Action	Manual promotion is required. Set the appliance database to asynchronous mode. When the master and potential replica are recovered, manually set them to synchronize against the new master. At that point, you can switch vRealize Automation back to synchronous mode. When two out of three nodes are down simultaneously, vRealize Automation will switch to read-only mode until you effect a manual repair. If only one database node is available, switch your deployment to asynchronous mode.

The following table describes behavior and actions related to Sync and Potential node failure in a high availability deployment.

Table 5. The Sync and Potential Replicas Fail

Expected Behavior	The master is unable to process read/write transactions. vRealize Automation functions in read only mode until a manual repair is performed.
Further Action	Manual promotion is required. Set the appliance database to asynchronous mode. When the sync and potential replicas are recovered, they should be manually reset to synchronize against the master. At this point, you can switch vRealize Automation back to synchronous mode. When two out of three nodes are down simultaneously, vRealize Automation will switch to read-only mode until you effect a manual repair. If only one database node is available, switch your deployment to asynchronous mode.

Links Failures Among Nodes

If a link failure occurs among nodes on a distributed deployment, the automatic failover agent attempts to repair the configuration.

The following table describes behavior and actions related to a link failure between two sites in a high availability deployment with the specified configuration when all nodes remain up and online.

Site A: Master and potential replica

Site B: Sync replica

Table 6. Link Failure Between Two Sites when all Nodes Remain Up and Online

Expected Behavior	No downtime for the vRealize Automation deployment. The potential replica automatically becomes the sync replica.
Further Action	No manual action is required.

The following table describes behavior and actions related to a link failure between two sites in a high availability deployment with the specified configuration when all nodes remain up and online.

Site A: Master

Site B: Sync and potential replica

Table 7. Link Failure Between Two Sites when all Nodes Remain Up and Online - Alternate Configuration

Expected Behavior	Sync replica becomes the master and automatically picks up appliance database functionality. Automatic failover agent promotes the potential replica to become the new sync replica. vRealize Automation deployment operates in read only mode until this promotion completes.
Further Action	No manual action is required. When the link is recovered, the automatic failover agent resets the former master as replica.