Failover system tests

A comprehensive set of scenarios must be developed and used to test the Failover System in various situations. The scenarios should include, for example:

• Active component is Up; Standby component is Up, Down, Unresponsive, Unreachable

• Active component is Down; Standby component is Up, Down, Unresponsive, Unreachable

• Active component is Unresponsive; Standby component is Up, Down, Unresponsive, Unreachable

• Host on which the Active component resides is Unresponsive; Standby component is Up, Down, Unresponsive, Unreachable

• Location of the Active component is Unreachable; Standby component is Up, Down, Unresponsive, Unreachable

  During the tests, email should be sent to the specified personnel. Type of email include:

• Informational email — Failure on Standby server, host, or location. This is to inform administrators of a failed Standby component although the Active component is functioning as expected and the whole system is unaffected. It provides an opportunity for administrators to fix the problem before the Active component fails and a smooth failover can be achieved when required.

• Warning email — Error during backup actions of component <serviceName>:... Sometimes, a launched action may fail because of timing where the resources may be abruptly unavailable, such as a location that becomes unreachable, or an Active or Standby component that goes Down.

• Failover email — A failover has occurred.

• Failover Error email — There is a failure on an Active component or host, but the system cannot failover because of one or more failed Standby components.