Database Failover - Legacy Appliances

When the primary appliance fails, you can perform a database failover. This topic describes the procedure for appliances running versions older than 9.0.0.

Note: This topic only applies to appliances running versions older than 9.0.0. For information pertinent to newer appliances, see Database Failover - Newer Appliances.

When the primary appliance fails, the secondary database is read-only. When failover occurs, perform the following tasks:

Enable write operations on the secondary database
Permanently promote the secondary service provider appliance to become the primary
Permanently promote the secondary tenant appliance to become the primary
Restart the primary appliance

Enable Write Operations on the Secondary Database

The goal of this procedure is to switch the primary database to the secondary appliance (tenant or service provider) if the primary appliance is not available. The goal is to enable write operations so that the secondary appliance's database is the primary datasource.

Stop the dtService, av-manager, and diagnose service on both the primary (if accessible) and the secondary appliance:
```
sudo service dtService stop
sudo service av-manager stop
sudo service diagnose stop
```
Stop all slony daemons on both the primary (if accessible) and the secondary appliances:
```
sudo killall slon
```
On the secondary appliance, connect to the fdb database and execute the following SQL command:
```
 drop schema _slony cascade;
```
Repeat step 3 for the EDB and AVDB if the appliance belongs to a tenant organization.
If the database on the primary appliance is still accessible, then backup the database, copy the database files, and restore the database into the secondary appliance (see Backing Up and Restoring Databases).
Open the file /usr/local/desktone/release/active/conf/fdb.properties for edit and remove the IP address of the primary appliance.
Repeat step 6 for /usr/local/desktone/release/active/edb.properties if the appliance belongs to a tenant organization.
Repeat step 6 for /usr/local/desktone/release/active/avdb.properties if the appliance belongs to a tenant organization.

Set DB_HOST and DB_PASSWORD environment variables:

export DB_HOST=IP_of_TA2_appliance
export DB_PASSWORD=database_password

Execute av-setup script as sudo:

sudo /usr/local/desktone/scripts/av-setup

Open the applications.properties file for editing:

vi /usr/local/xmpms/diagnose/config/application.properties

In the applications.properties file, edit lines as follows:

db.jdbc.url=jdbc:postgresql://<ip address>:5432/avdb?ssl=true
db.fdb.jdbc.url=jdbc:postgresql://<ip address>:5432/fdb?ssl=true

Start dtService, av-manager, and diagnose service on the secondary appliance:
```
service dtService start
service av-manager start
service diagnose start
```

Promote the Secondary Service Provider Appliance to Primary

To permanently promote the secondary service provider appliance to be the primary service provider appliance, perform the following steps.

In this example, the primary appliance is 'A', and the secondary appliance is 'B', and appliance A is accessible (database in this appliance is accessible).

Stop dtService on all the service provider appliances:
```
service dtService stop
```
Log into all the service provider appliances.
Open the file /usr/local/desktone/release/active/conf/fdb.properties for edit and remove the IP address of the primary appliance that is in failed status.
Stop all slony daemons on all service provider appliances:
```
killall slon
```
On the primary and secondary appliance, connect to the FDB database and execute the following SQL command:
```
drop schema _slony cascade;
```

Execute the following SQL commands in secondary appliance of datacenter (replace 'A' and 'B' with names of your appliances):

fdb=# update appliance set capabilities = (capabilities & 65343) where name='A'; 
fdb=# update appliance set capabilities = (capabilities | 192) where name='B';

Start dtService on the service provider appliances other than the failed appliance:
```
service dtService start
```
Confirm that the appliance is up and running before proceeding with the next step.
Stop dtService on all the service provider appliances except for the primary service provider appliance (the secondary service provider appliance is the primary):
```
service dtService stop
```
Start slon daemons in all of the service provider appliances:
```
/usr/local/desktone/scripts/start_slon_fdb.sh
```
Login to the Service Center of primary service provider appliance (the secondary service provider appliance is the primary) with your browser and perform the slony reinitialization operation:
1. Navigate to appliances > maintenance > Slony Operations.
2. In the Organization id drop-down list, select 1000.
3. Click on the Init Slony button to reinitialize slony.

When the slony re-initialization is complete, execute the following command in the FDB of the secondary appliance in the datacenter:

SELECT a.set_id, a.set_comment, (SELECT last_value FROM _slony.sl_local_node_id) AS local_id, CASE WHEN a.set_origin = (SELECT  last_value FROM _slony.sl_local_node_id)THEN TRUE ELSE FALSE END AS master_node from _slony.sl_set a;

Output should appear as follows:

set_id | set_comment | local_id | master_node
--------+--------------------------+----------+-------------   
1 | All tables and Sequences | 1 | t  
(1 row)

Start dtService on all appliances except for the failed service provider appliance.
```
service dtService start
```

Promote the Secondary Tenant Appliance to Primary

To permanently promote the secondary tenant appliance to be the primary tenant appliance, perform the following steps.

In this example, the primary appliance is 'A', and the secondary appliance is 'B', and appliance A is accessible (database in this appliance is accessible).

Stop dtService on all tenant appliances in the organization:
```
service dtService stop
```
Log into all the tenant appliances in the organization.
Open the file /usr/local/desktone/release/active/conf/fdb.properties for edit and remove the IP address of the primary appliance that is in failed status.
Stop all slony daemons on all tenant appliances in the organization:
```
killall slon
```
If required, back up the database on the primary appliance and restore it on the secondary appliances.
In the primary and secondary appliances, connect to the FDB database and execute the following SQL command:
```
drop schema _slony cascade;
```

Execute the following SQL commands on the primary service provider appliance (replace 'A' and 'B' with names of your appliances):

fdb=# update appliance set capabilities = (capabilities & 65343) where name='A';  
fdb=# update appliance set capabilities = (capabilities | 192) where name='B';

Start slon daemons on all of the tenant appliances in organization:

/usr/local/desktone/scripts/start_slon_fdb.sh  
/usr/local/desktone/scripts/start_slon_edb.sh  
/usr/local/desktone/scripts/start_slon_avdb.sh

Login to the Service Center of the primary service provider appliance with your browser and perform the slony reinitialization operation:
1. Navigate to appliances > maintenance > Slony Operations.
2. Select organization ID for the tenant from the Organization id drop-down list.
3. Select Fabric for the DB instance name label.
4. Click on the Init Slony button to reinit slony.

When the slony re-initialization is complete, execute following command in the FDB of the secondary tenant appliance in the datacenter:

SELECT a.set_id, a.set_comment, (SELECT last_value FROM _slony.sl_local_node_id) AS local_id, CASE WHEN a.set_origin = (SELECT  last_value FROM _slony.sl_local_node_id) THEN TRUE ELSE FALSE END AS master_node from _slony.sl_set a;

Output should appear as follows (if the primary appliance was failed in the first datacenter):

set_id | set_comment | local_id | master_node  
--------+--------------------------+----------+-------------   
1 | All tables and Sequences | 1 | t
(1 row)

Start dtService on all the tenant appliances except for the tenant appliance that is in failed status.
```
 service dtService start
```

Restart the Primary Appliance

When you restart the primary appliance after failover, perform a slony reinitialization on both the EDB and AVDB.