Creating Replica for Each Primary Node for Failover

It is important to create atleast 1 replica for each Redis primary node for failover. Execute all steps on all 3 cluster nodes to create one replica for each primary node.

Procedure

1. Copy redis.conf file under /etc directory as redis-6380.conf, by invoking below command:
   cp /etc/redis.conf /etc/redis-6380.conf
2. Change owner of /etc/redis-6380.conf to redis by using command:
   chown redis:root /etc/redis-6380.conf
3. Edit /etc/redis/redis-6380.conf file with below configurations:
   1. Change "bind 127.0.0.1" to "bind <IPAddress>"
   2. Change "port 6379" to "port 6380"
   3. Change 'logfile /var/log/redis/redis.log' to 'logfile /var/log/redis/redis-6380.log'
   4. Change "dbfilename dump.rdb" to "dbfilename dump-6380.rdb"
   5. Change "pidfile /var/run/redis_6379.pid" to "pidfile /var/run/redis_6380.pid". Change only in RHEL 7.5 or 7.6.
   6. Change '# cluster-enabled yes' to 'cluster-enabled yes'
   7. Change '# cluster-config-file nodes-6379.conf' to 'cluster-config-file redis-6380.conf'
   8. Change '# cluster-node-timeout 15000' to 'cluster-node-timeout 5000'
   9. Change 'requirepass' to 'requirepass <password>'
   10. Change '# masterauth <master-password>' to 'masterauth <password>'
4. For RHEL 7.5 or 7.6, create /usr/lib/systemd/system/redis-6380.service file:
   1. Copy /usr/lib/systemd/system/redis.service to /usr/lib/systemd/system/redis-6380.service and, edit /usr/lib/systemd/system/redis-6380.service file.
   2. Replace 'ExecStart=/usr/bin/redis-server /etc/redis.conf --supervised systemd' with 'ExecStart=/usr/bin/redis-server /etc/redis-6380.conf --supervised systemd'
   3. Replace 'ExecStop=/usr/libexec/redis-shutdown' with 'ExecStop=/usr/libexec/redis-shutdown redis-6380'
   For RHEL 6.9 or 6.10, create /etc/init.d/redis-6380 service file:

   1. Copy /etc/init.d/redis to /etc/init.d/redis-6380 and, edit /etc/init.d/redis-6380 file
   2. Replace 'shut="/usr/libexec/redis-shutdown"' with 'shut="/usr/libexec/redis-shutdown redis-6380"'
   3. Replace 'pidfile="/var/run/redis/redis.pid"' with 'pidfile="/var/run/redis/redis-6380.pid"'
   4. Replace 'REDIS_CONFIG="/etc/redis.conf"' with 'REDIS_CONFIG="/etc/redis-6380.conf"'
   5. Replace '[ -e /etc/sysconfig/redis ] && . /etc/sysconfig/redis' with '[ -e /etc/sysconfig/redis-6380 ] && . /etc/sysconfig/redis-6380'
   6. Replace 'lockfile=/var/lock/subsys/redis' with 'lockfile=/var/lock/subsys/redis-6380'
   7. Replace '[ -x $shut ] && $shut' with '$shut'
5. Enable the service:
   • For RHEL 7.5 or 7.6 command is: systemctl enable redis-6379
   • For RHEL 6.9 or 6.10 command is :

     chkconfig --add redis-6379 
     chkconfig redis-6379 on

6. Start the service:
   • For RHEL 7.5 or 7.6 command is : systemctl start redis-6379
   • For RHEL 6.9 or 6.10 command is : service redis-6379 start
7. Verify service status:
   • For RHEL 7.5 or 7.6 command is : systemctl status redis-6379 (service should be running)
   • For RHEL 6.9 or 6.10 command is : service redis-6379 status (service should be running)
8. Check the redis logs /var/log/redis/redis-6379.log. The log must contains "Ready to accept connections".
9. Execute step 1 to 8 on all 3 cluster nodes.
10. Add each redis process running on 6380 port as replicas to each primary running on 6379 port, by using commands:

    redis-cli -a <password> --cluster add-node <IP2>:6380 <IP1>:6379 --cluster-slave
    redis-cli -a <password> --cluster add-node <IP3>:6380 <IP2>:6379 --cluster-slave
    redis-cli -a <password> --cluster add-node <IP1>:6380 <IP3>:6379 --cluster-slave

    Note: Before executing above command, ensure that steps 1-9 are executed on each node in the cluster. Aove commands can be executed from any one cluster node.
    Expected output after executing each command:

    [OK] New node added correctly.

11. Validate redis primary and replication nodes by running below command:
    redis-cli -a <password> -h <IP> -p 6379 cluster nodes
    Output: Must show both the primary and secondary nodes along with primary to secondary mapping information.
    Other useful commands:

    1. Command to list all keys in redis cluster: redis-cli -a <password> -h <IP1/IP2/IP3> -p <6379/6380> keys "*"
    2. Command to see content/value of any particular keys: redis-cli -a <password> -h <IP1/IP2/IP3> -p <6379/6380> HGETALL "key"
    3. Command to reset redis cluster:

       redis-cli -a <password> -h <IP1/IP2/IP3> -p <6379/6380>
       IP:Port> flushall
       IP:Port> cluster reset
       IP:Port> exit