Before you set up the secondary data center, configure the primary data center for Elasticsearch, RabbitMQ, and Ehcache replication across clusters.

Before you begin

You have set up a VMware Identity Manager cluster in the primary data center.

About this task

Elasticsearch, RabbitMQ, and Ehcache are embedded in the VMware Identity Manager virtual appliance. Elasticsearch is a search and analytics engine used for auditing, reports, and directory sync logs. RabbitMQ is a messaging broker. Ehcache provides caching capabilities.

Configure these changes in all the nodes in the primary data center cluster.

Procedure

  1. Configure Elasticsearch for replication.

    Make these changes in each node of the primary data center cluster.

    1. Disable the cron job for Elasticsearch.
      1. Edit the /etc/cron.d/hznelasticsearchsync file:

        vi /etc/cron.d/hznelasticsearchsync

      2. Comment out this line:

        #*/1 * * * * root /usr/local/horizon/scripts/elasticsearchnodes.hzn

    2. Add the IP addresses of all the nodes in the primary data center cluster.
      1. Edit the /etc/sysconfig/elasticsearch file.

        vi /etc/sysconfig/elasticsearch

      2. Add the IP addresses of all the nodes in the cluster:

        ES_UNICAST_HOSTS=IPaddress1,IPaddress2,IPaddress3

    3. Add the load balancer FQDN of the secondary data center cluster to the /usr/local/horizon/conf/runtime-config.properties file.
      1. Edit the /usr/local/horizon/conf/runtime-config.properties file.

        vi /usr/local/horizon/conf/runtime-config.properties

      2. Add this line to the file:

        analytics.replication.peers=LB_FQDN_of_second_cluster

  2. Configure RabbitMQ for replication.

    Make these changes in each node of the primary data center cluster.

    1. Disable the cron job for RabbitMQ.
      1. vi /etc/cron.d/hznrabbitmqsync

      2. Comment out this line:

        #*/1 * * * * root /usr/local/horizon/scripts/rabbitmqnodes.hzn

    2. Make the following changes in the /usr/local/horizon/scripts/rabbitmqnodes.hzn file.
      1. vi /usr/local/horizon/scripts/rabbitmqnodes.hzn

      2. Comment out these lines.

        #make sure SAAS is up, otherwise we won't have an accurate node list
        #if test $(curl -X GET -k https://localhost/SAAS/API/1.0/REST/system/health/allOk -sL -w "% {http_code}\\n" -o /dev/null) -ne 200 ; then
        #    echo SAAS not running, aborting
        #    exit 0
        #fi

        Also comment out the following line.

        #nodes=$(uniqList true $(enumeratenodenames))

      3. Add the host names of all the nodes in the primary data center cluster. Use the host names only, not the fully qualified domain names. Separate the names with a space.

        nodes="node1 node2 node3"

    3. Add the IP address and host name mapping of the other nodes in the cluster to the /etc/hosts file. Do not add an entry for the node you are editing. This step is only required if there is no DNS entry that can resolve the fully-qualified domain name or partially-qualified domain names.

      IPaddress node2FQDN node2

      IPaddress node3FQDN node3

    4. Run the script to build the RabbitMQ cluster.

      /usr/local/horizon/scripts/rabbitmqnodes.hzn

  3. Configure Ehcache for replication.

    Make these changes in each node of the primary data center cluster.

    1. vi /usr/local/horizon/conf/runtime-config.properties

    2. Add the FQDN of the other nodes in the cluster. Do not add the FQDN of the node you are editing. Separate FQDNs by a colon.

      ehcache.replication.rmi.servers=node2FQDN:node3FQDN

      For example:

      ehcache.replication.rmi.servers=server2.example.com:server3.example.com

  4. Restart the VMware Identity Manager service on all nodes.

    service horizon-workspace restart

  5. Verify that the cluster is set up correctly.

    Run these commands on all the nodes in the first cluster.

    1. Verify the health of Elasticsearch.

      curl 'http://localhost:9200/_cluster/health?pretty'

      The command should return a result similar to the following.

               {
        "cluster_name" : "horizon",
        "status" : "green",
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "active_primary_shards" : 20,
        "active_shards" : 40,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 0,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0
      }

      If there are problems, see Troubleshooting Elasticsearch and RabbitMQ.

    2. Verify the health of RabbitMQ.

      rabbitmqctl cluster_status

      The command should return a result similar to the following.

      Cluster status of node 'rabbitmq@node3' ...
      [{nodes,[{disc,['rabbitmq@node2','rabbitmq@node3']}]},
       {running_nodes,['rabbitmq@node3']},
       {cluster_name,<<"rabbitmq@node2.example.com">>},
       {partitions,[]},
       {alarms,[{'rabbitmq@node3',[]}]}]

      If there are problems, see Troubleshooting Elasticsearch and RabbitMQ.

    3. Verify that the /opt/vmware/horizon/workspace/logs/ horizon.log file contains this line.

      Added ehcache replication peer: //node3.example.com:40002

      The host names should be those of the other nodes in the cluster.

What to do next

Create a cluster in the secondary data center. Create the nodes by exporting the OVA file of the first VMware Identity Manager virtual appliance from the primary data center cluster and using it to deploy the new virtual appliances in the secondary data center..