Use this information to troubleshoot problems with Elasticsearch and RabbitMQ in a cluster environment. Elasticsearch, a search and analytics engine used for auditing, reports, and directory sync logs, and RabbitMQ, a messaging broker, are embedded in the VMware Identity Manager virtual appliance.

Troubleshooting Elasticsearch

You can verify the health of Elasticsearch by using the following command in the VMware Identity Manager appliance.

curl 'http://localhost:9200/_cluster/health?pretty'

The command should return a result similar to the following.

         {
  "cluster_name" : "horizon",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 20,
  "active_shards" : 40,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

If Elasticsearch does not start correctly or its status is red, follow these steps to troubleshoot.

  1. Ensure port 9300 is open.

    1. Update node details by adding the IP addresses of all nodes in the cluster to the /usr/local/horizon/scripts/updateiptables.hzn file:

      ALL_IPS="node1IPadd node2IPadd node3IPadd"

    2. Run the following script on all nodes in the cluster.

      /usr/local/horizon/scripts/updateiptables.hzn

  2. Restart Elasticsearch on all nodes in the cluster.

    service elasticsearch restart

  3. Check logs for more details.

    cd /opt/vmware/elasticsearch/logs

    tail -f horizon.log

Troubleshooting RabbitMQ

You can verify the health of RabbitMQ by using the following command in the VMware Identity Manager appliance.

rabbitmqctl cluster_status

The command should return a result similar to the following.

Cluster status of node 'rabbitmq@node3' ...
[{nodes,[{disc,['rabbitmq@node2','rabbitmq@node3']}]},
 {running_nodes,['rabbitmq@node3']},
 {cluster_name,<<"rabbitmq@node2.example.com">>},
 {partitions,[]},
 {alarms,[{'rabbitmq@node3',[]}]}]

If RabbitMQ does not start or the health URL https://hostname/SAAS/API/1.0/REST/system/health/ shows "MessagingConnectionOk":"false", follow these steps to troubleshoot.

  1. Ensure ports 4369, 5700, 25672 are open. To open ports:

    1. Create the file by using this command:

      touch /usr/local/horizon/conf/flags/enable.rabbitmq

    2. Run the following script:

      /usr/local/horizon/scripts/updateiptables.hzn

  2. Restart RabbitMQ.

    1. Kill any existing rabbitmq processes.

    2. rabbitmqctl stop

    3. rabbitmq-server -detached

  3. You may need to restart the VMware Identity Manager service if RabbitMQ does not start gracefully.

    service horizon-workspace restart