Notification cache is empty

Follow these steps when notifications are not appearing in the Notification cache.

Prerequisites

Review this before you start the troubleshooting steps.

• If alerts show up in the Alerts EDAA but not in M&R, the issue is in M&R.
• If you see a backup of messages in RabbitMQ, the SAM publisher is working but the ingestion service or elasticsearch service may be failing.
• If no messages are passing through into RabbitMQ, the problem is on the SAM publisher side.

Procedure

1. Verify all services are running, as described in Status checks on services.
2. Check that SAM has started to publish to RabbitMQ and started with --bootstrap=bootstrap-amqp.conf.
3. Verify SAM is exposed via the EDAA and started with --edaa=sam-presentation/2.0.
4. Under the Connections tab in the RabbitMQ web-based management interface, do the following:
   1. Verify that there are two connections: one from RabbitMQ / Java and the other from rabbitmq-c.
   2. If RabbitMQ / Java is not present, check if Tomcat is running and view the <SAM_base_directory>/SAM/smarts/local/logs/tomcat/alert-ingestion.
   3. If rabbitmq-c is not present and if SAM is running, check the presentation SAM log to see if it is publishing to RabbitMQ.
5. From the host running the presentation SAM server, check if you can access the elasticsearch database and verify the count from a browser or using a tool like curl on Linux.
   Browser example:
   http://localhost:9200/smarts/alert/_count
   Linux example:
   curl http://localhost:9200/smarts/alert/_count
   A result similar to this appears:
   {"count":50002,"_shards":{"total":5,"successful":5,"failed":0}}
6. If the count is zero, check if there are any notifications in SAM.
7. From a browser, check if you can access Tomcat in the SAM server host.
   For example:
   http://hostname:8080
8. Verify elasticsearch is running by checking <SAM_base_directory>/SAM/smarts/local/logs/tomcat/alert-ingestion.log
   If the service was running and is now shut down or unable to process notifications, a message similar to this may appear in the alerts-ingestion.log file:

   {date} {time},{pid} INFO  'ALERTS-IN' EventConsumerService EventConsumerPool-1 watchdog attemps to reconnect   
    {date} {time},{pid} INFO  'ALERTS-IN' plugins EventConsumerPool-1 [La Lunatica] loaded [], sites [] {date} {time}
    {date} {time},{pid} ERROR 'ALERTS-IN' ElasticsearchNodeConfigurationService EventConsumerPool-1  
    ElasticSearchNodeConfigurationService::verifyIndexing: ElasticSearch :   
    SERVICE_UNAVAILABLE :org.elasticsearch.client.transport.NoNodeAvailableException: No node available
    {date} {time},{pid} INFO  'ALERTS-IN' transport elasticsearch[Ent][generic][T#8] [Ent] failed to get node info for [#transport#-1][inet  
    [localhost/127.0.0.1:9300]], disconnecting...  
    org.elasticsearch.transport.NodeDisconnectedException: [][inet[localhost/127.0.0.1:9300]][cluster/nodes/info] disconnected