ElasticSearch data and the Events pods are crashing in longevity setup

To check the probe readiness, use the command:

#kubectl describe pod/elasticsearch-data-x -n <pod-namespace>

Displays the latest event that Readiness probe failed, as below:

Root Cause

The master nodes in the ElasticSearch cluster are not in sync.

1. Log in to one of the ElasticsSearch master pod, by executing the command:

   #kubectl exec -it  elasticsearch-master- -n <pod-namespace>

2. Check the cluster master node status, by using the command:

   #curl http://elasticsearch:9200/_cat/nodes

   Output:

   

3. To logout, enter command Exit.
4. Compare the IP's address for:
   • elasticsearch-master-0
   • elasticsearch-master-1
   • elasticsearch-master-2
   • With, the IP's address of the elasticsearch-master pods, using command:

     #kubectl get pods -A -o wide | grep elasticsearch-master

     Output:

     If the IP's of any master node mismatch, there might be problem with the master cluster formation.

Solution

1. Shut down the master nodes, using command:

   #kubectl scale --replicas=0 statefulset.apps/elasticsearch-master -n tcsa-system

2. Ensure that, all the master nodes are down.
3. Restart the master nodes, using command:

   # kubectl scale --replicas=3 statefulset.apps/elasticsearch-master -n tcsa-system

4. Ensure that, all the master nodes and data nodes are up and running and ready.