Logstash pipeline responsible for ingesting logs from all services, ran into exceptions due to some out-of-context (older timestamp) log entries from some of the services.

Solution

  1. Delete the logstash indices, using command:
    kubectl exec -it elasticsearch-master-0 -n tcsa-system -c opensearch -- curl -XDELETE -ku admin:admin https://elasticsearch:9200/logstash-.*
  2. Restart logstash pods:
    kubectl delete pod -n tcsa-system logstash-0 logstash-1