Logstash pipeline responsible for ingesting logs from all services, ran into exceptions due to some out-of-context (older timestamp) log entries from some of the services.
Solution
- Delete the logstash indices, using command:
kubectl exec -it elasticsearch-master-0 -n tcsa-system -c opensearch -- curl -XDELETE -ku admin:admin https://elasticsearch:9200/logstash-.*
- Restart logstash pods:
kubectl delete pod -n tcsa-system logstash-0 logstash-1