The end-to-end life cycle of a log message or event includes multiple stages as the data flows in and out of vRealize Log Insight from agent read, parse, ingestion, indexing (buckets), alerting, query, archive (bucket seal and ship), and deletion.
An event transitions through the following stages.
It is generated on a device (outside of vRealize Log Insight).
It is picked up and sent to vRealize Log Insight (inside and/or outside vRealize Log Insight) in one of the following ways:
By a vRealize Log Insight agent using ingestion API or syslog
Through a third-party agent such as rsyslog, syslog-ng or log4j using syslog
By custom writing to ingestion API (such as log4j appender)
By custom writing to syslog (such as log4j appender)
The event is received by vRealize Log Insight.
If you are using the integrated load balancer (ILB), the event is directed to a single node that is responsible for processing the event.
If the event is declined, the client handles declines by means of UDP drops, TCP with protocol settings, or CFAPI with a disk-backed queue.
If the event is accepted, the client is notified.
The event is passed through the vRealize Log Insight ingestion pipeline, from which the following steps occur:
A keyword index is created or updated. The index is stored in proprietary format on local disk.
Machine learning is applied to cluster events.
The event is stored in compressed proprietary format on the local disk in a bucket .
The event is queried.
Keyword and glob queries are matched against the keyword index
Regex is matched against compressed events
The event is archived.
Bucket seal and marked as archived
The event is deleted.
Buckets are deleted in a FIFO model