Understanding how vRealize Log Insight works with messages and events is key to product usage.
The end-to-end life cycle of a log message or event includes multiple stages in vRealize Log Insight from agent read, parse, ingestion, indexing (buckets), alerting, query, archive (bucket seal and ship), and deletion.
An event transitions through the following stages.
It is generated on a device (outside of vRealize Log Insight).
It is picked up and sent to vRealize Log Insight (inside and/or outside vRealize Log Insight) in one of the following ways:
By a vRealize Log Insight agent using ingestion API or syslog
Through a third-party agent such as rsyslog, syslog-ng or log4j using syslog
By custom writing to ingestion API (such as log4j appender)
By custom writing to syslog (such as log4j appender)
The event is received by vRealize Log Insight.
If you are using the integrated load balancer (ILB), the event is directed to a single node that is responsible for processing the event.
If the event is declined, the client handles declines by means of UDP drops, TCP with protocol settings, or CFAPI with a disk-backed queue.
If the event is accepted, the client is notified.
The event is passed through the vRealize Log Insight ingestion pipeline, from which the following steps occur:
A keyword index is created or updated. The index is stored in proprietary format on local disk.
Machine learning is applied to cluster events.
The event is stored in compressed proprietary format on the local disk in a bucket.
The event is queried.
Keyword and glob queries are matched against the keyword index
Regex is matched against compressed events
The event is archived.
Bucket is seal and marked as archived when is reaches .5GB
The event is deleted.
Buckets are deleted in FIFO order
For More Information
For more information, see the VMware Technical Publications video,