Understanding how vRealize Log Insight processes messages and events is key to using vRealize Log Insight effectively.

The life cycle of a log message or event has multiple stages including reading, parsing, ingestion, indexing, alerting, query application, archiving, and deletion.

Events and messages transition through the following stages.

  1. It is generated on a device (outside of vRealize Log Insight).
  2. It is picked up and sent to vRealize Log Insight in one of the following ways:
    • By a vRealize Log Insight agent using ingestion API or syslog
    • Through a third-party agent such as rsyslog, syslog-ng, or log4j using syslog
    • By custom writing to ingestion API (such as log4j appender)
    • By custom writing to syslog (such as log4j appender)
  3. vRealize Log Insight receives the event.
    • If you are using the integrated load balancer (ILB), the event is directed to a single node that is responsible for processing the event.
    • If the event is declined, the client handles declines with UDP drops, TCP with protocol settings, or CFAPI with a disk-backed queue.
    • If the event is accepted, the client is notified.
  4. The event is passed through the vRealize Log Insight ingestion pipeline, from which the following steps occur:
    • A keyword index is created or updated. The index is stored in a proprietary format on a local disk.
    • Machine learning is applied to cluster events.
    • The event is stored in a compressed proprietary format on the local disk in a bucket.
  5. The event is queried.
    • Keyword and glob queries are matched against the keyword index.
    • Regex is matched against compressed events.
  6. The event is moved to a bucket and archived.
    • A bucket is sealed and archived when it reaches 0.5 GB.
  7. The event is deleted.
    • Buckets are deleted in FIFO order.

For More Information

For more information, see the VMware Technical Publications video,