In the consolidated SDDC, vRealize Log Insight provides real-time log management and log analysis with machine learning-based intelligent grouping, high-performance searching, and troubleshooting across physical, virtual, and cloud environments.

Overview

vRealize Log Insight collects data from ESXi hosts using the syslog protocol. It connects to other VMware products, like vCenter Server, to collect events, tasks, and alarm data. vRealize Log Insight also integrates with vRealize Operations Manager to send notification events and enable launch in context. vRealize Log Insight also functions as a collection and analysis point for any system that is capable of sending syslog data.

In addition to syslog data, to collect logs, you can install an ingestion agent on Linux or Windows servers or you can use the pre-installed agent on certain VMware products. This agent approach is useful for custom application logs and operating systems that do not natively support the syslog protocol, such as Windows.

Deployment Models

You can deploy vRealize Log Insight as a virtual appliance in one of the following configurations:

  • Standalone master node.

  • Cluster of one master and at least two worker nodes. You can establish high availability by using the integrated load balancer (ILB).

The compute and storage resources of the vRealize Log Insight instances can scale up as growth demands.

Architecture

The architecture of vRealize Log Insight in the SDDC enables several channels for the collection of log messages.

Figure 1. Architecture of vRealize Log Insight


Each node in the vRealize Log Insight cluster contains repository, database, load balancing electing, syslog ingestion and UI modules.

vRealize Log Insight clients connect to the ILB Virtual IP (VIP) address, and use the syslog or the Ingestion API via the vRealize Log Insight Agent to send logs to vRealize Log Insight. Users and administrators interact with the ingested logs using the user interface or the API.

By default, vRealize Log Insight collects data from vCenter Server systems and ESXi hosts. For forwarding logs from NSX for vSphere and vRealize Automation, use content packs which contain extensions or provide integration with other systems in the SDDC.

Types of Nodes

For functionality, high availability and scalability, vRealize Log Insight supports the following types of nodes which have inherent roles:

Master Node

Required initial node in the cluster. In standalone mode, the master node is responsible for all activities, including queries and log ingestion as well as operations that are related to the lifecycle of a cluster, for example, performing upgrades, and adding and removing of worker nodes. In a scaled-out and highly available environment, the master node still retains the role of performing operations that are related to the lifecycle of a cluster, for example, performing upgrades, and adding and removing worker nodes. However, it functions as a generic worker about queries and log ingestion activities.

The master node stores logs locally. If the master node is down, the logs on it become unavailable.

Worker Node

Optional. This component enables scale out in larger environments. As you add and configure more worker nodes in a vRealize Log Insight cluster for high availability (HA), queries and log ingestion activities are delegated to all available nodes. You must have at least two worker nodes to form a cluster with the master node.

The worker node stores logs locally. If any of the worker nodes is down, the logs on the worker become unavailable.

The VMware Validated Design for Workload and Management Consolidation does not use worker nodes. For high availability and a scaled-out vRealize Log Insight cluster, refer to the VMware Validated Design for Software-Defined Data Center.

Integrated Load Balancer (ILB)

In cluster mode, the ILB provides a centralized entry point which ensures that vRealize Log Insight accepts incoming ingestion traffic. As additional nodes are added to the vRealize Log Insight instance to form a cluster, the ILB feature simplifies the configuration for high availability. It balances the incoming traffic fairly among the available vRealize Log Insight nodes.

The ILB runs on one of the cluster nodes at all times. In the VMware Validated Design for Workload and Management Consolidation, the ILB runs on the single master node. In environments that contain several nodes in a cluster, an election process is held that will determine the leader of the cluster. Periodically, a health check is performed to determine whether a re-election process needs to be performed. If the node that hosts the ILB Virtual IP (VIP) address stops responding, the VIP address is failed over to another node in the cluster via an election process.

All queries against data are directed to the ILB. The ILB delegated the query request to a query master for the duration of the query. The query master queries all nodes, both master and worker nodes for data, and then sends the aggregated data back to the client.

The Web user interface of the ILB serves is a single pane of glass, presenting data from the master, and from the worker nodes in a scaled-out cluster, in a unified display. Although you can access individual nodes by using their Web user interfaces, unless you are performing administrative activities on these nodes, use the ILB interface.

Application Functional Components

The functional components of a vRealize Log Insight instance interact with each other to perform the following operations:

  • Analyze logging data that is ingested from the components of a data center

  • Visualize the results in a Web browser, or support results query using API calls.

Figure 2. vRealize Log Insight Logical Node Architecture


Each Log Insight node support an API or UI access point, performs log collection, and participates in load balancing and load balancing leader election.

The components of vRealize Log Insight perform these tasks:

Product/Admin UI and API

The UI server is a Web application that serves as both user and administration interface, and hosts the API for accessing collected statistics.

Syslog Ingestion

Responsible for ingesting syslog logging data.

Log Insight Native Ingestion API (CFAPI) Ingestion

Responsible for ingesting logging data over the Ingestion API by using one of the following methods:

  • vRealize Log Insight Agent that has been deployed or pre-configured on SDDC components .

  • Log Insight Importer that is used for ingestion of non-real time data.

Integration Load Balancing and Election

Responsible for balancing incoming UI and API traffic, and incoming data ingestion traffic.

The Integrated Load Balancer is a Linux Virtual Server (LVS) that is built in the Linux Kernel for Layer 4 load balancing . Each node in vRealize Log Insight contains a service running the Integrated Load Balancer, but only a single node functions as the leader at all times. In a single-node vRealize Log Insight instance, this is always the master node. In a scaled-out vRealize Log Insight cluster, this role can be inherited by any of the available nodes during the election process. The leader periodically performs health checks to determine whether a re-election process is required for the cluster.

Configuration Database

Stores configuration information about the vRealize Log Insight nodes and cluster. The information that is stored in the database is periodically replicated to all available vRealize Log Insight nodes.

Log Repository

Stores logging data that is ingested in vRealize Log Insight. The logging repository is local to each node and not replicated. If a node is offline or removed, the logging data which is stored on that node becomes inaccessible. In environments where an ILB is configured, incoming logging data is evenly distributed across all available nodes.

When a query arrives from the ILB, the vRealize Log Insight node holding the ILB leader role delegates the query to any of the available nodes in the cluster.

Authentication Sources

You can configure one or more of the following authentication models on vRealize Log Insight:

  • Microsoft Active Directory

  • Local Accounts

  • VMware Identity Manager

Content Packs

Content packs help extend Log Insight with valuable troubleshooting information by providing structure and meaning to raw log data that is collected from either a vRealize Log Insight agent, vRealize Log Insight Importer or a Syslog Stream. Content packs can contain vRealize Log Insight agent configurations, providing out-of-the-box parsing capabilities for standard logging directories and logging formats, along with dashboards, extracted fields, alert definitions, query lists, and saved queries from the log data related to a specific product in vRealize Log Insight. For details about and to download content packs, see Log Insight Content Pack Marketplace or the VMware Solutions Exchange.

Integration with vRealize Operations Manager

The integration of vRealize Log Insight with vRealize Operations Manager provides data from multiple sources to a central place for monitoring the SDDC. The integration has the following advantages:

  • vRealize Log Insight sends notification events to vRealize Operations Manager.

  • vRealize Operations Manager can provide the inventory map of any vSphere object to vRealize Log Insight. In this way, you can view log messages from vRealize Log Insight in the vRealize Operations Manager Web user interface, taking you either directly to the object itself or to the location of the object within the environment.

  • Access to the vRealize Log Insight user interface is embedded in the vRealize Operations Manager user interface .

Archiving

vRealize Log Insight supports data archiving on an NFS shared storage that the vRealize Log Insight nodes can access. However, vRealize Log Insight does not manage the NFS mount used for archiving purposes nor does it perform cleanup of the archival files. If the NFS mount for archiving does not have enough free space or is unavailable for a period of time greater than the retention period of the virtual appliance, vRealize Log Insight stops ingesting new data until the NFS mount has enough free space, becomes available, or archiving is disabled. System notifications from vRealize Log Insight sends you an email when the NFS mount is about to run out of space or is unavailable if enabled.

Backup

You back up vRealize Log Insight using traditional virtual machine backup solutions that are compatible with vSphere Storage APIs for Data Protection (VADP) such as vSphere Data Protection.

Consolidated vRealize Log Insight Deployment

Because of its scope, the VMware Validated Design for Workload and Management Consolidation implements a small-scale vRealize Log Insight deployment. This implementation is designed to maintain the ability to scale up to the larger VMware Validated Design for Software-Defined Data Center. The validated design uses an integrated load balancer on top of the single master node so that you can scale out the cluster as required while minimizing downtime.