You can configure the vRealize Log Insight Linux agent to collect log events from one or more log files.

By default the vRealize Log Insight Linux agent collects hidden files created by applications or editors. The hidden filenames start with a period. You can prevent the vRealize Log Insight Linux agent from collecting hidden files by adding an exclude parameter, exclude=.*.

Field names are restricted. The following names are reserved and cannot be used as field names.

  • event_type
  • hostname
  • source
  • text

You can specify up to three destinations for agent information and filter the information before it is sent. See Forwarding Logs from a vRealize Log Insight Agent

Note: Monitoring a large number of files, such as a thousand or more, leads to a higher resource utilization by vRealize Log Insight Agent and impacts the overall performance of the host machine. To prevent this, configure the agent to monitor only the necessary files using patterns and globs, or archive the old log files. If monitoring a large number of files is a requirement, consider increasing the host parameters such as CPU and RAM.

Prerequisites

  • Log in as root or use sudo to run console commands.
  • Verify that the vRealize Log Insight Linux agent is installed and running. Log in to the Linux machine on which you installed the vRealize Log Insight Linux agent, open a console, and run pgrep liagent.

Procedure

  1. Open the /var/lib/loginsight-agent/liagent.ini file in any text editor.
  2. Locate the [server|<dest_id>] section of the file. Add configuration parameters and set the values for your environment. 
    [filelog|section_name]
directory=path_to_log_directory
include=glob_pattern
...
    Parameter Description
    [filelog|section_name] A unique name for the configuration section.
    directory=full-path-to-log-file The full path to the log file directory. Glob patterns are supported. Example configurations:
    • To collect from all sub-directories of D:\Logs\new_test_logsdirectory, use directory=D:\Logs\new_test_logs\*
    • If your sub-directories have their own sub-directories, use the following configuration to monitor all sub-directories directory=D:\Logs\new_test_logs\*\*
    Note: To limit the number of files and directories and avoid high resource consumption, you cannot define a directory glob for either the first or second level directories such as: directory=c:/tmp/* or directory=c:\Logs\*. The directory path must be at least two levels.

    You can define a path to a non-existing directory, and the agent will collect the log files in that directory once the directory and files are created.

    You can define the same directory under one or more different configuration sections, to collect logs from the same file multiple times. This process makes it possible to apply different tags and filters to the same source of events.
    Note: If you use identical configurations for these sections, duplicated events are observed on the server side.
    include=file_name; ... (Optional) The name of a filename or a file mask (glob pattern) from which to collect data. You can provide values as a semicolon separated list. The default value is *, which means that all files are included. The parameter is case-sensitive.

    A file mask (glob pattern) can be used to group files that follow the same naming convention, as well as within a single filename. For example, filenames that include spaces, such as vRealize Ops Analytics.log and vRealize Ops Collector.log, can be specified with vRealize?Ops?Analytics*.log or vRealize*.log. By using file masks, you can specify filenames that are acceptable for agent configuration under Linux and Windows hosts.

    By default .zip and .gz files are excluded from collection.

    Important: If you are collecting a rotated log file, use the include and exclude parameters to specify a glob pattern that matches both the primary and the rotated file. If the glob pattern matches only the primary log file, the vRealize Log Insight agents might miss events during rotation. The vRealize Log Insight agents automatically determine the correct order of rotated files and sends events to the vRealize Log Insight server in the right order. For example, if your primary log file is named myapp.log and rotated logs are myapp.log.1 and myapp.log.2 and so on, you can use the following include pattern:

    include= myapp.log;myapp.log.*

    exclude=regular_expression (Optional) A filename or file mask (glob pattern) to exclude from collection. You can provide values as a semicolon separated list. The default value is empty, which means that no file is excluded.
    event_marker=regular_expression (Optional) A regular expression that denotes the start of an event in the log file. If omitted defaults to newline. The expressions you type must use the Perl regular expressions syntax.
    Note: Symbols, for example quotation marks ( " "), are not treated as wrappers for regular expressions. They are treated as part of the pattern.

    Since the vRealize Log Insight agent is optimized for real-time collection, partial log messages written with an internal delay may be split into multiple events. If log file appending stops for more than 200 ms without a new observed event_marker, the partial event is treated as complete, parsed, and delivered. This timing logic is non-configurable and has priority over the event_marker setting. Log file appenders should flush full events.

    enabled=yes|no (Optional) A parameter to enable or disable the configuration section. The possible values are yes or no. The default value is yes.
    charset=char-encoding-type (Optional) The character encoding of the log files that the agent monitors. Possible values are:
    • UTF-8
    • UTF-16LE
    • UTF-16BE
    The default value is UTF-8.
    tags={"tag-name" : "tag-value", ...}

    (Optional) A parameter to add custom tags to the fields of collected events. Define tags using JSON notation. Tag names can contain letters, numbers, and underscores. A tag name can only begin with a letter or an underscore and cannot exceed 64 characters. Tag names are not case-sensitive. For example, if you use tags={"tag_name1" : "tag value 1", "Tag_Name1" : "tag value 2" }, Tag_Name1 is ignored as a duplicate. You cannot use event_type and timestamp as tag names. Any duplicates within the same declaration are ignored.

    If the destination is a syslog server, tags can override the APP-NAME field. For example, tags={"appname":"VROPS"}.

    exclude_fields (Optional) A parameter to exclude individual fields from collection. You can provide multiple values as a semicolon- or comma-separated list. For example,
    • exclude_fields=hostname; filepath
    • exclude_fields=type; size
    • exclude_fields=type, size
    raw_syslog=Yes|No For agents that use the syslog protocol, this option allows the agent to collect and send raw syslog events. The default is No, which means collected events are transformed with user-specified syslog attributes. Enable this option to collect events without any syslog transformations.
  3. Save and close the liagent.ini file.

Example: Configurations

[filelog|messages]
directory=/var/log
include=messages;messages.?

[filelog|syslog]
directory=/var/log
include=syslog;syslog.?

[filelog|Apache]
directory=/var/log/apache2
include=*