You can filter all collected log events on the vRealize Log Insight Linux agent based on their field values to specify which log events to pick or drop. You can use the whitelist and blacklist collector options to define filters.

Tip: By default the vRealize Log Insight Linux agent collects hidden files created by programs or editors. The hidden file names start with a period. You can prevent the vRealize Log Insight Linux agent from collecting hidden files, by adding an exclude exclude=.* parameter.

For each log event, the collector evaluates the whitelist and blacklist filter expression. If the whitelist expression evaluates to true and the blacklist expression evaluates to false or cannot be evaluated, the event moves to the queue for further processing. In any other case, the collector drops the event. The default value of the whitelist expression is true and the default value of the blacklist expression is false.

Tip: The Filelog collector provides fewer fields for filtering. To obtain fields for filtering, you can parse the logs. For more information, see Parsing Logs.

A whitelist or blacklist filter is a set of variables, literals, and operators that evaluates to a single logical or integer value. You use the log event fields as variables and double quoted strings and numbers as literals. For information about the operators that you can use within a filter expression, see Event Fields and Operators.

Note:

  • If you compare a number with a string or if the comparison involves numerical strings, each string is converted to a number and the comparison is performed numerically. For example:

    • The expression whitelist = 123.0 == "000123" evaluates to true.
    • The expression whitelist = "00987" == "987.00" evaluates to true.
    • In the expression whitelist = response_size >= "12.12", if the response_size field has a numeric value, the expression is evaluated numerically. If the response size is greater than 12.12, the expression is true, else it is false.
    • In the expression whitelist = "09123" < "234", both the string literals are converted to numeric values and the expression evaluates to false.

  • If one of the string operands cannot be converted to numeric values, both the operands are converted to string. A simple case-sensitive lexicographical comparison is performed. For example:

    • The expression whitelist = "1234a" == "1234A" is a string comparison that evaluates to false.
    • The expression whitelist = 4 < "four" converts 4 to "4" and evaluates to true.
    • In the expression whitelist = response_size > "thousand", the value of the response_size field is converted to a string value, which evaluates the expression to false.

  • If a filter expression evaluates to an integer value, it is treated as false if it is 0 and true otherwise.

    For example, the expression whitelist = some_integer & 1 evaluates to true if the some_integer field has a least significant bit set and false otherwise.

For a complete list of log event fields and operators see Collect Log Events from a Log File.

In this example, you collect Apache access logs from the file /var/log/httpd/access. Some sample logs from the file are:
  • 127.0.0.1 - frank [10/Oct/2016:13:55:36 +0400] "GET /apache_pb.gif HTTP/1.0" 200 2326
  • 198.51.100.56 - john [10/Oct/2016:14:15:31 +0400] "GET /some.gif HTTP/1.0" 200 8270
  • 198.51.100.12 - smith [10/Oct/2016:14:15:31 +0400] "GET /another.gif HTTP/1.0" 303 348
  • 198.51.100.32 - test [10/Oct/2016:15:22:55 +0400] "GET /experimental_page.gif HTTP/1.0" 400 46374
  • 127.0.0.1 - test [10/Oct/2016:15:22:57 +0400] "GET /experimental_page2.gif HTTP/1.0" 301 100

Prerequisites

  • Log in as root or use sudo to run console commands.
  • Log in to the Linux machine on which you installed the vRealize Log Insight Linux agent, open a console and run pgrep liagent to verify that the vRealize Log Insight Linux agent is installed and running.

Procedure

  1. Define a parser for the logs, as shown in the following snippet: 
    [parser|apache-access]
base_parser=clf
format=%h %l %u %t \"%r\" %s %b
    The parser that you have defined extracts the remote_host, remote_log_name, remote_auth_user, timestamp, request, status_code, and response_size fields for every log event collected from the file /var/log/httpd/access. You can use these fields to filter events.
  2. Open the /var/lib/loginsight-agent/liagent.ini file in any text editor.
  3. Define a Filelog section in the file to collect and parse logs, as shown in the following snippet: 
    [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
  4. Filter log events according to your requirement.

    • To collect logs where the HTTP status is 200, you can define a whitelist in the Filelog section as shown in the following snippet:

      [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
whitelist = status_code == 200

      The whitelist expression evaluates to true only for the first and second log events from the sample logs and the collector picks these events.

      If the status_code field does not exist in the log event because it is not present in the log or is not parsed, the whitelist expression cannot be evaluated, which means it evaluates to false and collector drops the event.

    • To drop a log event that you are not interested in, you can use the blacklist option. For example, if you are not interested in local traffic, you can block the local IP as shown in the following snippet:

      [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
blacklist = remote_host == "127.0.0.1"

      The collector picks the second, third, and fourth log events from the sample logs.

    • To filter log events based on more than one predicate, you can use or and and operators. For example, you can drop events generated from a local IP or events generated by test users from any host that you do not require, as shown in the following snippet:

      [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
blacklist = remote_host == "127.0.0.1" or remote_auth_user == "test"

      Using the or operator evaluates the blacklist expression to true to skip an unwanted log event. The expression instructs the collector to drop the event if the remote_host field value is "127.0.0.1" or the remote_auth_user field value is "test".

      The collector picks the second and the third log events from the sample logs.

    • To drop log events generated from a local IP by test users, you can use and in the blacklist expression, as shown in the following snippet:

      [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
blacklist = remote_host == "127.0.0.1" and remote_auth_user == "test"

      The collector drops the fifth log event from the sample logs.

    • You can use whitelist and blacklist filters together. For example, if you require log events where the response size is greater than 1024 bytes but you do not require events that originated from a local host, you can use the following snippet: 

      [filelog|apache-access]
directory = /var/log/httpd/
include = access
parser = apache-access
whitelist = response_size > 1024
blacklist = remote_host == "127.0.0.1" or remote_host == "localhost"

      The collector picks the second log event from the sample logs.