You can configure the Key/Value Pair (KVP) parser for both FileLog and WinLog collectors.

Key/Value Pair (KVP) Parser

The kvp parser finds and extracts all key=value matches from an arbitrary log message text. The following example shows the kvp parser format. 
[parser|kvp_parser]
base_parser=kvp
fields=*

For example, the key-value log can be in the following format: scope=local; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0;

With the kvp parser, you must specify the fields from which the values are to be extracted. For example, if the definition fields=name,lastname,country exists in the configuration, only the values with the specified keys are parsed and sent to the server.

Both the key and the value can be optionally surrounded by double quotes “ “ to define white space or other special characters.

When double quotes are used for the key or value, the backslash character “ \ “ can be used as the escape character. Any character following the backslash character is defined literally, including a double quote character or a backslash character. For example: “ \\ ”

Note the following considerations.

  • If the key in a key/value pair is not followed by an equals sign and a VALUE is not provided, the option is skipped, as with free text.
  • The key cannot be empty, the value can be empty.
  • An equals sign that is not followed by a value is treated as free text and is skipped.
  • A value can be a string of characters that are surrounded by double quote characters, or it can be empty. Use a backslash for escaping special characters that are part of the value.

KVP Parser Options

Note the following information about the structure of the kvp parser.

Option Description
fields

The information that you want to extract described as units of data. For example, fields=name,lastname,country.

If specific field names are defined by the fields option, then each invalid char in a field name extracted from a log is replaced with an underscore. For example, if the fields option looks for fields "x-A" and "a*(X+Y)", then the parser extracts these fields from logs and renames them to "x_a" and "a__x_y" fields respectively. This makes it possible to extract fields with any chars in the name.

If the fields option is specified as "*", which means the kvp parser recognizes field/value pairs automatically, then the parser looks for fields that have only "alphanumeric+underscore" chars (supported by LI server). All other invalid chars are dropped instead of being converted to underscores. This prevents the parser from extracting unnecessary information into static fields.

delimiter

Optional.

Default delimiters are the space character, tab, newline characters, comma, and semicolon characters.

If delimiters are not specified in the configuration, the kvp parser uses default delimiters for parsing.

To change the default delimiters to specific delimiters, you must define them between double quotes. For example: delimiter = "#^|". This definition means that each of the characters that are enclosed in the double quotes is used as a delimiter. For the kvp parser, any character can be considered as delimiter. You can include the default delimiters with other delimiters in the definition.

For example, the delimiter = "#^|\t\r\n\s" statement includes the tab, newline characters, and the space as delimiters. If these characters are used, they must be preceded by the escape character. For example, to define the space character as a delimiter, enter the escape character " \" before the space character when defining it as a delimiter, for example, delimiter="\s".

field_decoder

Nested parsers are specified as a JSON string in which the keys are the names of the field to apply to the nested parser, and the value is the name of the parser to use for that field.

Each nested parser is applied to the appropriate field, as decoded by the base parser.

Field decoders are useful when the value of a key-value pair is a complex value such as a timestamp or a comma-separated list.

debug =

Optional. The debug = value can be yes or no. The default value for debug is debug=no for parsers.

When the option is set to yes, you can view detailed logs of the parser ingestion in liagent_<date>.log.

Additional Keys Value Options

Key Definition
KVP_MESSAGE = *(MESSAGE_ENTRY [WSPR])

A list of message entries separated by optional white space

MESSAGE_ENTRY = KVP / FREE_TEXT

An entry is a key/value pair or just a free text

KVP = KEY ["=" VALUE]

Key/value pair. If KEY is not followed by an equal sign and VALUE, it is skipped like free text.

KEY = BARE_KEY / QUOTED_KEY
FREE_TEXT = "="

A free standing equal sign is considered a free text and is skipped.

BARE_KEY = *1BARE_KEY_CHAR

At least one character

BARE_KEY_CHAR = %0x00-08 / %0x10-19 / %0x21-3C / %3E-%FF

Any character excluding equal sign, space, or TAB

QUOTED_KEY = 0x22 *1(QUOTED_STRING_CHAR / "\" CHAR) 0x22

At least one character surrounded by double quote characters. The backslash is used as an escape character.

QUOTED_STRING_CHAR = %0x00-21 / %0x23-FF

Any character excluding double quote

VALUE = BARE_VALUE / QUOTED_VALUE
BARE_VALUE = *BARE_VALUE_CHAR

Zero or more characters

BARE_VALUE_CHAR = %0x00-08 / %0x10-19 / %0x21-FF

Any character excluding space or TAB

QUOTED_VALUE = 0x22 *(QUOTED_STRING_CHAR / "\" CHAR) 0x22

A string of characters surrounded by double quote characters. This can be empty. The backslash is used as an escape character.

KVP Parser Configuration Examples

You can use fields=* to parse all fields, if necessary. 

[parser|simple_kvp]
base_parser =kvp
fields=*

This example shows how to specify the field decoder. 

[parser|mykvp]
debug=no
base_parser=kvp
delimiter="#^|"
fields=*
;OR fields=scope,abstract,lazyInit,autowireMode,dependencyCheck
field_decoder={"field1":"field1_parser1"}

[parser|field1_parser1]
base_parser=clf
format=[%{value1}i]]
field_decoder={"value1":"field_parser2"}

To parse the following KVP log: 

Configuring transport... proto = cfapi server_hostname = LOCALHOST ssl = no port = 9000 reconnect = 30
Define the KVP parser configuration: 
[parser|kvp_log_parser]
base_parser=kvp
fields=*
The KVP parser returns the following fields: 
proto=cfapi
server_hostname=LOCALHOST
ssl=no
port=9000
reconnect=30
Note: In input logs, when an assignment operator ( =) is preceded by a randomly generated key, it is a best practice not to use the KVP parser with the * option, because it generates many random fields, which might lead to UI and query performance issues. Instead, you can use the parser by specifying the fields that need to be parsed.

Simple and Complex KVP Parser Examples

Simple KVP Parser Example 

[filelog|MyLog]
directory=C:\<folder_name>\Parser_logs
include=*.log
parser=my_KVP_parser

[parser|my_KVP_parser]
base_parser=kvp
fields=*

Complex KVP Parser Example 

[filelog|MyLog]
directory=C:\<folder_name>\Parser_logs
include=*.log
parser=my_KVP_parser

[parser|my_KVP_parser]
base_parser=kvp
fields=*
field_decoder={"field1":" field1_parser1"}

[parser| field1_parser1]
base_parser=clf
format=[%{value1}i]]
field_decoder={"value1":" field1_parser2"}