You can configure the Key/Value Pair (KVP) parser for both FileLog and WinLog collectors.

Key/Value Pair (KVP) Parser

The kvp parser finds and extracts all key=value matches from an arbitrary log message text. The following example shows the kvp parser format.

[parser|kvp_parser]
base_parser=kvp
fields=*

For example, the key-value log can be in the format: scope=local; abstract=false; lazyInit=false; autowireMode=0; dependencyCheck=0;

With the kvp parser, you must specify the fields from which the values are to be extracted. For example, if the definition fields=name,lastname,country exists in the configuration, only the values with the specified keys are parsed and sent to the server.

Both the key and the value can be optionally surrounded by double quotes “ “ to define white space or other special characters.

When double quotes are used for the key or value, the backslash character “ \ “ can be used as the escape character. Any character following the backslash character is defined literally, including a double quote character or a backslash character. For example: “ \\ ”

Note the following considerations.

  • If the key in a key/value pair is not followed by an equals sign and a VALUE is not provided, the option is skipped, as with free text.

  • The key cannot be empty, the value can be empty.

  • An equals sign that is not followed by a value is treated as free text and is skipped.

  • A value can be a string of characters that are surrounded by double quote characters, or it can be empty. Use a backslash for escaping special characters that are part of the value.

KVP Parser Options

Note the following information about the structure of the kvp parser.

Option

Description

fields

The information that you want to extract described as units of data. For example, fields=name,lastname,country.

delimiter

Optional.

Default delimiters are the space character, tab, newline characters, comma, and semicolon characters.

If delimiters are not specified in the configuration, the kvp parser uses default delimiters for parsing.

To change the default delimiters to specific delimiters, you must define them between double quotes. For example: delimiter = "#^|". This definition means that each of the characters which are enclosed in the double quotes will be used as a delimiter. For the kvp parser, any character can be considered as delimiter. You an include the default delimiters with other delimiters in the definition.

For example, the delimiter = "#^|\t\r\n\s" statement includes the tab, newline characters, and the space as delimiters. If these characters are used, they must be preceded by the escape character. For example, to define the space character as a delimiter, enter the escape character " \" before the space character when defining it as a delimiter, for example, delimiter="\s".

field_decoder

Nested parsers are specified as a JSON string in which the keys are the names of the field to apply to the nested parser, and the value is the name of the parser to use for that field.

Each nested parser is applied to the appropriate field, as decoded by the base parser.

Field decoders are useful when the value of a key-value pair is a complex value such as a timestamp or a comma-separated list.

debug =

Optional. The debug = value an be yes or no. The default value for debug is debug=no for parsers.

When the option is set to yes, you can view detailed logs of the parser ingestion in liagent_<date>.log.

Additional Keys Value Options

Key

Definition

KVP_MESSAGE = *(MESSAGE_ENTRY [WSPR])

A list of message entries separated by optional white space

MESSAGE_ENTRY = KVP / FREE_TEXT

An entry is a key/value pair or just a free text

KVP = KEY ["=" VALUE]

Key/value pair. If KEY is not followed by an equal sign and VALUE, it is skipped like free text.

KEY = BARE_KEY / QUOTED_KEY

FREE_TEXT = "="

A free standing equal sign is considered a free text and is skipped.

BARE_KEY = *1BARE_KEY_CHAR

At least one character

BARE_KEY_CHAR = %0x00-08 / %0x10-19 / %0x21-3C / %3E-%FF

Any character excluding equal sign, space or TAB

QUOTED_KEY = 0x22 *1(QUOTED_STRING_CHAR / "\" CHAR) 0x22

At least one character surrounded by double quote characters. The backslash is used as an escape character.

QUOTED_STRING_CHAR = %0x00-21 / %0x23-FF

Any character excluding double quote

VALUE = BARE_VALUE / QUOTED_VALUE

BARE_VALUE = *BARE_VALUE_CHAR

Zero or more characters

BARE_VALUE_CHAR = %0x00-08 / %0x10-19 / %0x21-FF

Any character excluding space or TAB

QUOTED_VALUE = 0x22 *(QUOTED_STRING_CHAR / "\" CHAR) 0x22

A string of characters surrounded by double quote characters. This can be empty. The backslash is used as an escape character.

KVP Parser Configuration Examples

You can use fields=* to parse all fields, if required.

[parser|simple_kvp]
base_parser =kvp
fields=*

This example shows how to specify the field decoder.

[parser|mykvp]
debug=no
base_parser=kvp
delimiter="#^|"
fields=*
;OR fields=scope,abstract,lazyInit,autowireMode,dependencyCheck
field_decoder={"field1":"field1_parser1"}

[parser|field1_parser1]
base_parser=clf
format=[%{value1}i]]
field_decoder={"value1":"field_parser2"}

To parse the following KVP log:

Configuring transport... proto = cfapi server_hostname = LOCALHOST ssl = no port = 9000 reconnect = 30

Define the KVP parser configuration:

[parser|kvp_log_parser]
base_parser=kvp
fields=*

The KVP parser returns the following fields:

proto=cfapi
server_hostname=LOCALHOST
ssl=no
port=9000
reconnect=30

Simple and Complex KVP Parser Examples

Simple KVP Parser Example

[filelog|MyLog]
directory=C:\<folder_name>\Parser_logs
include=*.log
parser=my_KVP_parser

[parser|my_KVP_parser]
base_parser=kvp
fields=*

Complex KVP Parser Example

[filelog|MyLog]
directory=C:\<folder_name>\Parser_logs
include=*.log
parser=my_KVP_parser

[parser|my_KVP_parser]
base_parser=kvp
fields=*
field_decoder={"field1":" field1_parser1"}

[parser| field1_parser1]
base_parser=clf
format=[%{value1}i]]
field_decoder={"value1":" field1_parser2"}