You can configure common options for all parsers that produce named fields.
Reserved Words for Field Names
Field names are restricted. The following names are reserved and cannot be used as field names.
- event_type
- hostname
- source
- text
Common Parser Options
Options in the following table can be used with all supported parsers.
| Option | Description |
|---|---|
| base_parser | The name of the base parser that this custom parser extends. It can be a built-in parser name or another customer parser name. This configuration key is mandatory. |
| field_decoder |
Nested parsers specified as a JSON strings. Keys are the names of the field to apply the nested parser to and the value is the name of the parser to use for that field. Each nested parser is applied to the appropriate field decoded by the base parser. Field decoders are useful when the value of a field is a complex value, for example, a timestamp. The
field_decoder option also supports more comlpex JSON objects as arguments that allow you to use conditions for specific field values that are checked before the nested parser is applied.
Note: For more information on usage and conditional configurations, see Conditional Configurations for the field_decoder option section below.
|
| field_rename | Renames extracted fields. Use a JSON string where keys are the original names of the fields and values are the new desired names of the fields. The field_decoder option is always applied before field_rename. The order of these options in the INI file is not important. For clarity, specify field_decoder first. |
| next_parser | Name of the next parser to run. Allows multiple parsers to run sequentially on the same input.
Note: Parsers process all consequent parsers defined by the
next_parser keyword and may replace a field value already extracted by a previous parser.
|
| exclude_fields | A list of semicolon separated field names to remove from the event before it is delivered to the server. Field names are removed before event filtering is performed so that the field that you excluding during parsing cannot be used in the filter condition. |
| debug | Yes or No option that enables debugging of a particular parser. With debugging enabled, the parser performs detailed logging of input it receives, the operation it performed and the result it produced. The option applies per-section, that is, only to the parser defined by the particular section. The default value for debug is debug=no for parsers. |
Conditional configurations for the field_decoder option
For logs with the same common format but signficant differences related to specific field values, logs with info and errorseverities for example, you can use the conditional nested parser to reduce the application of unnecessary parsers to the corresponding fields of already parsed logs.
2019-03-29T11:00:54.858Z
host-FQDN Hostd: error hostd[2099230]
[Originator@6876 sub=Default opID=1983bdbe-c1-800f user=admin.user] AdapterServer caught
exception: SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read:
The connection was closed by the remote end during handshake.)
2019-03-29T11:00:55.477Z
host-FQDN Hostd: info hostd[6D620B70]
['commonhost' opID=5759adcc-cf] [transportConnector] -- FINISH task-internal-5726666 -- -- Completed connection restart --
You can use the following configuration to parse them:
[parser|clf_parser]
base_parser=clf
format=%t %{generator_host}i %i: %{log_severity}i %i[%{thread_id}i]%M
field_decoder={"log_message" : {"log_severity" : {"error" : "error_parser", "info" : "info_parser"}}}
exclude_fields=log_message
[parser|info_parser]
base_parser=clf
format=[%{common_info}i] [%{process}i] %M
field_rename={"log_message" : "info_log_content"}
[parser|error_parser]
base_parser=clf
format=[%{common_info}i] %{exception_handler}i %i:%{exception_type}i:%i:%{error_id}i:%i:%i:%i: %M
field_rename={"log_message" : "exception_content"}
This configuration produces the following results:
timestamp=2019-03-29T11:00:54.858000 generator_host="host-FQDN" log_severity="error" thread_id="2099230" common_info=Originator@6876 sub=Default opID=1983bdbe-c1-800f user=admin.user exception_handler="AdapterServer" exception_type="SSLExceptionE(SSL Exception" error_id="140000DB" exception_content="The connection was closed by the remote end during handshake.)"
Additioanlly the following fields are parsed for the info log:
timestamp=2019-03-29T11:00:55.477000 generator_host="host-FQDN" log_severity="info" thread_id="6D620B70" log_message="['commonhost' opID=5759adcc-cf] [transportConnector] -- FINISH task-internal-5726666 -- -- Completed connection restart --" common_info="'commonhost' opID=5759adcc-cf" process="transportConnector" info_log_content="-- FINISH task-internal-5726666 -- -- Completed connection restart --"
