Event Processing Manager configuration

This module must be declared in the Event-Processing-Manager <DCF Install>/Event-Processing/ Event-Processing-Manager/<instance name>/conf/processing.xml file in order to be used.

<!-- Events Property Tagger -->
<processing-element name="EPT" config="Event-Property-Tagger/Default/conf/event-property-tagger
.xml" data="Next" />

Each Event Property Tagger instance must be declared with its own processing-element tags with any name as long as it is unique.

Event Property Tagger configuration

Configuration file: The event-property-tagger.xml configuration file is needed to run the module. It is located at: <DCF Install>/Event- Processing/Event-Property-Tagger/<instance name>/conf.

<?xml version="1.0" encoding="UTF-8"?>
<property-tagging-filter-config
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.watch4net.com/APG/Filter/PropertyTaggingFilter
property-tagging-filter-config.xsd"
xmlns="http://www.watch4net.com/APG/Filter/PropertyTaggingFilter">
<refresh unit="hours">1</refresh>
<files>
<text-file path="input.data" encoding="UTF-8">
<!-- How to process data, default is optimized. linear is also available. -->
<data-processing type="optimized" />
<!-- The field separator for the input file -->
<field-separator>,</field-separator>
<!-- The quoting character. Double it to escape. -->
<field-quoting>"</field-quoting>
<!-- The default symbol for the input file. To use with key properties only. -->
<default-symbol>**</default-symbol>
<!-- The null symbol for the input file. Means property not there. -->
<null-symbol>@@</null-symbol>
<!-- Match-all symbol. To use with key properties only. -->
<match-all-symbol>%%</match-all-symbol>
<!-- Use to insert property value for new properties. To use with new properties
only.-->
<property-insertion start="[" end="]" />
<key-properties>
<key-property delete-after-use="false" string-type="string" decoder="IP">
src_addr</key-property>
<key-property delete-after-use="false" string-type="regex" decoder="INT_NATIVE"
parser="INT_NATIVE">src_port</key-property>
<!-- more key-property can be added -->
</key-properties>
<new-properties>
<new-property encoder="IP">failover_server</new-property>
<new-property encoder="INT">system_id</new-property>
<!-- more new-property can be added -->
</new-properties>
</text-file>
<!-- more text-file can be added -->
</files>
</property-tagging-filter-config>

The refresh tag is mandatory. The unit can be days , hours , minutes or seconds . The value must be a positive integer. This amount of time indicates to the module when to verify for file modifications and reload them when needed. Setting the value to 0 will shut down the automatic refresh. A restart of the module with a new configuration file without 0 for the refresh value will be needed to reenable it. It is not recommended to choose a low refresh rate (under 5 minutes), because loading configurations slows down the processing. The file tag is mandatory. It contains the configuration for each input file. There is one text-file for each input file. The path of the input file is mandatory. The module will scan input files in the order they appear in the configuration file. The encoding attribute is optional. Its default value is UTF-8.

The data-processing type tag is optional. Its default type is optimized (Same as before). It is also possible to set it to linear to allow data processing line by line with no exceptions. That means no default behaviour and no performance improvement from combining similar rows. The field-separator tag is mandatory. It sets the separator used in the input file to split fields from a line. It can be more than one character. Be aware that if you put a space or any other whitespace character between the opening and the closing tag, these will be used, taken as is, to split the fields.

The field-quoting tag is optional and defaults to ". It sets the quoting string used in the input file to quote fields on a line. Quoting field is optional but you can use it only to escape special characters and the field separator. Double the quotes to escape it if quotes appear in the values themselves. The default-symbol is mandatory. It may be empty if no default symbol is needed. You should not use a symbol that can conflict with property names, SQL patterns, or regular expressions. It is used to indicate everything else that has not yet been matched. For more information on how different value types are processed when there are one or more default-symbols used, see General Concepts on Matching Order section.

The null-symbol is optional. It may be empty if no null symbol is needed. You should not use a symbol that can conflict with property names, SQL patterns or regular expressions. It is used to signify a "the property does not exist" symbol for key-property . It can also be used to prevent a new property from being created when placed in a new-property column in the data file. The match-all-symbol is optional. It is used to always match for a key-property even if the property does not exist. It is not available for use with string string-type as its behaviour would conflict with it. The property-insertion is optional. It is used only for new-properties . It allows the insertion in the value of a new property, the value of another property, using the target property's name. For example server-[type] results in server-mail assuming the property type has the value mail . The key-properties tag is mandatory. It contains all properties that, all together, are the "key". The key-property tag defines a property that will be used to compose the "key". If no key-property is defined the new properties will be added to each event processed by the module. There is no limit to the number of key properties defined, while the order in which they appear in the file must match the columns of the input file. The value represents the name of a property to search in the event. The delete-after-use attribute is optional and false by default. If set to true, the property will be deleted from the processed event AFTER all the filtering is done. That means the property can be reused in the configuration file to create other new properties. The string-type attribute defines the type of the value. It can be a string , an sqlpattern , a regex or a range. Using only the string type makes the module run faster because it does not have to check each line for each property which it does with sqlpattern s, regex es and ranges. The decoder attribute is optional and defines the kind of content that the field will have and it will decode it in a readable format (ex: the IP decoder will take a raw IP address in a flow event and will translate it to the dotted format). The possible decoders are IP (both IPv4 and IPv6), INT and STRING which is the default decoder if omitted.

The parser attribute is optional and defines how the configuration csv value will be loaded into the tagging engine. This allows direct match to be faster for some case where you don't need to convert the incoming data but only the configuration data and decode the incoming data into it's native representation. The current implementations are the following:
  • BYTES: This will retrieve a field value from the event through getBytes().
  • INT NATIVE: This will retrieve a field value from the event through getInt().
  • INT: This will retrieve a field value from the event through getString() and then convert it to int.
  • IP: This retrieve the bytes from the event through getBytes() call and will create either an IPv4 or IPv6 string representation depending on the number of bytes associated to that field.
  • STRING: Default implementation which returns a String object for comparison The new-properties tag is mandatory. It contains all new properties to add to events that will match a key.

The new-property tag defines a new property to add to an event that will match the "key". The indicated value becomes the name of the new property in the output. At least one tag is needed. There is no limit to the number of new properties. The order they appear in the configuration must match the order their values appear in the input file.

The encoder attribute is optional and defines the kind of content added in the event for the field (ex: the IP encoder will take a dotted formatted address and insert it in his numeric format). The possible encoders are IP , INT , and STRING : which is the default encoder if the encoder attribute is omitted. The IP encoder supports IPv4 in dot notation and IPv6 in hexadecimal notation. If the encoding field is an IPv6 address, it will insert two numeric fields if the conversion is successful and the fields will be have the suffix 0 and 1.

Input file: The input file uses a "chosen string" as a seperator for the values it lists. That means that CSV files are supported if in the configuration file the field-separator is set to , . Fields with an embedded "chosen strings" (in this case ",") must be delimited with double-quotes ( " ). Blank spaces before and after will be trimmed if they are not embedded in double-quotes. If you wish to use a double-quotes character ( " ) in one of the fields, the whole field must be surrounded by double-quote characters and each of the embedded double-quotes characters must be doubled with another double-quotes. So, if you wish to have the following field:

This is a "quoted" field

You must quote the whole field like the following:

"This is a " "quoted"" field "

This represents an input file that can be used with the previous configuration file.

192.168.1.25,69*,192.168.5.69,25

192.168.1.69,25*,192.168.5.25,69

In this case, the first column represents the src addr values, the second column the src port values, the third column the failover server values and the fourth column the system id values. Refer the configuration file, the two first columns represent the "key" and the two last the new properties to add to the event.

The column order depends on the configuration file. The key properties are always in the first columns and the new properties are always in the last columns.

KeyProperty1,KeyProperty2,...,KeyPropertyN,NewProperty1,NewProperty2,...,NewPropertyM

For example, we have an event that has a property named src addr set to 192.168.1.25 . If the module also matches on the second entry, then the module adds the new properties failover server and system id.

If a line does not match with key properties and new properties, it will be ignored and a log entry will be created to warn you about it.
Note: An event could have other event properties besides those used for the key and for the matching. These properties are simply ignored in the Event Property Tagger , but are preserved in the events themselves.

Accessors

File Accessor: The file accessor is used to fetch data from a text file. The format of the data is determine by the field-separator and field-quoting parameter. For example, if the field-quoting is equal to ” and the field-separator is equal to , then the data should look like this:

”device1”,” interface ” , ”eth0”, ”desc1”,”desc2”
”device1”,” interface ” , ”eth1”, ”description1” , ”description 2” 

Configuration example:

<accessor accessor-class="FileAccessor">
<parameter name="field-separator">,</parameter>
<parameter name="field-quoting">"</parameter>
<parameter name="encoding">UTF-8</parameter>
<parameter name="path">conf/default-values.csv</parameter>
</accessor> 

The field-separator parameter is optional. It is the separator characters of the data file. If it's not set, then it will be equal to the field-separator parameter of the text-file.

The field-quoting parameter is optional. It is the quoting character of the data file. If it's not set, then it will be equal to the field-quoting parameter of the text-file.

The encoding parameter is optional. It is the encoding of the data file. If it's not set, then it will be equal to the encoding parameter of the text-file.

The path parameter is mandatory. It is the path to the data file, it can be a relative or absolute path.

Static Accessor: The Static accessor is used when you only have a few line of data. Instead of having a file accessor the data can be directly put in the configuration. It is useful for default values. Like the file accessor, the format of the data is determine by the field-separator and field-quoting parameter.

Configuration example:

<accessor accessor-class="StaticAccessor">
<parameter name="field-separator">,</parameter>
<parameter name="field-quoting">"</parameter>
<parameter name="line">**,**,eth0,desc1,desc2</parameter>
<parameter name="line">**,**,**,description1,description 2</parameter>
</accessor>

In the above example the default-symbol is equal to **.

The field-separator parameter is optional. It is the separator characters of each line . If it's not set, then it will be equal to the field-separator parameter of the text-file.

The field-quoting parameter is optional. It is the quoting character of each line . If it's not set, then it will be equal to the field-quoting parameter of the text-file.

The line parameter is optional. It can be repeated multiple times for each line of data.

Smarts domain tagging accessor: The smarts domain tagging accessor is used to fetch domain tagging data from Smarts. It does so by retrieving all the ICS DomainTagConfiguration instances and extract the pattern property. After that, it follows the containment link of the tags property and extracts the Name and tagString properties. This accessor is static in the number of fields it returns. The return entry list is defined as follow: "domain", "pattern", "tag" where domain "ICS-Domain-" prefix is stripped.

Configuration example:

<accessor accessor-class="SmartsDomainTaggingAccessor">
<parameter name="auth-mode">BROKER</parameter>
<parameter name="host">localhost</parameter>
<parameter name="port">426</parameter>
<parameter name="broker-user">admin</parameter>
<parameter name="broker-pass">changeme</parameter>
<parameter name="domain-user">admin</parameter>
<parameter name="domain-pass">changeme</parameter>
<parameter name="domain">INCHARGE-SA</parameter>
</accessor>

The auth-mode parameter is mandatory. The possible values are : BROKER or DOMAIN. The broker mode means that the authentication must pass by the broker, domain will be a direct connection to the domain manager.

The host parameter is mandatory. It is the host or ip of the Smarts you want to connect to.

The port parameter is mandatory. It is the broker or domains port of Smarts.

The broker-user parameter is mandatory when BROKER auth mode. It is the broker user name.

The broker-pass parameter is mandatory when BROKER auth mode. It is the broker user's password.

The domain-user parameter is mandatory when DOMAIN auth mode and optional in BROKER auth mode. It is the domain user name.

The domain-pass parameter is mandatory when DOMAIN auth mode and optional in BROKER auth mode. It is the domain user's password.

The domain parameter is mandatory. It is the name of the domain to attach to.

Smarts retrieve groups accessor: The smarts retrieve groups accessor is used to extract all the hierarchical groups from Smarts to tag all the referees of the group. It does so by retrieving all the HierarchicalGroup instances and extract the ParentGroup , ChildGroups , ConsistsOf and Name properties. It will follow all the parents and childs groups to get all the groups related to one instance of an object. After that, it uses the the instance name of the reference link of the ConsistsOf property. This accessor is static in the number of fields it returns. The return entry list is defined as follow : "instance name of the referee", "group" where groups is the a concatenated list split by the separator property. When retrieving the group names, we compute the name of a group according to it's hierarchy. That means that it's name plus all it's parent will be concatenated together.

Configuration example:

<accessor accessor-class="SmartsRetrieveGroupsAccessor">
<parameter name="auth-mode">BROKER</parameter>
<parameter name="host">localhost</parameter>
<parameter name="port">426</parameter>
<parameter name="broker-user">admin</parameter>
<parameter name="broker-pass">changeme</parameter>
<parameter name="domain-user">admin</parameter>
<parameter name="domain-pass">changeme</parameter>
<parameter name="domain">INCHARGE-SA</parameter>
<!-- Separator for the groups. Defaults to | -->
<parameter name="group-separator">|</parameter>
<!-- Separator for the parent names. Optional, defaults to unspecified. -->
<parameter name="name-separator">/</parameter>
<!-- Separator for the groups. Defaults to Name -->
<parameter name="group-name-property">Name</parameter>
<!-- Separator for the groups. Defaults to Name -->
<parameter name="device-name-property">Name</parameter>
</accessor>

The auth-mode parameter is mandatory. The possible values are : BROKER or DOMAIN. The broker mode means that the authentication must pass by the broker, domain will be a direct connection to the domain manager.

The host parameter is mandatory. It is the host or ip of the Smarts you want to connect to.

The port parameter is mandatory. It is the broker or domains port of Smarts.

The broker-user parameter is mandatory when BROKER auth mode. It is the broker user name.

The broker-pass parameter is mandatory when BROKER auth mode. It is the broker user's password.

The domain-user parameter is mandatory when DOMAIN auth mode and optional in BROKER auth mode. It is the domain user name.

The domain-pass parameter is mandatory when DOMAIN auth mode and optional in BROKER auth mode. It is the domain user's password.

The domain parameter is mandatory. It is the name of the domain to attach to.

The group-separator parameter is optional. It is the string used as group separator. It defaults to the pipe ( - ) character.

The name-separator parameter is optional. It is the string used to concatenate the parent group names with. If not specified, the parent group names will not be concatenated.

The group-name-property parameter is optional. It is the property to use on the group instance to use as name.

The device-name-property parameter is optional. It is the property to use on the device instance to use as name.

Post processing

Post processing is used to modify data obtained through accessors. It allow to apply regular expression on values (column) and for whole lines. The column number starts at 1.

<accessor accessor-class="StaticAccessor">
<parameter name="line">deviceA,groupAA</parameter>
<post-process>
<column replacement="$1" column="2" regex="(a+)bb" />
<!-- more columns regex -->
<line replacement="$1,$2,$1" regex="^(.+),(.+),(.+)$" />
<!-- more line regex -->
</post-process>
</accessor>