Fields are a powerful way to add structure to unstructured events and allow the manipulation of both the textual and visual representation of data.
Fields are one of the most important items in a content pack as they can be used in different ways including aggregations and filters. Aggregations allow you to apply functions and groupings to fields. Filters allow you to perform operations over fields.
You must extract any part of a log message that might be applicable to a query or aggregation. Fields are a type of regular expression query and are useful for complex pattern matching so you do not need to know, remember, or learn complicated regular expressions.
| Field Context Value | Definition |
|---|---|
| Regex before value | Include as many keywords as possible. If this field is empty or only contains special characters, then the Regex after value must include keywords. |
| Regex after value | Include as many keywords as possible. If this field is empty or only contains special characters, then the Regex before value must include keywords. |
| Name | Only use alphanumeric characters. Ensure all characters are lower case and use underscores instead of spaces as this makes fields easier to view. Keep in mind that names for content pack fields and user fields can be the same, though content pack fields will have a namespace in parenthesis to the right of the field name. Prefix content pack fields with an abbreviation, for example vmw_, to avoid confusion. |
| Keyword Search Terms | One or more keywords, separated by space, that appear within events containing the field. |
| Filter | A static field, operator, and a potential value that appears within events containing the field. It is common to use this in conjunction with the vRealize Log Insight agent and tags for events that do not contain keywords. |
| Information ("i" button) | Used to provide information about the field including what it means, what potential values could be returned, and possibly a user-friendly mapping of values to human-understandable information. |
Best Practices
In addition to the various components that make up a field, several best practices apply.
- Only create fields for regular expression patterns. If a field can be queried using keyword queries, or will only ever return a single value, then use keyword queries instead of a pre-defined field. If a field will only return two values then consider constructing individual queries instead of extracting a field. Fields are meant to add structure to unstructured data as well as provide a way to query over specific parts of an event.
- Only create fields for regular expression patterns that return a fraction of the total events. Fields that will match most events and/or return a very large number of results are not a good candidate for field extraction. The regular expression will need to be applied to a large quantity of events resulting in a resource intensive operation. If possible add additional keywords to reduce the number of results returned and optimize the query.
- If a field contains keywords within regular expression syntax, then add such keywords as a filter without regular expression syntax. For example, if the value or the context of a field contains keywords within regular expression syntax such as this|that, then add the keywords as a text filter to optimize the query like text contains this, that.
- Use of the additional context with one or more keywords is recommended over complex regular expressions in the before or after context.
- Add additional context to all extracted fields in order to optimize query performance.
