The <Metric> section is the core element in the configuration XML where the metric mappings are actually performed. This section contains information about how a Prometheus metric or recording rule is to be dynamically renamed into VMware Aria Operations, and added to the default policy. This section also holds other details such filtering by label name (supporting both positive and negative filtering) and instance information (if the instanced label names are provided).
The attributes of this section are as follows:
Attribute | Description |
---|---|
Metric | Contains the name of metric as it should appear in the VMware Aria Operations user interface. |
promMetricName | Contains the exact metric name which is present in Prometheus. Note: This attribute is not required if a recording rule is provided in the rule attribute. |
metricUnit | Contains the name of the unit to measurethe value of the metric. The supported units are provided in the next section. |
instancedLabelName | Holds the list of the exact Prometheus metric label names containing the instance information over which the metrics can be grouped. |
isRate | Contains whether the metric is in a rate format in Prometheus. Set this attribute to true if the metric is in rate form. |
filterByLabelName | This attribute is used to filter or drop the metric based on matching a label name. Both positive and negative matching can be done. |
Rule | Contains the exact recording rule with label matchers that needs to be executed on the Prometheus server. |
ruleName | Contains the name of the rule which will be used by VMware Aria Operations internally. This attribute is required only when ruleis provided. |
The table below lists the supported units for the metrics added in the <Metrics> section:
Bytes |
KB |
MB |
GB |
TB |
Bps |
KBps |
MBps |
Hz |
KHz |
MHz |
GHz |
Count |
Percent |
IOPS |
Nanoseconds |
Microseconds |
Milliseconds |
Seconds |
Minutes |
Hours |
Days |
Configuring a Non-Instanced Metric
Below is a sample metric configuration in the XML where a non-instanced Prometheus metric kubernetes_node_fs_available_bytes is to be mapped in VMware Aria Operations. The display name chosen for this metric is Kubernetes Node Fs Available Bytes with the unit of Bytes. Once this metric is configured in the XML, it will also be added to the default policy so that this can be controlled through the Default Policy on the VMware Aria Operations.
<Metric instancedLabelName="" metricName="Kubernetes Node Fs Available Bytes" metricUnit="Bytes" promMetricName="kubernetes_node_fs_available_bytes" isRate="false"/>
Instanced Metric Configuration
Below is a sample metric configuration in the XML where an instanced Prometheus metric node_filesystem_avail_bytes is to be mapped in VMware Aria Operations. The display name chosen for this metric is File System Available Bytes with the unit of Bytes. The label names that contains the instance information should be added in the instancedLabelName attribute in a comma separated format. VMware Aria Operations will use the same order of labels to create the hierarchy in the metrics section of the user interface. Once this metric is configured in the XML, it will also be added to the default policy so that this can be controlled through the Default Policy on the VMware Aria Operations user interface.
<Metric instancedLabelName="device, mountpoint" metricName="File System Available Bytes" metricUnit="Bytes" promMetricName="node_filesystem_avail_bytes" isRate="false"/>
Setting Up Recording Rules
The <Metric> section also provides the capability of adding recording rules which will be executed on the Prometheus server during runtime. These rules will added as new metrics in VMware Aria Operations. While creating a rule, it is necessary to have a label matcher section for any metric present in the rule using {}. See the example to understand when the label matchers and placeholders are required and how to configure a rule in the XML.
The table below describes the placeholders required for each rescourceKind attribute
Value of resourceKind attribute | Placeholder to be added label matcher group |
---|---|
Namespace | <label_name>="$NAMESPACE_NAME" |
Node | <label_name>="$NODE_NAME" |
Deployment | <label_name>="$DEPLOYMENT_NAME" |
Replicaset | <label_name>="$REPLICASET_NAME" |
Pod | <label_name>="$POD_NAME" |
container (if metric contains only container name) | <label_name>="$CONTAINER_NAME" |
container (if metric contains only container id) | <label_name>=~"$CONTAINER_ID" |
Note:
- For recording rule based metric mappings, ensure that the provided rule returns only one metric and value for a metric that does not have instanced metrics. Conversely, ensure that the rule can return multiple metric values in case of instanced metrics.
- The rule should contain the respective placeholder constants inside label matcher group{} for any resourceKind.
- The <label_name> mentioned in the above table should be replaced with correct label name according to the exporter used.
The following example shows how a recording rule can be mapped and translated into a metric in VMware Aria Operations. In this example, consider a metric that you want for each container object.
- Define the rule as you would run it on Prometheus. In this example, the rule performs a rate calculation on the container metric container_cpu_usage_seconds_total. This rule does not contain any label matcher.
rate (container_cpu_usage_seconds_total{container!="POD",pod!="",image!=""}[5m])
This rule will result in rate metrics for all the containers since a specific container name or ID has not been provided as a label matcher.
- To calculate the same rate metric for a specific container, add a label matcher. The rule would now look as shown below:
rate (container_cpu_usage_seconds_total{container!="POD",pod!="",image!="",id=~".76fbece3d569f7a2e3fe63abd6a0da3f84db87e9aa7cc8013f28704d28889388."}[5m])
The label matcher here is id i.e id=~".*76fbece3d569f7a2e3fe63abd6a0da3f84db87e9aa7cc8013f28704d28889388.*" . The rule will now return only one metric value since the specific container ID has been specified using label name id.
- Create the rule for VMware Aria Operations using a label matcher for the container and a placeholder to denote that the metric should be calculated for each container separately.
rate (container_cpu_usage_seconds_total{container!="POD",pod!="",image!="",id=~"$CONTAINER_ID"}[5m])
This is the final rule where the specific container id has been replaced with a placeholder $CONTAINER_ID. This placeholder constant will be dynamically replaced with the appropriate container ID during the collection cycle. Similarly, you can create rules for other resources such as namespaces, nodes, pods, deployments and ReplicaSets with the respective placeholders.
- Configure the rule in the XML file with the respective attribute information.
- Configure the rule in the XML file with the respective attribute information.
<MetricGroup groupName="Rate Metrics (Instanced)" isInstanced="true"> <Metric instancedLabelName="cpu" metricName="Container Cpu Usage Seconds Total[5m]" metricUnit="sec" rule='rate(container_cpu_usage_seconds_total{container!="POD",image!="",id=~"$CONTAINER_ID"}[5m])' ruleName="container_cpu_usage_seconds_total[5m]" isRate="true"/> </MetricGroup>
Filtering a metric using label Name
The <Metric> section also provides the additional capability to conditionally map and rename a Prometheus metric by matching the label names using filterByLabelName. The filtering can be done for both positive and negative match. Adding the ! operator to the label name will skip metrics containing specified label name.
<Metric instancedLabelName="" filterByLabelName="!cpu" metricName="Container Cpu Usage Seconds Total[5m]" metricUnit="sec" rule='rate(container_cpu_usage_seconds_total{container!="POD",image!="",id=~"$CONTAINER_ID"}[5m])' ruleName="container_cpu_usage_seconds_total[5m]" isRate="true"/>