There are multiple third-party Prometheus exporters and you may want to use them to monitor entities in your Kubernetes environment. You can ingest the metrics in vROps by modifying the configuration file in vRealize Operations to support third-party Prometheus exporters. For example, the following metric is received using the Prometheus collector and is in the Prometheus format:

kubernetes_system_container_memory_major_page_faults{container_name="kubelet",host="
vrops-telegraf-k8s-fvcvr", 
job="k8s_telegraf_job",namespace="kube-system",node_name="k8s-01-node-01"}

This metric has the following components:

  1. Metric Name: kubernetes_system_container_memory_major_page_faults.

  2. Label Name: container_name, host, job, namespace, node_name.

  3. Label Value: kubelet, vrops-telegraf-k8s-fvcvr, k8s_telegraf_job, kube-system", k8s-01-node-01.

Given a metric name and a set of labels, time series are frequently identified using this notation:

<metric name>{<label name>=<label value>, ...}

From the above sample metric, the service-related information can be extracted from the labels:

  1. container_name – contains the name of Container

  2. namespace – contains the name of Namespace

  3. node_name – contains the name of Node

For more details, refer to https://prometheus.io/docs/concepts/data_model/#notation.

To map any Prometheus Metric to vRealize Operations Objects such as Namespace, Node, Pod, and Container, the metric Label Value should possess the identifier or name of these service instances.Ensure that these labels have the right mappings to the objects you want to see the metric in, so that vRealize Operations can parse the labels and associate them to the right objects.

The management pack for Kubernetes contains an XML file packaged with the management pack that contains mappings for common Prometheus metrics. The metrics have been mapped to the appropriate objects and have alias names to make them readable. This file is not visible and cannot be edited.

For third-party exporters or custom Prometheus metrics, you can map metrics and provide alias names for the metrics using configuration files that can be edited from the vRealize Operations user interface. This configuration file should be in an XML format and follows a format so that vRealize Operations can perform the mapping correctly. To ensure that vRealize Operations can perform the mapping correctly, some identifier information is mandatory in the labels of the Prometheus metrics. The following objects are supported by the configuration file and the identifier information required for them are as follows:

Object Type in vRealize Operations Identifier Information Required in Metric as part of the Label Value
Kubernetes Namespace Kubernetes Name
Kubernetes Node Node Name
Kubernetes Deployment Deployment Name
Kubernetes ReplicaSet Namespace Name, ReplicaSet Name
Kubernetes Pod Namespace Name, Pod Name
Container (Namespace Name, Pod Name, Container Name ) OR only Container ID
Note:
  • Best practices:

    • All the values of promMetricName and ruleName in the mapping files should be unique. If there are duplicate values, the existing mappings will be overridden.

    • To map a metric against the respective Kubernetes object in vRealize Operations, the metric should contain the necessary identifier information as part of its labels.

    • Since the promMetricName field is case-sensitive, the exact metric name received from Prometheus should be provided under the mappings.

    Limitations:

    • Ensure that duplicate metric names are not generated by the exporter as it will cause conflicts in mapping across the service objects in vRealize Operations.

    • For rule based metric mappings, the provided rule should return only 1 metric for a resource instance and can return multiple metrics in case of instanced nature.

    • The Return values of the provided Rule should be a valid metric or numeric instead of NaN value.

The steps to set up the configuration file for the first time are as follows:

  1. Click Configure in the left menu.

  2. Click Configuration Files > Solution Configuration.

  3. Click Add.

  4. Enter the name of the file as k8s-custom-prom-metric-mappings.xml.

  5. Add the mapping definitions and click Save.

  6. Perform Redescribe for all the Kubernetes adapters.

    1. Click Administration in the left menu.

    2. Click Redescribe.

Once the file has been created for the first time, you can add more metrics and map them by editing the file. The steps to edit the configuration file are as follows:

  1. Click Configure in the left menu.

  2. Click Configuration Files > Solution Configuration.

  3. Select the file k8s-custom-prom-metric-mappings.xml.

  4. Click the ellipse (…) icon and click Edit.

  5. Add or edit the mapping definitions and click Save.

  6. Perform Redescribe for all the Kubernetes adapters.

    1. Click Administration in the left menu.

    2. Click Redescribe.

Description of the XML fields

You can configure the XML file to add custom metrics or metrics from third-party exporters by making appropriate entries in the XML file. The sections and fields in the XML file are described below.

Below is a sample file for reference. 
<MappingDefinitions
    xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <CustomMetricConfig collectCustomMetrics="false"
        enableAllLabels="false"
        labelNamesToExclude="beta_kubernetes_io_os,container_label_annotation_io_kubernetes_container_hash,container_label_annotation_io_kubernetes_container_ports,container_label_annotation_io_kubernetes_container_restartCount,container_label_annotation_io_kubernetes_container_terminationMessagePath,container_label_annotation_io_kubernetes_container_terminationMessagePolicy,container_label_annotation_io_kubernetes_pod_terminationGracePeriod,container_label_annotation_kubeadm_kubernetes_io_etcd_advertise_client_urls,container_label_annotation_kubeadm_kubernetes_io_kube_apiserver_advertise_address_endpoint,container_label_annotation_kubernetes_io_config_hash,container_label_annotation_kubernetes_io_config_seen,container_label_annotation_kubernetes_io_config_source,container_label_app,container_label_app_kubernetes_io_name,container_label_app_kubernetes_io_version,container_label_component,container_label_controller_revision_hash,container_label_description,container_label_io_kubernetes_container_logpath,container_label_io_kubernetes_docker_type,container_label_io_kubernetes_pod_uid,container_label_io_kubernetes_sandbox_id,container_label_k8s_app,container_label_maintainer,container_label_maintainers,container_label_pod_template_generation,container_label_pod_template_hash,container_label_purpose,container_label_tier,container_label_version,container_runtime_version,dockerVersion,goversion,image,image_id,instance,timezone,container_label_io_kubernetes_container_name,container_label_io_kubernetes_pod_name,container_label_io_kubernetes_pod_namespace,name"
        maxGroupNames="3"></CustomMetricConfig>
    <IdentifierMappings>
        <NodeIdentifierMapping
            labelContainingNodeName="nodename,node_name,node,kubernetes_node_name" />
        <NamespaceIdentifierMapping
            labelContainingNamespaceName="namespace,kubernetes_namespace,container_label_io_kubernetes_pod_namespace" />
        <PodIdentifierMapping
            labelContainingPodName="pod,pod_name,podname,container_label_io_kubernetes_pod_name" />
        <ContainerIdentifierMapping
            labelContaingContainerID="id,container_id"
            labelContainingContainerName="container_name,container" />
        <DeploymentIdentifierMapping
            labelContainingDeploymentName="deployment" />
        <ReplicasetIdentifierMapping
            labelContainingReplicasetName="replicaset" />
    </IdentifierMappings>
    <MappingDefinition resourceKind="node">
        <MetricGroup groupName="Summary" isInstanced="false">
            <Metric instancedLabelName="" metricName="Node Boot Time" metricUnit="sec" promMetricName="node_boot_time_seconds" isRate="false"/>
        </MetricGroup>
    </MappingDefinition>
</MappingDefinitions>