Querying the Metric Store

Learn how to configure and use Metric Store.

Metric Store implements the Prometheus Query Language for querying metrics for which you have access.

Prerequisites

• Completed installation of the Metric Store Tile.
• curl
• CF Auth Token

Using Metric Store

Authorization & Authentication

When querying the API through HTTPS, each request must have the Authorization header set with a UAA provided token.

For Metric Store admins

As a Metric Store admin, you have access to all recorded metrics (platform and application) and can run Label and Series queries. You need to add either doppler.firehose or logs.admin scope to your admin account.

You then can run the following query:

cf login
curl -vvv -H "Authorization: $(cf oauth-token)" -G https://metric-store.SYSTEM-DOMAIN/api/v1/label/source_id/values

The status code is 200 if everything is working properly.

For Tanzu Application Service developers

As a Tanzu Application Service developer, you can query metrics for applications that you have access. When querying with PromQL, you must specify the source_id label with the application guid as the label value.

For example:

curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" \
  --data-urlencode "query=cpu{source_id=\"$(cf app --guid your-app-name)\"}"

Standard Tanzu Application Service application metrics

Metric Name	Description	PromQL Example
cpu	CPU usage percentage	avg(cpu{source_id=“$app_guid”})
memory	Memory usage in bytes	avg(memory{source_id=“$app_guid”})
memory_quota	Memory Quota allocated in bytes	memory{source_id=“$app_guid”} / memory_quota{source_id=“$app_guid”}
disk	Disk usage in bytes	avg(disk{source_id=“$app_guid”}) / 1024 / 1024 / 1024 # in gigabytes
disk_quota	Disk quota allocated in bytes	disk{source_id=“$app_guid”} / disk_quota{source_id=“$app_guid”}
http_total	HTTP request counts includes status	sum(rate(http_total{source_id=“$app_guid”})) by (status)
http_duration_seconds_*	HTTP request request latencies bucketed by duration	histogram_quantile(0.95, sum(rate(http_duration_seconds_bucket[5m])) by (le))

Querying through Prometheus-Compatible HTTP Endpoints

Notes on PromQL

The ultimate goal of these endpoints is to create a fully-compliant, Prometheus-compatible interface. This allows tools such as Grafana to work directly with Metric Store without any additional translation.

A valid PromQL metric name consists of the character [a-Z][0-9] and underscore. Names can begin with [a-Z] or underscore. Names cannot begin with [0-9]. As a measure to work with existing metrics that do not comply with the previous mentioned format a conversion process takes place when matching on metric names. Any character that is not in the set of valid characters is converted to an underscore. The metric is not changed in the store.

For example, to match on a metric name http.latency use the name http_latency as a search term.

GET /api/v1/query

Issues a PromQL instant query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

• query is a Prometheus expression query string.
• time is an optional UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s).

Admins (doppler.firehose or logs.admin) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" --data-urlencode 'query=metric_name_0{source_id="source_id_0"}'

Response Body

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {"__name__": "metric_name"},
        "value": [ "<timestamp>", "<value>" ]
      },
      "..."
    ]
  }
}

GET /api/v1/query_range

Issues a PromQL range query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

• query is a Prometheus expression query string.
• start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
• end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
• step is a query resolution step width in duration format or float number of seconds.

Admins (doppler.firehose or logs.admin) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query_range" \
    --data-urlencode 'query=metric_name_0{source_id="source_id_0"}' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)" \
    --data-urlencode 'step=1h'

Response Body

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {"__name__": "metric_name"},
        "values": [
          [ "<timestamp>", "<value>" ],
          "..."
        ]
      },
      "..."
    ]
  }
}

GET /api/v1/series

Issues a PromQL series query against Metric Store data. You can read more detail in the Prometheus documentation here.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Query Parameters:

• start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
• end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
• match[] is a series selector.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/series" \
    --data-urlencode 'match[]=metric_name_0' \
    --data-urlencode 'match[]=metric_name_1' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)"

Response Body

{
  "status": "success",
  "data": [
    {
      "__name__": "metric_name_0",
      "source_id": "source_id_0",
      "...": ".."
    },
    "..."
  ]
}

GET /api/v1/labels

Retrieve all label names for authorized source ids from the store.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Request

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/labels"

Response Body

{
  "status": "success",
  "data": [
    "__name__",
    "deployment",
    "..."
    "ip",
    "origin",
    "request_type",
    "source_id",
    "status_code",
    "unit"
  ]
}

GET /api/v1/label/<label_name>/values

Retrieve label values by label_name for all authorized source ids. The special label __name__ can be used to retrieve metric names from the store.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Request

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/<label-name>/values"

Response Body

{
    "status":"success",
    "data":["10", "1"]
}

Request

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/__name__/values"

Response Body

{
    "status":"success",
    "data":["metric_name_0", "metric_name_1", "..." ]
}