Learn how to configure and use Metric Store.

Metric Store implements the Prometheus Query Language for querying metrics for which you have access.

Prerequisites

Using Metric Store

Authorization & Authentication

When querying the API through HTTPS, each request must have the Authorization header set with a UAA provided token.

For Metric Store admins

As a Metric Store admin, you have access to all recorded metrics (platform and application) and can run Label and Series queries. You need to add either doppler.firehose or logs.admin scope to your admin account.

You then can run the following query:

cf login
curl -vvv -H "Authorization: $(cf oauth-token)" -G https://metric-store.SYSTEM-DOMAIN/api/v1/label/source_id/values

The status code is 200 if everything is working properly.

For Tanzu Application Service developers

As a Tanzu Application Service developer, you can query metrics for applications that you have access. When querying with PromQL, you must specify the source_id label with the application guid as the label value.

For example:

curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" \
  --data-urlencode "query=cpu{source_id=\"$(cf app --guid your-app-name)\"}"

The following table shows the standard Tanzu Application Service application metrics.

Metric Name Description PromQL Example
cpu CPU usage percentage avg(cpu{source_id=“$app_guid”})
memory Memory usage in bytes avg(memory{source_id=“$app_guid”})
memory_quota Memory Quota allocated in bytes memory{source_id=“$app_guid”} / memory_quota{source_id=“$app_guid”}
disk Disk usage in bytes avg(disk{source_id=“$app_guid”}) / 1024 / 1024 / 1024 # in gigabytes
disk_quota Disk quota allocated in bytes disk{source_id=“$app_guid”} / disk_quota{source_id=“$app_guid”}
http_total HTTP request counts includes status sum(rate(http_total{source_id=“$app_guid”})) by (status)
http_duration_seconds_* HTTP request request latencies bucketed by duration histogram_quantile(0.95, sum(rate(http_duration_seconds_bucket[5m])) by (le))

Querying through Prometheus-Compatible HTTP Endpoints

Notes on PromQL

The ultimate goal of these endpoints is to create a fully-compliant, Prometheus-compatible interface. This allows tools such as Grafana to work directly with Metric Store without any additional translation.

A valid PromQL metric name consists of the character [a-Z][0-9] and underscore. Names can begin with [a-Z] or underscore. Names cannot begin with [0-9]. As a measure to work with existing metrics that do not comply with the previous mentioned format a conversion process takes place when matching on metric names. Any character that is not in the set of valid characters is converted to an underscore. The metric is not changed in the store.

For example, to match on a metric name http.latency use the name http_latency as a search term.

GET /api/v1/query

Issues a PromQL instant query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

Admins (doppler.firehose or logs.admin) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" --data-urlencode 'query=metric_name_0{source_id="source_id_0"}'

And this is the corresponding response body:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {"__name__": "metric_name"},
        "value": [ "<timestamp>", "<value>" ]
      },
      "..."
    ]
  }
}

GET /api/v1/query_range

Issues a PromQL range query against Metric Store data. You can read more detail in the Prometheus documentation here.

Query Parameters:

  • query is a Prometheus expression query string.
  • start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
  • end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
  • step is a query resolution step width in duration format or float number of seconds.

Admins (doppler.firehose or logs.admin) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.

$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query_range" \
    --data-urlencode 'query=metric_name_0{source_id="source_id_0"}' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)" \
    --data-urlencode 'step=1h'

And this is the corresponding response body:

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {"__name__": "metric_name"},
        "values": [
          [ "<timestamp>", "<value>" ],
          "..."
        ]
      },
      "..."
    ]
  }
}

GET /api/v1/series

Issues a PromQL series query against Metric Store data. You can read more detail in the Prometheus documentation here.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Query Parameters:

  • start is a UNIX timestamp in seconds or RFC3339. (e.g. date -d '24 hours ago' +%s). Start time is inclusive. [start..end)
  • end is a UNIX timestamp in seconds or RFC3339. (e.g. date +%s). End time is exclusive. [start..end)
  • match[] is a series selector.
$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/series" \
    --data-urlencode 'match[]=metric_name_0' \
    --data-urlencode 'match[]=metric_name_1' \
    --data-urlencode "start=$(date -d '24 hours ago' +%s)" \
    --data-urlencode "end=$(date +%s)"

And this is the corresponding response body:

{
  "status": "success",
  "data": [
    {
      "__name__": "metric_name_0",
      "source_id": "source_id_0",
      "...": ".."
    },
    "..."
  ]
}

GET /api/v1/labels

Retrieve all label names for authorized source ids from the store.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Here is an example request:

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/labels"

And this is the corresponding response body:

{
  "status": "success",
  "data": [
    "__name__",
    "deployment",
    "..."
    "ip",
    "origin",
    "request_type",
    "source_id",
    "status_code",
    "unit"
  ]
}

GET /api/v1/label/<label_name>/values

Retrieve label values by label_name for all authorized source ids. The special label __name__ can be used to retrieve metric names from the store.

Non-admins (doppler.firehose or logs.admin) are not permitted to use this query.

Here is an example request:

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/<label-name>/values"

And this is the corresponding response body:

{
    "status":"success",
    "data":["10", "1"]
}

Here is an example request:

$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/__name__/values"

And this is the corresponding response body:

{
    "status":"success",
    "data":["metric_name_0", "metric_name_1", "..." ]
}
check-circle-line exclamation-circle-line close-line
Scroll to top icon