Learn how to configure and use Metric Store.
Metric Store implements the Prometheus Query Language for querying metrics for which you have access.
When querying the API through HTTPS, each request must have the Authorization
header set with a UAA provided token.
As a Metric Store admin, you have access to all recorded metrics (platform and application) and can run Label and Series queries. You need to add either doppler.firehose
or logs.admin
scope to your admin account.
You then can run the following query:
cf login
curl -vvv -H "Authorization: $(cf oauth-token)" -G https://metric-store.SYSTEM-DOMAIN/api/v1/label/source_id/values
The status code is 200 if everything is working properly.
As a Tanzu Application Service developer, you can query metrics for applications that you have access. When querying with PromQL, you must specify the source_id
label with the application guid
as the label value.
For example:
curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" \
--data-urlencode "query=cpu{source_id=\"$(cf app --guid your-app-name)\"}"
The following table shows the standard Tanzu Application Service application metrics.
Metric Name | Description | PromQL Example |
---|---|---|
cpu | CPU usage percentage | avg(cpu{source_id=“$app_guid”}) |
memory | Memory usage in bytes | avg(memory{source_id=“$app_guid”}) |
memory_quota | Memory Quota allocated in bytes | memory{source_id=“$app_guid”} / memory_quota{source_id=“$app_guid”} |
disk | Disk usage in bytes | avg(disk{source_id=“$app_guid”}) / 1024 / 1024 / 1024 # in gigabytes |
disk_quota | Disk quota allocated in bytes | disk{source_id=“$app_guid”} / disk_quota{source_id=“$app_guid”} |
http_total | HTTP request counts includes status | sum(rate(http_total{source_id=“$app_guid”})) by (status) |
http_duration_seconds_* | HTTP request request latencies bucketed by duration | histogram_quantile(0.95, sum(rate(http_duration_seconds_bucket[5m])) by (le)) |
The ultimate goal of these endpoints is to create a fully-compliant, Prometheus-compatible interface. This allows tools such as Grafana to work directly with Metric Store without any additional translation.
A valid PromQL metric name consists of the character [a-Z][0-9] and underscore. Names can begin with [a-Z] or underscore. Names cannot begin with [0-9]. As a measure to work with existing metrics that do not comply with the previous mentioned format a conversion process takes place when matching on metric names. Any character that is not in the set of valid characters is converted to an underscore. The metric is not changed in the store.
For example, to match on a metric name http.latency
use the name http_latency
as a search term.
/api/v1/query
Issues a PromQL instant query against Metric Store data. You can read more detail in the Prometheus documentation here.
Query Parameters:
date -d '24 hours ago' +%s
).Admins (doppler.firehose
or logs.admin
) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.
$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query" --data-urlencode 'query=metric_name_0{source_id="source_id_0"}'
And this is the corresponding response body:
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {"__name__": "metric_name"},
"value": [ "<timestamp>", "<value>" ]
},
"..."
]
}
}
/api/v1/query_range
Issues a PromQL range query against Metric Store data. You can read more detail in the Prometheus documentation here.
Query Parameters:
date -d '24 hours ago' +%s
). Start time is inclusive. [start..end)
date +%s
). End time is exclusive. [start..end)
duration
format or float number of seconds. Admins (doppler.firehose
or logs.admin
) are permitted to use this query without specifying a source_id in the query parameter. Non-admins cannot use regex matchers on source_id.
$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/query_range" \
--data-urlencode 'query=metric_name_0{source_id="source_id_0"}' \
--data-urlencode "start=$(date -d '24 hours ago' +%s)" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode 'step=1h'
And this is the corresponding response body:
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {"__name__": "metric_name"},
"values": [
[ "<timestamp>", "<value>" ],
"..."
]
},
"..."
]
}
}
/api/v1/series
Issues a PromQL series query against Metric Store data. You can read more detail in the Prometheus documentation here.
Non-admins (doppler.firehose
or logs.admin
) are not permitted to use this query.
Query Parameters:
date -d '24 hours ago' +%s
). Start time is inclusive. [start..end)
date +%s
). End time is exclusive. [start..end)
$ curl -H "Authorization: $(cf oauth-token)" -G "https://metric-store.SYSTEM-DOMAIN/api/v1/series" \
--data-urlencode 'match[]=metric_name_0' \
--data-urlencode 'match[]=metric_name_1' \
--data-urlencode "start=$(date -d '24 hours ago' +%s)" \
--data-urlencode "end=$(date +%s)"
And this is the corresponding response body:
{
"status": "success",
"data": [
{
"__name__": "metric_name_0",
"source_id": "source_id_0",
"...": ".."
},
"..."
]
}
/api/v1/labels
Retrieve all label names for authorized source ids from the store.
Non-admins (doppler.firehose
or logs.admin
) are not permitted to use this query.
Here is an example request:
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/labels"
And this is the corresponding response body:
{
"status": "success",
"data": [
"__name__",
"deployment",
"..."
"ip",
"origin",
"request_type",
"source_id",
"status_code",
"unit"
]
}
/api/v1/label/<label_name>/values
Retrieve label values by label_name
for all authorized source ids. The special label __name__
can be used to retrieve metric names from the store.
Non-admins (doppler.firehose
or logs.admin
) are not permitted to use this query.
Here is an example request:
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/<label-name>/values"
And this is the corresponding response body:
{
"status":"success",
"data":["10", "1"]
}
Here is an example request:
$ curl -H "Authorization: $(cf oauth-token)" "https://metric-store.SYSTEM-DOMAIN/api/v1/label/__name__/values"
And this is the corresponding response body:
{
"status":"success",
"data":["metric_name_0", "metric_name_1", "..." ]
}