Container metrics

This topic tells you about the metrics that are emitted by all containers managed by VMware Tanzu Application Service for VMs (TAS for VMs) and its scheduling system, Diego.

Application metrics include the container metrics, and any custom application metrics that developers create.

Diego Container metrics

Diego containers emit resource usage metrics for the application instance. Diego averages and emits each metric every 15 seconds.

The following table describes all Diego container metrics:

Metric Description Unit

CpuPercentage CPU time used by AI as a percentage of a single CPU core.

This is usually no greater than 100% * the number of vCPUs on the host Diego cell, but it might be more due to discrepancies in measurement timing. float64

absolute_entitlement The amount of CPU time that a Diego Cell has allocated to an app instance, in nanoseconds.

At minimum, the CPU time that a Diego Cell allocates to an app instance is min(app memory, 8 GB) * (Diego cell vCPUs/Diego cell memory) * 100%. The operator of your TAS for VMs deployment can provide the vCPUs/memory ratio of the Diego Cell to developers.
If a Diego Cell is not already working at capacity, or if other workloads on the Diego Cell are idle, the Diego Cell can allocate more than the minimum amount of CPU time to an app instance. float64

absolute_usage The CPU time that an app instance has used, in nanoseconds.

absolute_usage / absolute_entitlement calculates a 0-100% range of app instance usage per entitlement. float64

memory The amount of RAM memory in bytes that an app instance has used. uint64

memory_quota The amount of RAM memory in bytes that is available for an app instance to use. float64

disk The amount of disk space in bytes that an app instance has used. float64

disk_quota The amount of disk space in bytes that is available for an app instance to use. float64

container_age The age in nanoseconds of the Diego container. float64

log_rate The current log rate in bytes per second for an app instance. float64

log_rate_limit The log rate limit in bytes per second for an app instance. float64

Metric	Description	Unit
`CpuPercentage`	CPU time used by AI as a percentage of a single CPU core. This is usually no greater than `100% * the number of vCPUs on the host Diego cell`, but it might be more due to discrepancies in measurement timing.	`float64`
`absolute_entitlement`	The amount of CPU time that a Diego Cell has allocated to an app instance, in nanoseconds. At minimum, the CPU time that a Diego Cell allocates to an app instance is `min(app memory, 8 GB) * (Diego cell vCPUs/Diego cell memory) * 100%`. The operator of your TAS for VMs deployment can provide the `vCPUs/memory` ratio of the Diego Cell to developers. If a Diego Cell is not already working at capacity, or if other workloads on the Diego Cell are idle, the Diego Cell can allocate more than the minimum amount of CPU time to an app instance.	`float64`
`absolute_usage`	The CPU time that an app instance has used, in nanoseconds. `absolute_usage / absolute_entitlement` calculates a 0-100% range of app instance usage per entitlement.	`float64`
`memory`	The amount of RAM memory in bytes that an app instance has used.	`uint64`
`memory_quota`	The amount of RAM memory in bytes that is available for an app instance to use.	`float64`
`disk`	The amount of disk space in bytes that an app instance has used.	`float64`
`disk_quota`	The amount of disk space in bytes that is available for an app instance to use.	`float64`
`container_age`	The age in nanoseconds of the Diego container.	`float64`
`log_rate`	The current log rate in bytes per second for an app instance.	`float64`
`log_rate_limit`	The log rate limit in bytes per second for an app instance.	`float64`

The way that Diego emits container metrics differs depending on the version of Loggregator used in your TAS for VMs deployment:

Loggregator v1: Diego emits most container metrics in a ContainerMetric envelope. Diego emits the absolute_entitlement, absolute_usage, container_age, log_rate, and log_rate_limit container metrics in ValueMetric envelopes.
Loggregator v2: Diego emits all container metrics in gauge envelopes. Diego emits the absolute_entitlement, absolute_usage, container_age, log_rate, and log_rate_limit container metrics in separate gauge envelopes from other container metrics. The container metrics come in three envelopes:
- One envelope containing cpu, disk, disk_quota, memory, and memory_quota
- One envelope containing absolute_entitlement, absolute_usage, and container_age
- One envelope containing log_rate and log_rate_limit

Retrieving Container metrics from the cf CLI

You can retrieve container metrics using the Cloud Foundry Command Line Interface (cf CLI).

To retrieve CPU, memory, and disk metrics for all instances of an application, see Retrieve CPU, Memory, and Disk Metrics.

To retrieve CPU entitlement metrics for all instances of an application, see Retrieve CPU Entitlement Metrics.

To determine when an application has exceeded its CPU entitlement, see Monitor Apps That Exceed Their CPU Entitlement.

Retrieving CPU, Memory, and Disk metrics

To retrieve CPU, memory, and disk metrics for all instances of an application:

In a terminal window, run:
```
cf app APP-NAME
```
Where APP-NAME is the name of the app.

The above command returns CPU, memory, and disk metrics for all instances of the app, similar to the following example:
```
Showing health and status for app dora-example in org o / space s as admin...
```

Label in Output	Metrics listed, as described in Diego Container Metrics
`cpu`	`CpuPercentage`
`memory`	`MemoryBytes` of `MemoryBytesQuota`
`disk`	`DiskBytes` of `DiskBytesQuota`

type:           web
sidecars:
instances:      1/1
memory usage:   1024M
     state     since                  cpu    memory        disk          logging            details
&#35;0   running   2022-09-16T01:38:46Z   0.2%   36.3M of 1G   90.3M of 1G   0/s of unlimited
</pre>

The command shows the percentage of CPU the app is currently using relative to the total CPU on the host machine.

Retrieving CPU Entitlement metrics

The absolute_entitlement metric shows the amount of CPU an app is using relative to its CPU entitlement.

To retrieve absolute_entitlement metrics for all instances of an app:

Install the Cloud Foundry CPU Entitlement Plugin from the Cloud Foundry CPU Entitlement Plugin repository on GitHub.

To see application instance AbsoluteCPUEntitlement metrics from the command line:

```
cf cpu-entitlement APP-NAME
```
Where `APP-NAME` is the name of the app.
<br>
<br>
The above command returns `absolute_entitlement` metrics for all instances of the app, similar to the following example:
<pre class="terminal">
Showing CPU usage against entitlement for app dora-example in org example-org / space example-org-staging as dora@example.com ...

     avg usage   curr usage
&#35;0   1.62%       1.66%
&#35;1   2.93%       3.09%
&#35;2   2.51%       2.62%
</pre>

Determining when Applications exceed their CPU Entitlement

You can use the Cloud Foundry CPU Overentitlement Plug-in to determine when an application has exceeded its CPU entitlement and might need to be scaled up.

When you allow CPU throttling in your TAS for VMs deployment, apps are split into two groups:

Apps with an average CPU usage under 100% of their CPU entitlement
Apps with an average CPU usage that exceeds 100% of their entitlements

To determine which apps in your org have exceeded their CPU entitlements:

Install the Cloud Foundry CPU Entitlement Plugin from the Cloud Foundry CPU Entitlement Plugin repository on GitHub.
In a terminal window, run:
```
cf over-entitlement-instances
```
The above command returns output similar to the following example:
```
Note: This feature is experimental.
Showing over-entitlement apps in org example-org / space example-org-staging as dora@example.com ...

   space                     app
#0   example-org-staging       dora-example-2
#1   example-org-staging       dora-example-3
#2   example-org-staging       dora-example-4
```
The above example output shows that the three listed apps have an average CPU usage that exceeds 100% of their entitlements. When an app's average CPU usage exceeds its CPU entitlement, consider increasing their CPU entitlement to ensure that they do not become throttled.