Analytics Tab

Pool End-to-End Timing

The End-to-End Timing pane at the top of the Analytics tab of the Pool Details page provides a high-level overview of the quality of the end-user experience and where any slowdowns may be occurring. The chart breaks down the time required to complete a single transaction, such as an HTTP request.

It may be helpful to compare the end-to-end time against other metrics, such as throughput, to see how traffic increases impact the ability of the application to respond. For instance, if new connections double but the end-to-end time quadruples, you may need to consider adding more servers.

From left to right, this pane displays the following timing information:


Field	Description
Server RTT	This is Service Engine to server round trip latency. An abnormally high server RTT may indicate either that the network is saturated or more likely that a server’s TCP stack is overwhelmed and cannot quickly establish new connections.
App Response	The time the servers take to respond. This includes the time the server took to generate content, potentially fetch back-end database queries or remote calls to other applications, and begin transferring the response back to Avi Load Balancer. This time is calculated by subtracting the Server RTT from the time of the first byte of a response from the server. If the application consists of multiple tiers (such as web, applications, and database), then the App Response represents the combined time before the server in the pool began responding. This metric is only available for a layer 7 virtual service.

Pool Metrics

The sidebar metrics tiles contain the following metrics for the pool. Clicking any metric tile will change the main chart pane to show the chosen metric.


Field	Description
End to End Timing	Shows the total time from the pool’s End to End Timing graph. To see the complete end-to-end timing, including the client latency, refer to Analytics tab of the Virtual Service Details page, which includes the client to Service Engine metric.
Throughput	Total bandwidth passing between the virtual service and the servers assigned to the pool. This throughput number may be different than the virtual service throughput, which measures throughput between the client and the virtual service. Many features may affect these numbers between the client and server side of Avi Load Balancer, such as caching, compression, SSL, and TCP multiplexing. Hovering your mouse pointer over this graph displays the throughput in Mbps for the selected time period.
Open Connections	The number of open (existing) connections during the selected time period.
Estimated Capacity	The estimated capacity during the seleted time period.
Available Capacity	The available capacity during the seleted time period.
New Connections	The number of client connections that were completed or closed over the selected time period. Refer to this article for an explanation of new versus closed connections per second.
Requests	The number of HTTP requests sent to the servers assigned to the pool. This metric also shows errors sent to servers or returned by servers. Any client requests that received an error generated by Avi Load Balancer as a response (such as a 500 when no servers are available) are not forwarded to the pool and not tracked in this view.
Servers	Displays the number of servers in the pool and their health. The X-axis represents the number of HTTP requests or connections to the server, while the Y-axis represents the health score of the server. The chart enables you to view servers in relation to their peers within the pool, thus helping to spot outliers. Within the chart pane, click and drag the mouse-over server dots to select and display a table of the highlighted servers below the Chart pane. The table provides more details about these servers, such as hostname, IP address, health, new connections or requests, health score, and the server’s static load balanced ratio. Clicking on the name of a server will jump to the pool’s Server Insight page, which shows health and resource status.
CPU	The CPU usage during the seleted time period.
Memory	The memory usage during the seleted time period.

Pool Chart Pane

The main chart pane in the middle of the Analytics tab displays a detailed historical chart of the selected metric tile for the current pool.

Hovering the mouse over any point in the chart will display the results for that selected time in a popup window.
Clicking within the chart will freeze the popup. This may be useful when the chart is scrolling as the display updates over time.
Clicking again will unfreeze the highlighted point in time.

Many charts contain radio buttons in the top right that allow customization of data that must be included or excluded from the chart. For instance, if the End to End Timing chart is heavily skewed by one very large metric, then deselecting that metric by clearing the appropriate radio button will re-factor the chart based on the remaining metrics shown. This may change the value of the vertical Y-axis.

Some charts also contain overlay items, which will appear as color-coded icons along the bottom of the chart.

Pool Overlays Pane

The overlays pane is used to highlight important events within the timeline of the chart pane. This feature helps correlate anomalies, alerts, configuration changes, or system events with changes in traffic patterns.

Within the overlays pane:

Each overlay type displays the number of entries for the selected time period.
Clicking an overlay button toggles that overlay’s icons in the chart pane. The button lists the number of instances (if any) of that event type within the selected time period.
Selecting an overlay button displays the icon for the selected event type along the bottom of the chart pane. Multiple overlay icon types may overlap. Clicking the overlay type’s icon in the chart pane will display more data below the overlay Items bar. The following overlay types are available:
- Anomalies — Display anomalous traffic events, such as a spike in server response time, along with corresponding metrics collected during that time period.
- Alerts — Display alerts, which are filtered system-level events that have been deemed important enough to notify an administrator.
- Config Events — Display configuration events, which track configuration changes made to Avi Load Balancer by either an administrator or an automated process.
- System Events — Display system events, which are raw data points or metrics of interest. System Events can be noisy, and are best used as the basis of alerts which filter and classify raw events by severity.

Pool Anomalies Overlay

The anomalies overlay displays periods during which traffic behavior was considered abnormal based on recent historical moving averages. Changing the time interval will provide greater granularity and potentially show more anomalies.

Clicking Anomalies Overlay button displays yellow anomaly icons in the chart pane. Selecting one of these icons within the chart pane brings up more information in a table at the bottom of the page. During times of anomalous traffic, Avi Load Balancer records any metrics that have deviated from the norm, which may provide hints as to the root cause of the anomaly.

An anomaly is defined as a metric that has a deviation of four sigma or greater across the moving average of the chart.

Anomalies are not recorded or displayed while viewing with the real-time display period.


Field	Description
Timestamp	Displays the date and time when the anomaly was detected. This may either span the full duration of the anomaly, or merely be near the same time window.
Type	Displays the specific metric deviating from the norm during the anomaly period. To be included, the metric deviation must be greater than four sigma. Numerous types of metrics, such as CPU utilization, bandwidth, or disk I/O may trigger anomalous events.
Entity	Displays the name of the specific object that is reporting this metric.
Entity Type	Displays the type of entity that caused the anomaly. This may be one of the following: Virtual Machine (server): These metrics require Avi Load Balancer to be configured for either read or write access to the virtualization orchestrator such as vCenter or OpenStack. In the example shown above, CPU utilization of the two servers was learned by querying vCenter. Virtual service Service Engine
Metric Value	Displays the thumbnail historical graph for the selected metric, including the most current value for the metric which will be data on the far right. Moving the mouse over the chart pane will show the value of the metric for the selected time. Use this to compare the normal, current, and anomaly time periods.
Anomalous Value	Displays the anomalous value.
Prediction Interval	Displays the upper and lower bounds on the estimate of an outcome variable.

Pool Alerts Overlay

The alerts overlay displays the results of any events that meet the filtering criteria defined in the alerts tab. Alerts notify administrators about important information or changes to a site that may require immediate attention.

Alerts may be transitory, meaning that they may expire after a defined period of time. For instance, Avi Load Balancer may generate an alert if a server is down and then allow that alert to expire after a specified time period once the server comes back online. The original event remains available for later troubleshooting purposes.

Clicking the alerts icon in the overlay items bar displays any red alerts icons in the chart pane. Selecting one of these chart alerts will display more information below the overlay Items bar, which will show the following information:


Field	Description
Timestamp	Displays date and time when the alert occurred.
Resource Name	Displays name of the object that is reporting the alert.
Level	Displays the severity of the alert. You can use the priority level to determine whether more notifications are required, such as sending an email to administrators or sending a log to Syslog servers. The level may be one of the following: High — Red Medium — Yellow Low — Blue
Summary	Displays brief description of the event.

Pool Config Events Overlay

The config events overlay displays configuration events, such as changing the Avi Load Balancer configuration by adding, deleting, or modifying a pool, virtual service, or Service Engine, or an object related to the object being inspected. If traffic dropped off at precisely 10:00am, and at that time an administrator made a change to the virtual services security settings, there’s a good chance the cause of the change in traffic was due to the incorrect configuration.

Clicking Config Events icon in the Overlay Items bar displays any blue config event icons in the chart pane. Selecting one of these chart alerts will display more information below the Overlay Items bar, which will show the following information:


Field	Description
Timestamp	Displays date and time when the configuration change occurred.
Resource Name	Displays the name of the object that has been modified.
Resource Type	This event type will always be scoped to configuration event types.
Event Code	There are three event codes: `CONFIG_CREATE` `CONFIG_UPDATE` `CONFIG_DELETE`
User	Displays the user.
Description	Displays brief description of the event.
Expand/Contract	Clicking the plus (+) or minus sign (-) for a configuration event either expands or contracts a sub-table showing more detail about the event. When expanded, this shows a difference comparison of the previous configuration versus the new configuration, as follows: Additions to the configuration, such as adding a health monitor, will be highlighted in green in the new configuration. Removing a setting will be highlighted in red in the previous configuration. Changing an existing setting will be highlighted in yellow in both the previous and new configurations.

Pool System Events Overlay

This overlay displays system events relevant to the current object, such as a server changing status from up to down or the health score of a virtual service changing from 50-100.

Clicking the system events icon in the overlay items bar displays any purple system event icons in the Chart Pane. Select a system event icon in the chart pane to display more information below the overlay items bar.


Field	Description
Timestamp	Displays date and time when the system even occurred.
Resource Name	Displays the name of the object that triggered the event.
Resource Type	Displays the type of the resource. For instance, POOL, POOLSERVER, and so on.
Event Code	Displays high-level definition of the event, such as `VS_Health_Change` or `VS_Up`.
Description	Displays brief description of the system event.
Expand/Contract	Clicking the plus (+) or minus sign (-) for a system event expands or contracts that system event to show more information.