The SEs send metrics data to the Controllers. The Controllers monitor the data and send alarm and notify the customer when the data is up to some thresholds.

Prior to Avi Load Balancer version 30.2.1, Avi Controllers used hard-coded thresholds to generate alerts on CPU/ Memory/ Disk utilization for both Service Engines and Controllers. The ability to adjust these thresholds has been added so that you can configure the threshold value to a custom value.

Configuring Custom Threshold

The following example process will configure a custom threshold for Service Engine CPU utilization. The same process can be applied for Service Engine memory and disk utilization:

The following is the process to configure custom thresholds using the CLI configuration in the Controller shell:

[admin:avi-ctrl]: > configure serviceenginegroup Default-Group
[admin:avi-ctrl]: serviceenginegroup> se_group_analytics_policy
[admin:avi-ctrl]: serviceengine:se_group_analytics_policy > metrics_event_thresholds
New object being created
[admin:avi-ctrl]: serviceenginegroup:se_group_analytics_policy:metrics_event_thresholds> metrics_event_threshold_type se_cpu_threshold
[admin:avi-ctrl]: serviceenginegroup:se_group_analytics_policy:metrics_event_thresholds> reset_threshold 30
[admin:avi-ctrl]: serviceenginegroup:se_group_analytics_policy:metrics_event_thresholds> watermark_thresholds 56
[admin:avi-ctrl]: serviceenginegroup:se_group_analytics_policy:metrics_event_thresholds> save
[admin:avi-ctrl]: serviceenginegroup:se_group_analytics_policy> save
[admin:avi-ctrl]: serviceenginegroup:> save

In the above example, the configuration is performed in the Service Engine group. Under that Service Engine group you configure an se_group_analytics_policy, and below that, a metrics_event_threshold is created. Within this metrics_event_threshold object, the three settings can be configured:

  • metrics_event_threshold_type: The type of threshold being configured. The options are:

    • SE_CPU_THRESHOLD

    • SE_MEM_THRESHOLD

    • SE_DISK_THRESHOLD

    • CONTROLLER_CPU_THRESHOLD

    • CONTROLLER_MEM_THRESHOLD

    • CONTROLLER_DISK_THRESHOLD

  • watermark_threshold: The high water mark for the configured threshold type. When the metric exceeds this percentage, an event will be generated.

  • reset_threshold: The reset (low water) mark for the threshold. A second alert for an event will not be generated until the metric goes below this threshold, resetting the event state.

The following is a graphical representation and description of how the configured threshold will behave when generating an event:



In the above image, X-axis represents the time and Y-axis represents SE CPU usage in percentage.

If you monitor and record the SE CPU usage every five minutes (time interval is five minutes), each time point in the X-axis is the monitored time point.

  1. From time 1 to time 4, the SE CPU usage is lower than the watermark threshold 56%, so no SE_CPU_HIGH event is generated.

  2. At time 5, the SE CPU usage is 60%, which is higher than the watermark threshold, hence the SE_CPU_HIGH event is generated in the Controller UI.

  3. From time 6 to time 8, since one SE_CPU_HIGH event is already generated and the current SE CPU usage is higher than the reset threshold 30%, hence the repeated SE_CPU_HIGH event will not be generated.

  4. At time 9, the SE CPU usage is lower than the reset threshold, the state machine in the background should be reset.

  5. At time 10, the SE CPU usage is lower than the watermark threshold, no SE_CPU_HIGH event is generated.

  6. At time 11, the SE CPU usage is higher than the watermark threshold after it is reset, hence another SE_CPU_HIGH event is generated in the Controller UI.

Resource thresholds for Controllers can also be configured using the same basic workflow, except instead of configuring the Service Engine group, the systemconfiguration object is modified. See the following example for configuring a custom threshold for Controller CPU:

> configure systemconfiguration
	> controller_analytics_policy
		> metrics_event_thresholds
			> metrics_event_threshold_type controller_cpu_threshold
			> reset_threshold 20
			> watermark_thresholds 30
			> save