This section discusses the resolution when the auto-rebalance feature does not work as expected for Service Engines on Avi Load Balancer. Due to this, when the load on Service Engines goes above or below the defined threshold values, virtual services do not scale in or scale out to other Service Engines.
Resolution
The auto-rebalance option applies only to the elastic HA modes and is off by default on Avi Load Balancer. If selected, virtual services will be automatically migrated or scaled out / scaled in when the load on the SEs goes above the maximum threshold or falls below the minimum threshold. Avi Load Balancer Controller has rebalance.log in the directory /var/lib/avi/log. Auto-rebalance related event messages are available in the rebalance.log file.
The following error messages are logged in rebalance.log file for a successful scale-out event.
[2017-09-14 20:00:18,539] DEBUG [seautoscale.autoscale:481] as: se_vs_cpu[avi-se-hq][ssl-vs-3#1]=100 (This depicts the SE is at 100%)
[2017-09-14 20:00:18,553] DEBUG [resource_monitor.get_scaleout_candidates:864] vs[ssl-vs-3] 1 candidate-se-host-list: rpc_status: 0
error_message: “No SE found” (This indicates that the SE is trying to scale out by trying to find out if there are any available SEs).
[2017-09-14 20:00:19,554] DEBUG [seautoscale.create_new_se:246] as: scale-out from high to new se: avi_se_hq to NEW-SE with 1 cpu(s), vs = ssl-vs-3#1
The above mentioned error indicates that the virtual services have scaled out from SE having a high CPU (avi_se_hq) to the new SE having one CPU.
The following error messages are observed when auto-rebalance does not initiate a scale-out or a scale-in event.
[2017-09-13 16:44:00,976] DEBUG [resource_monitor.SeGroupRebalanceRpc:3250] [43837][] call later(1) [2017-09-13 16:44:01,977] DEBUG [resource_monitor.DeferSeGroupRebalance:3255] [43837][] Defer to Thread Pool [2017-09-13 16:44:01,978] DEBUG [resource_monitor.SeGroupRebalance:3266] [43837,PoolThread-worker-1][] START SEGROUP REBALANCE [2017-09-13 16:44:02,937] ERROR [resource_monitor.rebalance_task_err_callback:3245] [43837,MainThread] Error in rebalance task:Traceback (most recent call last):
Contact VMware Customer Support to troubleshoot the issue further or if similar error messages are observed in rebalance.log file.
To understand how auto-rebalance feature works for an Avi Load Balancer Service Engine group, consider the following configuration:
[admin:Controller]: > show serviceenginegroup rebalance-segroup +---------------------------------------+---------------------------------------------------------+ | Field | Value | +---------------------------------------+---------------------------------------------------------+ | uuid | serviceenginegroup-2896bcb1-6e46-4213-b789-d83367cacda6 | | name | rebalance-segroup | | max_vs_per_se | 10 | | min_scaleout_per_vs | 1 | | max_scaleout_per_vs | 4 | | max_se | 10 | | vcpus_per_se | 1 | | memory_per_se | 2048 | | disk_per_se | 10 gb | | max_cpu_usage | 50 percent | | min_cpu_usage | 20 percent | | | | | se_deprovision_delay | 120 min | | auto_rebalance | True | | auto_rebalance_interval | 10 sec |
The maximum CPU threshold is set to 50 percent for a scale-out operation or to set off the auto-rebalance feature. Auto-rebalance interval is set at 10 seconds: If the CPU of SEs for the virtual service goes beyond 50% for 10 seconds continuously, a scale-out event will start.
In the below screenshot, one of the SE for a virtual service is at 100% CPU for more than 10 seconds.
As the CPU of the SEs goes beyond the configured threshold level, an auto-rebalance event is triggered, and a new SE is created as a result of a scale-out operation.
The following Avi Load Balancer CLI output exhibits a successful scale-out event. Please note the state: SCALEOUT_SUCCESS as shown in the below output.
vs_scaleout_details se_assigned 0 standby: false connected: true name: avi-se-hooeb primary: true 1 standby: false connected: true name: avi-se-ktrez primary: false se_requested num_se: 2 num_standby_se: 0 vs_uuid: virtualservice-b70c8da3-2d14-47d6-a867-c15f1d9cb98c scale_status end_time_str: 2017-0914 20:00:02.661759 start_time_str: 2017-0914 19:59:57.901848 scale_se: avi-se-ktrez state: SCALEOUT_SUCCESS num_se_assigned: 2 action: SCALE_OUT num_se_requested: 2 ip: 10.1.1.1
The auto-rebalance option is removed from Avi Load Balancer UI, but it can be configured from Avi Load Balancer CLI. For more information, see Auto-Rebalance options not Available Using NSX Advanced Load Balancer UI topic in the VMware Avi Load BalancerConfiguration Guide.
To configure auto-rebalance using CLI, see How to Configure Auto-rebalance using NSX Advanced Load Balancer CLI topic in the VMware Avi Load BalancerConfiguration Guide.
