GSLB service is the representation of a global application deployed at multiple sites. The GSLB service configuration defines the FQDN of the application, the backing of virtual services across multiple sites, and the priority or ratios that govern the selection of a particular virtual service at any given time. The configuration also defines the health-monitoring methods by which unhealthy components can be identified so that the best alternatives may be selected.
GSLB Service Health Monitoring
GSLB service health monitoring is of two categories:
Control Plane
Data Plane
Default Behavior
HM Proxy
HM Sharding
Control-Plane-Based Global Application Health Monitoring
Independent of the Avi Load Balancer GSLB, every Controller cluster routinely performs local health checks to collect the health scores and performance metrics of virtual services under its direct control.
Control-plane health monitor is used to assess the health of Avi member services by collecting virtual service health status from their local Avi Controllers. This option is irrelevant to external virtual services, whose health can only be assessed via data path health checks.
The knob to turn on/off the control-plane health monitoring has been deprecated. The control-plane health monitoring will always be present irrespective of the knob.
The control-plane health monitor will always consider the 3rd party GSLB site members as UP.
Status and attributes related to the control-plane health monitor status can be checked using the show virtualservice <virtual service name> gslbservicehmonstat command through Avi Load Balancer CLI.
System-GSLB-Health-Monitor is the CLI field corresponding the control-plane health monitor status.
Below is an example exhibiting System-GSLB-Health-Monitor configured for a GSLB virtual service.
[10-1-1-1]: > show virtualservice gslb-service gslbservicehmonstat filter gs_ref demoavi_global_vs +-------------------------------------+----------------------------------------------------------------------------------+ | Field | Value | | | +-------------------------------------+----------------------------------------------------------------------------------+ | se_uuid | se-005056814352 | | | | uuid | gslbservice-1b033141-4100-46b5-875d-e11bfaf6c3d7 | | name | demoavi_global_vs | | | hm_off | False | | | | groups[1] | | | | | name | avidemo-gs-pool | | | | hmon[1] | | | | last_transition_timestamp_3 | Fri Oct 20 04:11:35 2023 ms(482662) UTC | | last_transition_timestamp_2 | Fri Oct 20 04:11:35 2023 ms(377018) UTC | | last_transition_timestamp_1 | Fri Oct 20 04:11:35 2023 ms(366519) UTC | | server_hm_stat[1] | | | | | server_name | 10.1.1.1:0 | | | oper_status | | | | | state | OPER_UP | | | | last_transition_timestamp_3 | Mon Feb 26 22:22:32 2024 ms(438810) UTC | | last_transition_timestamp_2 | Mon Feb 26 22:12:56 2024 ms(608106) UTC | | last_transition_timestamp_1 | Sat Feb 17 00:09:01 2024 ms(156616) UTC | | shm_runtime[1] | | | | health_monitor_name | System-GSLB-Health-Monitor | | | health_monitor_type | HEALTH_MONITOR_GSLB | | | last_transition_timestamp_3 | Fri Jan 12 06:04:40 2024 ms(794000) UTC | | state | 1 | | | | rise_count | 255 | | | | fall_count | 0 | | total_checks | 1071709 | | total_failed_checks | 0 | | hm_initial | 0 | | avg_response_time | 0 | | min_response_time | 9999000 | | max_response_time | 0 | | port | 0 | | curr_failed_checks | 0 | | ip_addr | 10.79.238.11 | | port | 0 | | server_hm_stat[2] | | | server_name | 10.79.186.141:0 | | oper_status | | | state | OPER_UP
The control-plane health monitor does periodic health sync-ups with the rest of the sites. This periodic sync-up is controlled by a config called send_interval. This field is configurable through the API and CLI to change the interval.
The send_interval is part of the GSLB configuration. Use the Show gslb <global GSLB object name > command to check value set for the send_interval.
[10-1-1-1]: > show gslb Default +--------------------------+-----------------------------------------------------+ | Field | Value | +--------------------------+-----------------------------------------------------+ | uuid | gslb-9ae46ed6-7b82-4f84-8be8-35d8e6faddfd | | name | Default | | dns_configs[1] | | | domain_name | avi.com | | sites[1] | | | cluster_uuid | cluster-b5769fcd-b04e-4eee-b95a-ebed59331bb3 | | name | glb | | ip_addresses[1] | 10.80.34.73 | | port | 443 | | username | admin | | password | <sensitive> | | member_type | GSLB_ACTIVE_MEMBER | | enabled | True | | dns_vses[1] | | | dns_vs_uuid | virtualservice-6774096b-da81-43ed-bd4d-5720a914caba | | hm_shard_enabled | False | | suspend_mode | False | | leader_cluster_uuid | cluster-b5769fcd-b04e-4eee-b95a-ebed59331bb3 | | send_interval | 15 sec | | clear_on_max_retries | 20 | | view_id | 0 | | client_ip_addr_group | | | type | GSLB_IP_PUBLIC | | async_interval | 0 sec | | error_resync_interval | 300 sec | | replication_policy | | | replication_mode | REPLICATION_MODE_CONTINUOUS | | maintenance_mode | False | | is_federated | True | | tenant_ref | admin | | tenant_scoped | True | | enable_config_by_members | False | +--------------------------+-----------------------------------------------------+ [admin:naveen.dhillon-controller-1-site-a]: >
The following image shows only the active Controller in DC1 collecting health information from 3 other Controllers. DC1’s Controller passes a coalesced picture of health status to its local DNS (solid arrow). In reality, the active Controllers in DC2 and AWS update their respective local DNS virtual service with control-plane-based health status.
Data-Plane-Based Global Application Health Monitoring
This is the default behavior of data-plane-based health monitoring. In contrast to control-plane-based health monitoring, no site’s Controller cluster is queried. Instead, health checks go directly to participating services, that is, to the data plane. At an active site, an SE hosting a GSLB DNS virtual service performs periodic health checks against all GSLB pool members (including the virtual services local to it). Active monitors generate synthetic traffic from the DNS SE to mark a GSLB pool member up or down, based on its response.
The following diagram shows the DNS in DC1 (the only active site) performing this function against its local virtual service (VS-A1),and VS-A2, VS-A3, and VS-A4. Similarly, if Avi Load Balancer DNS virtual service is running on other sites then those DNS SEs would perform the data plane health monitoring in the same way.
Ping, TCP, UDP, DNS, and HTTP(S) health monitors are supported as mentioned in the Health Monitor Types section in the VMware Avi Load Balancer Configuration Guide. Also, a custom monitor can be configured as per the requirement.
It is recommended to configure different SE groups for GSLB DNS and load balancing virtual services.
For more information on health monitors, see Health Monitoring in the VMware Avi Load Balancer Configuration Guide.
Options and Combinations for GSLB Service Health Monitoring
Control plane health checking only:
Active data-plane health monitors are not configured for this mode. All active Controllers are configured to coalesce health status collected locally with the statistics collected from remote Controllers.
Coalesced stats are then passed from each active Controller (cluster) to its local DNS. This method is only available for members implemented as Avi Load Balancer virtual services.
Both control plane and data plane health checking:
In the case of data-plane health monitoring, each GSLB DNS performs health checks on all GSLB member virtual services (including those hosted on external sites).
When data plane health checking is enabled, for a member virtual service to be marked UP, both control and data health must report UP. If the control-plane health check is failing due to the remote site being unreachable, in this case, only the data plane health monitors will determine the state of the GSLB service member.
An important point of consideration is that the default control-plane health monitor is always set by default. The control-plane health monitor does periodic health sync-ups with the rest of the sites. This periodic sync-up is controlled by a config called send_interval. Therefore, the total number of health monitors(hm) will always be number of health monitors configured + 1 (control plane default health monitor) or hm + 1.
The multiple data plane health monitors can be added if required. We can chose when a pool member should be UP - when all the health monitors are UP or when some of the health monitors are UP. This behavior can be controlled by the the configuration knob Min. Health Monitors to Consider Server 'UP.
To understand the behavior in various failure scenarios, Failure Scenarios and Resolutions.
In addition, for data-plane health monitors to work, all active sites must be able to probe all virtual services at every site participating in a GSLB service and configured to be probed (a GSLB service can be configured to probe all members or just non-Avi Load Balancer members). (In case firewall rules cannot be modified to achieve this, see Health Monitor Sharding).
For more information, see:
DNS Health Monitor in the VMware Avi Load Balancer Configuration Guide
