This topic discusses route aggregation configuration for provisioning a large number of virtual services on an NSX Advanced Load Balancer Controller operating in Google Cloud Platform (GCP).
Route Aggregation
Route aggregation is a method used to minimize the number of route entries required in an IP network. Unlike flat routing, in which each routing table contains a unique entry for every route, route aggregation consolidates multiple routes into a single route advertisement.
For NSX Advanced Load Balancer deployed in GCP without Route aggregation, the Controller creates a specific /32 route for each virtual service so that the incoming traffic is rightly forwarded to the specified virtual service. Depending on the network, the number of virtual services can scale up to large numbers and with this the route entries would also increase. The number of route entries supported in GCP is limited (about 2000). To address this limitation, the routes that belong to a specific subnet are aggregated to create a mask-route, so that the number of route entries in GCP is reduced.
The steps to configure this feature is discussed in the following sections.
Using Route Integration with NSX Advanced Load Balancer in GCP
To enable aggregation of GCP routes corresponding to multiple virtual services, the virtual IP (VIP) allocated to virtual services must fall within the same subnet range. In case of auto-allocation, the VIP gets allocated from a subnet specified in the Usable Networks of IPAM configuration.
The following are few examples to demonstrate the route aggregation feature:
For a single Service Engine
a. Without route aggregation:
Virtual service |
Virtual IP |
Route programmed in GCP |
|---|---|---|
vs1 |
10.1.1.1 |
10.1.1.1/32, next hop=SE1 |
vs2 |
10.1.1.2 |
10.1.1.2/32, next hop=SE1 |
vs3 |
10.1.1.3 |
10.1.1.3/32, next hop=SE1 |
vs4 |
10.1.1.4 |
10.1.1.4/32, next hop=SE1 |
Note that there is only a single route entry (10.1.1.0/29) in GCP.
Virtual service |
Virtual IP |
Route programmed in GCP |
|---|---|---|
vs1 |
10.1.1.1 |
10.1.1.0/29, next hop=SE1 |
vs2 |
10.1.1.2 |
|
vs3 |
10.1.1.3 |
|
vs4 |
10.1.1.4 |
Virtual Service scaled out to two Service Engines within a SE Group
Without route aggregation:
Virtual service |
Virtual IP |
Route programmed in GCP |
|---|---|---|
vs1 |
10.1.1.1 |
10.1.1.1/32, next hop=SE1 10.1.1.1/32, next hop=SE2 |
vs2 |
10.1.1.2 |
10.1.1.2/32, next hop=SE1 10.1.1.2/32, next hop=SE2 |
vs3 |
10.1.1.3 |
10.1.1.3/32, next hop=SE1 10.1.1.3/32, next hop=SE2 |
vs4 |
10.1.1.4 |
10.1.1.4/32, next hop=SE1 10.1.1.4/32, next hop=SE2 |
Virtual service |
Virtual IP |
Route programmed in GCP |
|---|---|---|
vs1 |
10.1.1.1 |
10.1.1.0/29, next hop=SE1 10.1.1.0/29, next hop=SE2 |
vs2 |
10.1.1.2 |
|
vs3 |
10.1.1.3 |
|
vs4 |
10.1.1.4 |
Virtual Services scaled out to multiple SE Groups
While configuring a large number of virtual services, the limit of virtual services per SE might hit the maximum, based on the size of the Service Engine. To overcome this, virtual services will need to be distributed across multiple SE groups, so that the groups of virtual services within each SE group can be aggregated through route aggregation.
On enabling route aggregation and configuring service_ip_subnet for individual SE groups, the Controller automatically places the virtual service to one of the SE groups, while the VIP for the virtual services gets allocated from within the service_ip_subnet configured on that SE group.
Consider an instance with eight virtual services. Based on the expected throughput and the size of the selected SE VM, the maximum number of virtual services per SE needs to be restricted to two. To configure eight virtual services, four SE groups must be created. Each SE group programs two aggregated routes (two SEs in each SE group). So, four subnets need to be configured. This needs to be carved out from the larger IPAM subnet.
The following are the calculation details:
Configured GCP IPAM subnet = 10.1.1.0/24
Configured maximum number of virtual service per SE = 2
Maximum number of virtual services expected = 8
Number of SE groups required = 8/2 = 4
Number of
service_ip_subnetsto be configured = 4
The IPAM subnet 10.1.1.0/24 needs to be divided into four subnets, and each must be assigned to one SE group as follows:
Virtual service |
SE group |
|
Virtual IP |
Route programmed in GCP |
|---|---|---|---|---|
vs1 |
SE Group 1 |
10.1.1.0/26 |
10.1.1.1 |
10.1.1.0/26, next hop = SE1 10.1.1.0/26, next hop = SE2 |
vs2 |
SE Group 1 |
10.1.1.0/26 |
10.1.1.2 |
|
vs3 |
SE Group 2 |
10.1.1.64/26 |
10.1.1.65 |
10.1.1.64/26, next hop = SE3 10.1.1.64/26, next hop = SE4 |
vs4 |
SE Group 2 |
10.1.1.64/26 |
10.1.1.66 |
|
vs5 |
SE Group 3 |
10.1.1.128/26 |
10.1.1.129 |
10.1.1.128/26, next hop = SE5 10.1.1.128/26, next hop = SE6 |
vs6 |
SE Group 3 |
10.1.1.128/26 |
10.1.1.130 |
|
vs7 |
SE Group 4 |
10.1.1.192/26 |
10.1.1.193 |
10.1.1.192/26, next hop = SE7 10.1.1.192/26, next hop = SE8 |
vs8 |
SE Group 4 |
10.1.1.192/26 |
10.1.1.194 |
Configuration Steps
Prerequisite: The NSX Advanced Load Balancer Controller, cloud, IPAM, and networks must be set up.
For various network configuration options available with GCP cloud, see GCP Network Configuration.
To update the GCP Cloud to enable Route Aggregation:
Step 1: Configure the Cloud.
Enable the match_se_group_subnet field in vip_allocation_strategy field of the GCP Cloud Object. This field is currently not available in the NSX Advanced Load Balancer UI and can be configured only through the CLI.
[admin:10-138-10-68]: > configure cloud gcp-cloud-subnetroutes-inband [admin:10-138-10-68]: cloud> gcp_configuration [admin:10-138-10-68]: cloud:gcp_configuration> vip_allocation_strategy [admin:10-138-10-68]: cloud:gcp_configuration:vip_allocation_strategy> routes [admin:10-138-10-68]: cloud:gcp_configuration:vip_allocation_strategy:routes> match_se_group_subnet Overwriting the previously entered value for match_se_group_subnet [admin:10-138-10-68]: cloud:gcp_configuration:vip_allocation_strategy:routes> save [admin:10-138-10-68]: cloud:gcp_configuration:vip_allocation_strategy> save [admin:10-138-10-68]: cloud:gcp_configuration> save [admin:10-138-10-68]: cloud> save +------------------------------+--------------------------------------------+ | Field | Value | +------------------------------+--------------------------------------------+ | uuid | cloud-50b7a230-d378-4f6f-9700-3a068bfbce94 | | name | gcp-cloud-subnetroutes-inband | | vtype | CLOUD_GCP | | apic_mode | False | | gcp_configuration | | | cloud_credentials_ref | avi-service-account | | region_name | us-central1 | | zones[1] | us-central1-a | | zones[2] | us-central1-b | | se_project_id | se-project-id | | network_config | | | config | INBAND_MANAGEMENT | | inband | | | vpc_subnet_name | subnet-1 | | vpc_project_id | network-project | | vpc_network_name | dev-vnet-1 | | firewall_target_tags[1] | http-server | | firewall_target_tags[2] | https-server | | vip_allocation_strategy | | | mode | ROUTES | | routes | | | match_se_group_subnet | True | | dhcp_enabled | True | | mtu | 1500 bytes | | prefer_static_routes | False | | enable_vip_static_routes | False | | license_type | LIC_CORES | | ipam_provider_ref | ipam-gcp-cloud-subnetroutes-inband | | state_based_dns_registration | True | | ip6_autocfg_enabled | False | | dns_resolution_on_se | False | | enable_vip_on_all_interfaces | False | | tenant_ref | admin | | license_tier | ENTERPRISE | | autoscale_polling_interval | 60 seconds | +------------------------------+--------------------------------------------+
admin:10-138-10-68]: > show ipamdnsproviderprofile ipam-gcp-cloud-subnetroutes-inband +--------------------------+-------------------------------------------------------------+ | Field | Value | +--------------------------+-------------------------------------------------------------+ | uuid | ipamdnsproviderprofile-775b580b-f4c1-4edb-93f6-e13a85eb7c68 | | name | ipam-gcp-cloud-subnetroutes-inband | | type | IPAMDNS_TYPE_INTERNAL | | internal_profile | | | ttl | 30 sec | | usable_network_refs[1] | net-gcp-cloud-subnetroutes-inband | | allocate_ip_in_vrf | False | | tenant_ref | admin | +--------------------------+-------------------------------------------------------------+
[admin:10-138-10-68]: > show network net-gcp-cloud-subnetroutes-inband +----------------------------+----------------------------------------------+ | Field | Value | +----------------------------+----------------------------------------------+ | uuid | network-09e899e3-a61f-4147-9575-5a6e9c508620 | | name | net-gcp-cloud-subnetroutes-inband | | vcenter_dvs | True | | dhcp_enabled | True | | exclude_discovered_subnets | False | | configured_subnets[1] | | | prefix | 10.10.0.0/24 | | static_ranges[1] | | | begin | 10.10.0.2 | | end | 10.10.0.254 | | vrf_context_ref | global | | synced_from_se | False | | ip6_autocfg_enabled | True | | tenant_ref | admin | | cloud_ref | gcp-cloud-subnetroutes-inband | +----------------------------+----------------------------------------------+
Step 2: Plan the number of SEs and SE groups required.
The minimum number of SE groups required can be calculated as follows:
Estimate the total number of virtual services to be provisioned (represented by V).
Assume M to be the maximum number of virtual services a single SE can host. For more information, see NSX Advanced Load Balancer Controller Sizing.
The total number of SE groups is V/M (let it be represented by N).
Divide the IPAM network into N smaller subnets, such that each subnet has a maximum of M IP addresses. The Controller uses these subnets to program the routes in GCP, instead of having individual /32 routes corresponding to each VIP and SE(s) on which the VIP is placed.
The following are a few examples for the above mentioned calculation:
Example 1:
The total number of virtual services to be provisioned is 256.
The maximum number of virtual services that can be hosted on a single SE is 64.
So, the number of SE groups needing to be created is 256/64 = 4.
The IPAM network created for virtual service is 10.10.0.0/24, which has 254 IP addresses.
Divide the IPAM network into four smaller subnets of 64 IP addresses each: 10.10.0.0/26, 10.10.0.64/26, 10.10.0.128/26 and 10.10.0.192/26. These are the SE Group service subnets.
Example 2:
The total number of virtual services to be provisioned is 512.
The maximum number of virtual services that can be hosted on a single SE is 32.
So, the number of SE groups needing to be created is 512/32 = 16.
The IPAM network created for virtual service is 10.10.0.0/23, which has 510 IP addresses.
Divide the IPAM network into 16 smaller subnets of 32 IP addresses each: 10.10.0.0/27, 10.10.0.32/27, 10.10.0.64/27 till 10.10.1.224/27. These are the SE Group service subnets.
Step 3: Set up SE groups.
Active/ Active and Active/ Standby HA modes are supported with Route Aggregation mode.
Configure service-IP subnets for each SE group. Service-IP subnets provide IP addresses to the respective virtual services in their respective SE group. To enable this, configure a service-IP subnet object in the SE group configuration.
The service-IP subnet must be set to smaller subnets, calculated as shown in Step 2 (4).
This field can only be set through the CLI.
Once the field service-IP subnet is used, virtual service IPs are allocated only from the specified SE group.
If the subnet is not set, you must program a /32 route for the virtual service IPs.
Configure min_scaleout_per_vs and max_scaleout_per_vs values to the number of SEs in the SE group as determined in Step 2 (example 1, 2nd point)).
Configure max_vs_per_se value to the maximum number of virtual services that a single SE can host (64 in this case) as determined in Step 2 (example 1, 2nd point)).
[admin:10-138-10-68]: > configure serviceenginegroup gcp-route-aggregation [admin:10-138-10-68]: serviceenginegroup> ha_mode ha_mode_shared_pair [admin:10-138-10-68]: serviceenginegroup> max_vs_per_se 64 [admin:10-138-10-68]: serviceenginegroup> min_scaleout_per_vs 2 [admin:10-138-10-68]: serviceenginegroup> max_scaleout_per_vs 2 [admin:10-138-10-68]: serviceenginegroup> service_ip_subnets 10.10.0.0/26 [admin:10-138-10-68]: serviceenginegroup> save +---------------------------------------+---------------------------------------------------------+ | Field | Value | +---------------------------------------+---------------------------------------------------------+ | uuid | serviceenginegroup-fbd33752-996d-48aa-9534-08372ef10095 | | name | gcp-route-aggregation | | max_vs_per_se | 64 | | min_scaleout_per_vs | 2 | | max_scaleout_per_vs | 2 | ……… | ha_mode | HA_MODE_SHARED_PAIR | ……… | service_ip_subnets[1] | 10.10.0.0/27 | ……… +---------------------------------------+---------------------------------------------------------+
All SE groups in a cloud must have similar configuration. Only the service-IP subnets are different for each SE group.
Step 4: Create a virtual service.
Create a virtual service using auto-allocation. Note that all virtual services must use auto-allocation with this feature. The SE group is automatically selected by the Controller to distribute virtual services across different SE groups. Any manual SE-group selection is overridden by the IPAM, based on the auto-allocated VIP. For more information on virtual service placement, see Elastic High Availability for NSX Advanced Load Balancer Service Engines section in VMware NSX Advanced Load BalancerConfiguration Guide.
With Route Aggregation, only one service subnet Route is created in GCP for one SE in a SE-Group.
Virtual Service Scale out
Virtual Services can be scaled out to a new Service Engine by increasing the min_scaleout_per_vs and max_scaleout_per_vs values in the Service Engine Group Configuration. This scales out all the virtual services in the Service Engine Group to the new SE count.
There might be some traffic disruption if a large number of SEs are present in the SE group. This is due to the fact that route in GCP is created with the first VS Scale-out itself but the SE might not start listening for the remaining Virtual Services.
Steps to scale-out the VSs on an SE-Group:
Increase the min_scaleout_per_vs, max_scaleout_per_vs and max_se in the SE group configuration to the number of required SEs.
[admin:10-138-10-68]: > configure serviceenginegroup gcp-route-aggregation [admin:10-138-10-68]: serviceenginegroup> min_scaleout_per_vs 4 Overwriting the previously entered value for min_scaleout_per_vs [admin:10-138-10-68]: serviceenginegroup> max_scaleout_per_vs 4 Overwriting the previously entered value for max_scaleout_per_vs [admin:10-138-10-68]: serviceenginegroup> max_se 4 Overwriting the previously entered value for max_se [admin:10-138-10-68]: serviceenginegroup> save +-----------------------------------------+---------------------------------------------------------+ | Field | Value | +-----------------------------------------+---------------------------------------------------------+ | uuid | serviceenginegroup-fbd33752-996d-48aa-9534-08372ef10095 | | name | gcp-route-aggregation | | max_vs_per_se | 64 | | min_scaleout_per_vs | 4 | | max_scaleout_per_vs | 4 | | max_se | 4 | ……… +---------------------------------------+-----------------------------------------------------------+
Controller creates new Service Engines in the Service Engine Group and will scaleout all the the Virtual Service in the Service Engine Group to new SEs.
Troubleshooting
Scenario 1: Selecting a different IPAM profile.
As explained in Step 3 (4), the service_ip_subnet field must be configured for an SE group that matches the respective IPAM profile. If a different IPAM profile is chosen, the following error is thrown:
Scenario 2: Static IP address pool not configured.
You can fix this by choosing a static range, as displayed in earlier examples.
