This section explains the TSO, GRO, and RSS configuration process.
Enabling TSO and GRO on an NSX Advanced Load Balancer SE
The TSO feature is enabled by default on an NSX Advanced Load Balancer SE while the GRO feature is deactivated by default.
Upgrading from a prior version will carry forward the GRO configuration for an SE group. However, if an SE group is newly created in NSX Advanced Load Balancer version 22.1.2, GRO configuration is in auto-mode. If the SE group has SEs with vCPUs greater than or equal to eight, GRO will be enabled.
Enabling TSO-GRO is non-disruptive and does not require an SE restart.
If a PUT API call is made to update existing ServiceEngineGroup properties with no explicit value set for GRO configuration knob (
"disable_gro": true); then GRO will be reset/ enabled by default starting with NSX Advanced Load Balancer version 22.1.1. This behavior is not noticed when SE group properties is updated through CLI.
The following are the steps to enable TSO and GRO on SE:
Login to the CLI and use the configure serviceenginegroup command to enable TSO and GRO features.
[admin:cntrl]: > configure serviceenginegroup Default-Group Updating an existing object. Currently, the object is: | disable_gro | True | | disable_tso | True | [admin:cntrl]: serviceenginegroup> no disable_gro Overwriting the previously entered value for disable_gro [admin:cntrl]: serviceenginegroup> no disable_tso Overwriting the previously entered value for disable_tso [admin:cntrl]: serviceenginegroup> save | disable_gro | False | | disable_tso | False |
To verify if the features are correctly turned ON in the SE, you can check the following statistics on the Controller CLI:
GRO statistics are part of interface statistics. For GRO, check the statistics for the following parameters:
gro_mbufs_coalesced
TSO statistics are part of
mbufstatistics. For TSO, check the statistics for the following parameters:num_tso_bytes
num_tso_chain
Execute the show serviceengine <interface IP address> interface command and filter the output using the
grepcommand shown as follows:[admin:cntrl]: > show serviceengine 10.1.1.1 interface | grep gro | gro_mbufs_coalesced | 1157967 | | gro_mbufs_coalesced | 1157967 |
Note:The sample output mentioned above is for 1-queue (No RSS).
See the following output for RSS-enabled, a 4-queue RSS:
Note:In the case of a port-channel interface, provide the relevant physical interface name as the filter in the intfname option. For reference, see the output mentioned below for the Ethernet 4 interface.
show serviceengine 10.1.1.1 interface filter intfname eth4 | grep gro | gro_mbufs_coalesced | 320370 | | gro_mbufs_coalesced | 283307 | | gro_mbufs_coalesced | 343143 | | gro_mbufs_coalesced | 217442 | | gro_mbufs_coalesced | 1164262 |
Note:The statistics for an NIC is the sum of the statistics for each queue for the specific interface.
[admin:cntrl]: > show serviceengine 10.1.1.1 mbufstats | grep tso | num_tso_bytes | 4262518516 | | num_tso_chains | 959426 |
If the features are enabled, the statistics in the output mentioned above will reflect non-zero values for TSO parameters.
Enabling RSS on an NSX Advanced Load Balancer SE
The distribute_queues knob in the SE-group properties enables and disables the RSS. Login to the CLI, and use the distribute_queues command to enable the RSS feature.
Any change in the distribute_queues parameters requires an SE restart.
| distribute_queues | False | [admin:cntrl]: serviceenginegroup> distribute_queues Overwriting the previously entered value for distribute_queues [admin:cntrl]: serviceenginegroup> save | distribute_queues | True |
When RSS is turned ON, all the NICs in the SE configure and use an optimum number of queue pairs as calculated by the SE. The calculation of this optimum number is described in the section on configurable dispatchers.
For instance, the output of a four-queue, RSS-supported interface is as follows:
[admin:cntrl]: > show serviceengine 10.1.1.1 interface filter intfname bond1 | grep ifq | ifq_stats[1] | | ifq_stats[2] | | ifq_stats[3] | | ifq_stats[4] |
The value of counters for ipackets (input packets) and opackets (output packets) per interface queue is a non-zero value as shown below:
[admin:cntrl]: > show serviceengine 10.1.1.1 interface filter intfname bond1 | grep pack | ipackets | 40424864 | | opackets | 42002516 | | ipackets | 10108559 | | opackets | 11017612 | | ipackets | 10191660 | | opackets | 10503881 | | ipackets | 9873611 | | opackets | 10272103 | | ipackets | 10251034 | | opackets | 10208920 |
The output includes statistics of each queue and one combined statistics overall for the NIC.
Configuration Samples
The example mentioned below exhibits the configuration on a bare-metal machine with 24 vCPUs, two 10G NICs, and one bonded of two 10G NICs, and distribute_queues enabled.
Set the value of the configure
num_dispatcher_coresparameter to 8.[admin:cntrl]: serviceenginegroup> num_dispatcher_cores 8 Overwriting the previously entered value for num_dispatcher_cores [admin-ctrlr]: serviceenginegroup> save
[admin:cntrl]:> show serviceengine 10.1.1.1 seagent | grep -E "dispatcher|queues" |num_dispatcher_cpu | 8 |num_queues | 8
Set the value of the configure num_dispatcher_cores parameter to zero (the default value). After restarting the SE, though the configured value for dispatchers is set to zero, the number of queues and the number of dispatchers is changed to four, as shown in the following output:
[admin:cntrl]:> show serviceengine 10.1.1.1 seagent | grep -E "dispatcher|queues" |num_dispatcher_cpu | 4 |num_queues | 4
Configuring Maximum Queues per vNIC
The max_queues_per_vnic parameter supports the following values:
- Zero (Reserved):
-
Auto (deduces optimal number of queues per dispatcher based on the NIC and operating environment)
- One (Reserved):
-
One queue per NIC (Default)
- Integer Value:
-
Power of 2. The maximum limit is 16.
The max_queues_per_vnic parameter deprecates the distribute_queues parameter, which is used to enable the RSS mode of operation, where the number of queues is equal to the number of dispatchers.
The migration routine ensures that the max_queues_per_vnic parameter is set to num_dispatcher_cores if the distribute_queues is enabled, else max_queues_per_vnic will be set to one.
You can use the following command to configure max_queues_per_vnic:
[admin:admin-controller-1]: serviceenginegroup> max_queues_per_vnic INTEGER 0,1,2,4,8,16 Maximum number of queues per vnic Setting to '0' utilises all queues that are distributed across dispatcher cores. [admin:admin-controller-1]: > configure serviceenginegroup Default-Group Updating an existing object. Currently, the object is: +-----------------------------------------+----------------------------+ | Field | Value | +-----------------------------------------+----------------------------+ [output truncated] | se_rum_sampling_nav_percent | 1 | | se_rum_sampling_res_percent | 100 | | se_rum_sampling_nav_interval | 1 sec | | se_rum_sampling_res_interval | 2 sec | | se_kni_burst_factor | 2 | | max_queues_per_vnic | 1 | | core_shm_app_learning | False | | core_shm_app_cache | False | | pcap_tx_mode | PCAP_TX_AUTO | +-----------------------------------------+----------------------------+ [admin:admin-controller-1]: serviceenginegroup> max_queues_per_vnic 2 Overwriting the previously entered value for max_queues_per_vnic [admin:admin-controller-1]: serviceenginegroup> save
The show serviceegine [se] seagent displays the number of queues per dispatcher and the total number of queues per interface.
show serviceengine [se] seagent | num_dp_heartbeat_miss | 0 | | se_registration_count | 2 | | se_registration_fail_count | 0 | | num_dispatcher_cpu | 1 | | ------------------------- truncated output----------------------------| | num_flow_cpu | 1 | | num_queues | 1 | | num_queues_per_dispatcher | 1 |
Configuring Hybrid RSS
The hybrid_rss_mode is an SE-Group configurable property. It toggles SE hybrid only mode of operation in DPDK mode with RSS configured where, each SE data path instance operates as an independent standalone hybrid instance performing both dispatcher and proxy function. This requires a reboot.
The following is the configuration command:
configure serviceenginegroup <se-group> > hybrid_rss_mode [admin:10-102-66-36]: serviceenginegroup> max_queues_per_vnic 0 Overwriting the previously entered value for max_queues_per_vnic [admin:10-102-66-36]: serviceenginegroup>> hybrid_rss_mode Overwriting the previously entered value for hybrid_rss_mode [admin:10-102-66-36]: serviceenginegroup> save
The hybrid_rss_mode is protected by a must check that requires RSS to be enabled before toggling this property to True.
The property also shows up as per SE. The following is the configuration command:
----------------------------------------+ | num_dispatcher_cpu | 4 | | num_flow_cpu | 4 | | num_queues | 4 | | num_queues_per_dispatcher | 1 | | hybrid_rss_mode | True | +---------------------------------------+
Configuring Auto RSS for Public Clouds
This features enables you to configure RSS for the NSX Advanced Load Balancer SEs deployed on public clouds such as AWS, Azure, and GCP.
You can set the SE group properties max_queues_per_vnic and num_dispatcher_cores to zero. The SE requires a reboot post configuration.
[admin:controller]: > configure serviceenginegroup Default-Group [admin:controller]: serviceenginegroup> max_queues_per_vnic 0 [admin:controller]: serviceenginegroup> num_dispatcher_cores 0 [admin:controller]: serviceenginegroup> save
The default value of RSS knobs are as follows:
max_queues_per_vnic = 1
num_dispacther_core = 0
Configuring Dedicated Dispatcher for Public Clouds
The Dedicated Dispatcher (DD) is a runtime SE-Group configurable property which can be set or unset by toggling the dedicated_dispatcher_core knob.
[admin:controller]: > configure serviceenginegroup Default-Group [admin:controller]: serviceenginegroup> dedicated_dispatcher_core [admin:controller]: serviceenginegroup> save [admin:controller]: > configure serviceenginegroup Default-Group [admin:controller]: serviceenginegroup> no dedicated_dispatcher_core [admin:controller]: serviceenginegroup> save
LRO in Routing Use Case
If NSX Advanced Load Balancer is deployed in Preserve-Client IP environments, the LRO needs to be disabled, because NSX Advanced Load Balancer acts as default gateway. There is known issue where LRO assembles the Routing packets that are originated from the server and if the DF bit is set, it leads to a packet loss. So it is recommended to disable LRO in Routing Use-Cases of NSX Advanced Load Balancer.
Since LRO is a boot up property, you need to disable LRO, before upgrading to 22.1.3 or later release.
Configuration of LRO on ESXi Host
You need to configure LRO Settings applied on SE level and few settings at ESXI Host level (Hardware Level).
To determine whether LRO is enabled for VMXNET3 Adapters on an ESXi Host, refer Determine Whether LRO Is Enabled for VMXNET3 Adapters on an ESXi Host guide.
Configuration of LRO in NSX Advanced Load Balancer
Starting with NSX Advanced Load Balancer 22.1.3 version, LRO is enabled on serviceenginegroup by default for supported environments, such as, Vmware/ NSX-T Cloud.
The following is the CLI to determine if LRO is enabled at SE (Service Engine) Level:
[admin:10-102-65-80]: > show serviceenginegroup Default-Group | grep lro | se_lro | True
The following is the CLI to disable LRO on SE Group level:
[admin:10-102-65-80]: > configure serviceenginegroup Default-Group [admin:10-102-65-80]: serviceenginegroup> no se_lro [admin:10-102-65-80]: serviceenginegroup> save
The SEs should be rebooted for this configuration to work.
Verifying LRO Status in NSX Advanced Load Balancer
LRO is an interface property, hence you need to check the LRO status per interface layer of NSX Advanced Load Balancer.
admin:vmwareft-ctlr1]: > show serviceengine Avi-se-owwhw interface | grep lro | lro_on | True [admin:vmwareft-ctlr1]: >
