Hyper-threading works by duplicating certain sections of the processor that store the architectural state. However, the logical processors in a hyper-threaded core share the execution resources, including the execution engine, caches, and system bus interface.This allows a logical processor to borrow resources from a stalled logical core (assuming both logical cores are associated with the same physical core). A processor stalls due to a delay in sent data owing to a cache miss, branch misprediction, or data dependency so it can finish processing the current thread.

Avi Load Balancer has two knobs to control the use of hyper-threaded cores and the distribution (placement) of se_dps on the hyper-threaded CPUs. These two knobs are part of the SE group. The following are the two knobs:

  • You can enable hyper threading on the SE using:

threaded cores use_hyperthreaded_cores –
True [default] | False     
enable or disable se_dps to use hyper-

  • You can control the placement of se_dps on the hyper threaded CPUs using:

se_hyperthreaded_mode – 
         SE_CPU_HT_AUTO[default]
         SE_CPU_HT_SPARSE_DISPATCHER_PRIORITY
         SE_CPU_HT_SPARSE_PROXY_PRIORITY
         SE_CPU_HT_PACKED_CORES 
         
         controls the distribution of se_dps on hyper-threads
Note:

To use these knobs:

  • The processor must support hyper-threading.

  • Use of hyper-threading must be enabled at BIOS.

use_hyperthreaded_cores: You can use this knob to enable or deactivate the use of hyper-threaded cores for se_dps. This knob can be configured using CLI or UI.

The following are the CLI commands:

[admin:vpr-ctrl]: serviceenginegroup> use_hyperthreaded_cores [admin:vpr-ctrl]: serviceenginegroup> se_hyperthreaded_mode SE_CPU_HT_AUTO [admin:vpr-ctrl]: serviceenginegroup> save

se_hyperthreaded_mode: You can use this knob to influence the distribution of se_dp on the hyper-threaded CPUs when the number of datapath processes is less than the number of hyper-threaded CPUs online. The knob can be configured only through the CLI.

Note:

Set use_hyperthreaded_cores to True for the mode configured using se_hyperthreaded_mode to take effect.

The following are the supported values for se_hyperthreaded_mode:

  • SE_CPU_HT_AUTO — This is the default mode. The SE automatically determines the best placement. This mode preserves the existing behavior following CPU hyper-threading topology. If the number of data path processes is less than the number of CPUs, this is equivalent to SE_CPU_HT_SPARSE_PROXY_PRIORITY mode.

  • SE_CPU_HT_SPARSE_DISPATCHER_PRIORITY — This mode prioritises the dispatcher instances by attempting to place only one data-path process in the physical CPU. This mode exhausts the physical cores first and then hyper-threads in numerically descending order of CPU number.

  • SE_CPU_HT_SPARSE_PROXY_PRIORITY — This mode prioritises the proxy (non-dispatcher) instances by attempting to place only one data-path process in the physical CPU. This is useful when the number of data path processes is less than the number of CPUs. This mode exhausts the physical cores and then hyper-threads in numerically ascending order of CPU number.

  • SE_CPU_HT_PACKED_CORES — This mode places the data path processes on the same physical core. Each core can have two dispatchers or two non-dispatcher (proxy) instances being next to each other. This mode is useful when the number of data path processes is less than the number of CPUs. This mode exhausts the hyper-threads serially on each core before moving on to the next physical core.

When hyper threading is enabled, there is a change in behaviour with DP isolation enabled. The modes that will be affected are SE_CPU_HT_SPARSE_DISPATCHER_PRIORITY, SE_CPU_HT_SPARSE_PROXY_PRIORITY and through extension SE_CPU_HT_AUTO. With DP isolation, number of physical cores is reserved to be excluded from the dp_set(cgroup of the se_dp processes). This will result in certain cores being masked from datapath’s HT distribution logic.

This number is calculated as follows: floor(num_non_dp_cpus/ 2).

For instance, if num_non_dp_cpus is 5, 2 cores are reserved for non-datapath exclusivity. To use HT, (with or without DP isolation), the following config knobs are provided in SEgroup:

  1. use_hyperthreaded_cores (true/ false)

  2. se_hyperthreaded_mode (one of the 4 modes discussed here)

Example Configuration

 | use_hyperthreaded_cores                 | True                                                    |
 | se_hyperthreaded_mode                   | SE_CPU_HT_SPARSE_DISPATCHER_PRIORITY       |