Some key components such as flow cache, RSS, rx/tx ring size, and software queue size determine the performance of Bare Metal Edge. Follow the recommendations for these components only in cases where you want to improve performance. Depending on the traffic and the application that is running, you might need to change some settings to achieve the best performance.
Ring Buffer Size
Ring Buffer is a common building block to store packets that are received on the physical NIC or about to be sent. The Ring Buffer is a data structure used in computer programming to efficiently manage a fixed-size buffer or queue. Its circular or ring shape indicates that when the buffer reaches its maximum capacity, new data starts overwriting the old data in a continuous loop.
For Bare Metal Edge, the default ring size for both Tx and Rx is 512. Sometimes, when the traffic is heavy, using a small ring size might result in significant packet loss. However, using a bigger ring size might increase the memory footprint resulting in performance penalty on latency. We recommend that you use a proper ring size with which packet loss is minimized. However, it may be cases where may consider increasing Ring Buffer. VMware NSX has an alarm that helps you identify whether the Ring Buffer size needs to be changed and whether the packet loss is possible. For more information, see Edge Health Events.
Change the ring buffer size as follows:
Validate the dataplane ring size.
Edge-node> get dataplane | find ring Bfd_ring_size : 512 Lacp_ring_size : 512 Learning_ring_size : 512 Livetrace_ring_size: 512 Rx_ring_size : 512 ---> current receive ring size Slowpath_ring_size : 512 Tx_ring_size : 512 ---> current transmit ring size
Set the dataplane ring size to one of these values: 512, 1024, 2048, or 4096 (number of descriptors).
Example: Edge-node> set dataplane ring-size rx 1024 Edge-node> set dataplane ring-size tx 1024Restart the dataplane service:
Edge-node> restart service dataplane
Caution:Restarting the dataplane service would impact traffic forwarding.
Flow Control
Flow Control is a mechanism that regulates the data flow between network devices, preventing data loss or congestion. However, deactivating flow control can lead to performance optimization in certain scenarios.
Buffering and latency reduction: When the receiving device cannot handle the incoming data rate, flow control mechanisms such as the IEEE 802.3x standard for Ethernet rely on buffering packets. This buffering introduces additional latency, as packets are held in the buffer until they are processed. By deactivating flow control, you eliminate the need for buffering and reduce latency, which is essential for time-sensitive applications.
Avoiding packet drops: Flow control can cause dropped packets in some cases. When a receiving device gets overloaded and sends a pause frame to the transmitting device, the data transmission is halted until the buffer is cleared. If the buffer fills up faster than it can be emptied, packets might be dropped. By deactivating flow control, you prevent the possibility of dropped packets and ensure continuous data flow.
Optimizing for high-speed links: Flow control mechanisms were originally designed for slower network speeds. With the advancements in Ethernet technology, especially with the emergence of 10 Gigabit and higher-speed links, flow control may not be necessary or beneficial. The higher bandwidth available in these links can accommodate the data flow without the need for flow control, making it more efficient to deactivate it.
Predictable performance: Deactivating flow control can lead to more deterministic behavior in the network. Data transmission occurs without interruptions, enabling more predictable performance characteristics. This predictability is important in real-time applications, such as video streaming or VoIP, where a consistent and low-latency data delivery is essential.
Before making any changes to flow control settings, you must assess the network's needs, monitor performance, and conduct thorough testing. Enabling flow control might impact the traffic performance. Pause frames are related to Ethernet flow control and are used to manage the data transmission pace on a network segment.
Sometimes a sending node ToR/Edge may transmit data faster than a receiving node can accept it. In such cases, the overloaded receiving node sends pause frames back to the sender, pausing the transmission of traffic for a brief period. This slows down the speed with which packets are forwarded and impacts performance. By default, flow control must be deactivating on the Top of the Rack switch/router and NSX Bare Metal Edge.
Receive Side Scaling
Receive Side Scaling (RSS) allows network packets from a pNIC to be processed in parallel on multiple CPUs by creating multiple hardware queues. RSS is configured if multiple fast-path cores are used. For each packet, a hash value is generated based on 5-tuple to distribute that packet to a specific core, with sufficient entropy 5-tuple variations in the traffic. With this feature, the hashing is based on the IP address source, IP address destination, IP protocol, layer-4 port source, and layer-4 port destination. This enables improved traffic distribution across all processing cores, scaling up and maximizing system throughput.
For NSX Bare Metal Edge, RSS is configured automatically by using all cores from one NUMA node as fast-path cores and configuring each of them to poll from all physical pNICs. Note that RSS is a receiver side configuration, and for packets transmission, each core has a dedicated tx queue from each pNIC.
Flow Cache
Flow Cache helps reduce CPU cycles spent on known traffic flows. NSX Edge node uses flow cache to achieve high packet throughput. This feature records actions applied on each flow when the first packet in the flow is processed so that subsequent packets can be processed using a match-and-action procedure.
When the key collisions rates are high, increasing the flow cache size help process the packets most efficiently. However, increasing the cache size might impact memory consumption. Typically, the higher the hit rates, the better the performance. For example, an Edge with close to 100% hit rate can process packets most efficiently. We recommend that you allocate a large capacity for flow cache, if the system resources allow. The default flow cache memory allocation is 115 MB per core.
Flow cache is enabled by default. Beginning with VMware NSX 4.1.1, deactivating flow cache result in an alarm.
Change the flow cache size as follows:
Validate flow cache configuration.
Edge-node> get dataplane flow-cache config Enabled : true Mega_hard_timeout_ms: 4985 Mega_size : 262144 Mega_soft_timeout_ms: 4937 Micro_size : 262144
Change the flow cache size.
Edge-node> set dataplane flow-cache-size 524288
Restart the dataplane service.
Edge-node> restart service dataplane
Caution:Restarting the dataplane service impact traffic forwarding.
Hyperthreading
Enabling hyperthreading on Bare Metal Edges can cause performance issues as data-path fast-path threads incorrectly sharing a physical core with another process/thread. By default, hyperthreading is always deactivated through a grub parameter when you boot the Edge.
