Learn how to use Large Receive Offload (LRO) to reduce the CPU overhead for processing packets that arrive from the network at a high rate.

LRO reassembles incoming network packets into larger buffers and transfers the resulting larger but fewer packets to the network stack of the host or virtual machine. The CPU has to process fewer packets than when LRO is deactivated, which reduces its utilization for networking especially in the case of connections that have high bandwidth.

To benefit from the performance improvement of LRO, enable LRO along the data path on an ESXi host including VMkernel and guest operating system. By default, LRO is activated in the VMkernel and in the VMXNET3 virtual machine adapters.

For information about the location of TCP packet aggregation in the data path, see VMware Knowledge Base article Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment.

Manage Hardware LRO for All VMXNET3 Adapters on an ESXi Host

Activate the hardware capabilities of host physical adapters to aggregate incoming TCP packets for VXMNET3 VM adapters by using the LRO technology instead of consuming resources for assembling in the guest operating system.

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Edit the value of the Net.Vmxnet3HwLRO parameter.
    • To activate hardware LRO, set Net.Vmxnet3HwLRO to 1.
    • To deactivate hardware LRO, set Net.Vmxnet3HwLRO to 0.
  5. Click OK to apply the changes.

Manage Software LRO for All VMXNET3 Adapters on an ESXi Host

Use software LRO in the VMkernel backend of VMXNET3 adapters to improve networking performance of virtual machines if the host physical adapters do not support hardware LRO.

vSphere supports software LRO for both IPv4 and IPv6 packets.

Prerequisites

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Edit the value of the Net.Vmxnet3SwLRO parameter for VMXNET3 adapters.
    • To activate software LRO, set Net.Vmxnet3SwLRO to 1.
    • To deactivate software LRO, set Net.Vmxnet3SwLRO to 0.
  5. Click OK to apply the changes.

Determine Whether LRO Is Enabled for VMXNET3 Adapters on an ESXi Host

Examine the status of LRO on an ESXi when you estimate the networking performance on a host that runs latency-sensitive workloads.

Prerequisites

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Examine the value of the LRO parameters for VMXNET2 and VMXNET3.
    • For hardware LRO, examine the Net.Vmxnet3HwLRO parameter. If it is equal to 1, hardware LRO is enabled.
    • For software LRO, examine the Net.Vmxnet3SwLRO parameter. If it is equal to 1, hardware LRO is enabled.

Change the Size of the LRO Buffer for VMXNET 3 Adapters

You can change the size of the buffer for packet aggregation for virtual machine connections through VMXNET 3 network adapters. Increase the buffer size to reduce the number of TCP acknowledgments and improve efficiency in workloads.

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Enter a value between 1 and 65535 for the Net.VmxnetLROMaxLength parameter to set the LRO buffer size in bytes.
    By default the size of the LRO buffer is equal to 32000 bytes.

Activate or Deactivate LRO for All VMkernel Adapters on an ESXi Host

Use LRO in the VMkernel network adapters on an ESXi host to improve the networking performance for incoming infrastructure traffic.

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Edit the value of the Net.TcpipDefLROEnabled parameter.
    • To activate LRO for the VMkernel network adapters on the host, set Net.TcpipDefLROEnabled to 1.
    • To deactivate software LRO for the VMkernel network adapters on the host, set Net.TcpipDefLROEnabled to 0.
  5. Click OK to apply the changes.

Change the Size of the LRO Buffer for VMkernel Adapters

You can change the size of the buffer for packet aggregation for VMkernel connections. Increase the buffer size to reduce the number of TCP acknowledgments and improve efficiency in the VMkernel.

Procedure

  1. In the vSphere Client, navigate to the host.
  2. On the Configure tab, expand System.
  3. Click Advanced System Settings.
  4. Enter a value between 1 and 65535 for the Net.TcpipDefLROMaxLength parameter to set the LRO buffer size in bytes.
    By default the size of the LRO buffer is equal to 32768 bytes.

Manage LRO on a VMXNET3 Adapter on a Linux Virtual Machine

If LRO is enabled for VMXNET3 adapters on the host, activate LRO support on a network adapter on a Linux virtual machine to ensure that the guest operating system does not spend resources to aggregate incoming packets into larger buffers.

Prerequisites

Verify that the Linux kernel is 2.6.24 and later.

Procedure

  • ♦ In a terminal window on the Linux guest operating system, run the ethtool command with the -K and lro options.
    • To activate LRO, run the following command: 
      ethtool -K ethY lro on

      where Y in ethY is the sequence number of the NIC in the virtual machine.

    • To deactivate LRO, run the following command: 
      ethtool -K ethY lro off

      where Y in ethY is the sequence number of the NIC in the virtual machine.

Manage LRO on a VMXNET3 Adapter on a Windows Virtual Machine

If LRO is enabled for VMXNET3 adapters on the host, activate LRO support on a network adapter on a Windows virtual machine to ensure that the guest operating system does not spend resources to aggregate incoming packets into larger buffers.

On Windows, the LRO technology is also referred to as Receive Side Coalescing (RSC).

Prerequisites

  • Verify that the virtual machine runs Windows Server 2012 and later or Windows 8 and later.
  • Verify that the virtual machine compatibility is ESXi 6.0 and later.
  • Verify that the version of the VMXNET3 driver installed on the guest operating system is 1.6.6.0 and later.
  • Verify that LRO is enabled globally on a virtual machine that runs Windows Server 2012 and later or Windows 8 and later. See Manage LRO Globally on a Windows Virtual Machine.

Procedure

  1. In the Network and Sharing Center of the guest operating system's Control Panel, click the name of the network adapter.
    A dialog box displays the status of the adapter.
  2. Click Properties, and under the VMXNET3 network adapter type, click Configure.
  3. On the Advanced tab, set both Recv Segment Coalescing (IPv4) and Recv Segment Coalescing (IPv6) to Enabled or Disabled.
  4. Click OK.

Manage LRO Globally on a Windows Virtual Machine

To use Large Receive Offload (LRO) on a VMXNET3 adapter on a virtual machine that runs Windows 8 and later or Windows Server 2012 and later, you must enable LRO globally on the guest operating system. On Windows, the LRO technology is also referred to as Receive Side Coalescing (RSC).

Procedure

  1. To verify whether LRO is deactivated globally on a Windows 8 and later or Windows Server 2012 guest OS, run the netsh int tcp show global command at the command prompt. 
    netsh int tcp show global

    The command displays the status of the global Transmission Control Protocol (TCP) parameters that are set on the Windows 8.x OS. 

    TCP Global Parameters
----------------------------------------------
Receive-Side Scaling State          : enabled
Chimney Offload State               : disabled
NetDMA State                        : disabled
Direct Cache Access (DCA)           : disabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : none
ECN Capability                      : disabled
RFC 1323 Timestamps                 : disabled
Initial RTO                         : 3000
Receive Segment Coalescing State    : disabled

    If LRO is globally deactivated on the Windows 8 and later or Windows Server 2012 machine, the Receive Segment Coalescing State property appears as disabled.

  2. To enable LRO globally on the Windows OS, run the netsh int tcp set global command at the command prompt: 
     netsh int tcp set global rsc=enabled

What to do next

Enable LRO for the VMXNET3 adapter on the Windows 8 and later or Windows Server 2012 virtual machine. See Manage LRO on a VMXNET3 Adapter on a Windows Virtual Machine.