When a Windows virtual machine in vSphere 5.1 and later transmits large UDP packets, the throughput is lower than expected or is oscillating even when other traffic is negligible.

Problem

When a Windows virtual machine transmits UDP packets larger than 1024 bytes, you experience lower than expected or oscillating throughput even when other traffic is negligible. In case of a video streaming server, video playback pauses.

Cause

For every UDP packet larger than 1024 bytes, the Windows network stack waits for a transmit completion interrupt before sending the next packet. Unlike for earlier releases, vSphere 5.1 and later releases do not provide a transparent workaround of the situation.

Solution

  • Increase the threshold in bytes at which Windows changes its behavior for UDP packets by modifying the registry of the Windows guest OS.
    1. Locate the HKLM\System\CurrentControlSet\Services\Afd\Parameters registry key.
    2. Add a value with the name FastSendDatagramThreshold of type DWORD equal to 1500.
    For information about fixing this issue in the Windows registry, see http://support.microsoft.com/kb/235257.
  • Modify the coalescing settings of the virtual machine NIC.
    If the Windows virtual machine has a VMXNET3 vNIC adapter, configure one of the following parameters in the .vmx file of the virtual machine. Use the vSphere Web Client, or directly modify the .vmx file.
    Action Parameter Value
    Increase the interrupt rate of the virtual machine to a higher rate than expected packet rate. For example, if the expected packet rate is 15000 interrupts per second, set the interrupt rate to 16000 interrupts per second. Set the ethernetX.coalescingScheme parameter to rbc and the ethernetX.coalescingParams parameter to 16000. The default interrupt rate is 4000 interrupts per second.

    ethernetX.coalescingScheme

    ethernetX.coalescingParams

    rbc

    16000

    Disable coalescing for low throughput or latency-sensitive workloads. For information about configuring low-latency workloads, see Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs. ethernetX.coalescingScheme disabled

    Revert to the coalescing algorithm from earlier ESXi releases.

    Note: The ability to revert to the earlier algorithm will not be available in later vSphere releases.
    		ethernetX.coalescingScheme calibrate

    X next to ethernet stands for the sequence number of the vNIC in the virtual machine.

    For more information about configuring parameters in the .vmx file, see the vSphere Virtual Machine Administrationdocumentation.

  • Modify ESXi host coalescing settings.
    This approach affects all virtual machines and all virtual machine NICs on the host.

    You can edit the advanced system settings list for the host in the vSphere Web Client, or by using a vCLI console command on the host from the ESXi Shell.

    Action Parameter in the vSphere Web Client Parameter for the esxcli system settings sdvanced set Command Value
    Set a default interrupt rate higher than the expected packet rate. For example, set the interrupt rate to 16000 if 15000 interrupts are expected per second.

    Net.CoalesceScheme

    Net.CoalesceParams

    /Net/CoalesceScheme

    /Net/CoalesceParams

    rbc

    16000

    Disable coalescing for low throughput or latency-sensitive workloads. For information about configuring low-latency workloads, see Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs. Net.CoalesceDefaultOn /Net/CoalesceDefaultOn 0

    Revert to the coalescing scheme from earlier ESXi releases.

    Note: The ability to revert to the earlier algorithm will not be available in later vSphere releases.
    		Net.CoalesceScheme /Net/CoalesceScheme calibrate

    For information about configuring a host from the vSphere Web Client, see the vCenter Server and Host Management documentation. For information about setting host properties by using a vCLI command, refer to the vSphere Command-Line Interface Reference documentation.