Configuring Lossless Ethernet for NVMe over RDMA

NVMe over RDMA in ESXi requires a lossless Ethernet network to operate efficiently.

To establish lossless networks, you can select one of the available QoS settings.

Enabling Global Pause Flow Control

In this network configuration, ensure global pause flow control is enabled on the network switch ports. Also, ensure that RDMA capable NICs in the host auto-negotiate to the correct flow control automatically.

To check flow control, run the following commands.

#esxcli network nic get -n vmnicX
   Pause RX: true
   Pause TX: true

If the above command options are not set to true, run the following command.

#esxcli network nic pauseParams set -r true -t true -n vmnicX

Enabling Priority Flow Control

For RoCE traffic to be lossless, you must configure the PFC priority value to 3 in the physical switch and hosts. You can configure the PFC in the ESXi host in two ways:

Automatic Configuration. Starting ESXi 7.0, you can apply DCB PFC configuration automatically on the host RNIC, if the RNIC driver supports DCB and DCBx.
You can verify the current DCB settings by running the following command.
```
#esxcli network nic dcb status get -n vmnicX
```
Manual configuration. In some cases, the RNIC drivers provide a method to manually configure the DCB PFC using driver specific parameters. To use this method, see vendor specific driver documentation. For example,in Mellanox ConnectX-4/5 driver, you can set the PFC priority value to 3 by running the following command and reboot the host.
```
#esxcli system module parameters set -m nmlx5_core -p "pfctx=0x08 pfcrx=0x08"
```

Enabling DSCP based PFC

DSCP based PFC is another way to configure lossless network. In physical switches and hosts, you must set the DSCP value to 26. To use this option, see vendor specific driver documentation. For example, in Mellanox ConnectX-4/5 driver, you can set the DSCP tag value to 26 by running the following commands.

Enable PFC and DSCP trust mode

#esxcli system module parameters set -m nmlx5_core -p "pfctx=0x08 pfcrx=0x08 trust_state=2"

Set DSCP value to 26

#esxcli system module parameters set -m nmlx5_rdma -p "dscp_force=26"

Verify parameters to check to confirm if settings are correct and set.

esxcli system module parameters list -m nmlx5_core | grep 'trust_state\|pfcrx\|pfctx'

Reboot the host