In addition to traditional iSCSI, ESXi supports the iSCSI Extensions for RDMA (iSER) protocol. When the iSER protocol is enabled, the iSCSI framework on the ESXi host can use the Remote Direct Memory Access (RDMA) transport instead of TCP/IP. You can configure iSER on your ESXi host.

For more information about the iSER protocol, see Using iSER Protocol with ESXi.

The entire setup and configuration process for VMware iSER involves several steps.

Step Description
Install and View an RDMA Capable Network Adapter To configure iSER with ESXi, you must first install an RDMA capable network adapter, for example, Mellanox Technologies MT27700 Family ConnectX-4. After you install this type of adapter, the vSphere Client displays its two components, an RDMA adapter and a physical network adapter vmnic#.
Enable the VMware iSER Adapter To be able to use the RDMA capable adapter for iSCSI, use the esxcli to enable the VMware iSER storage component. The component appears in the vSphere Client as a vmhba# storage adapter under the VMware iSCSI over RDMA (iSER) Adapter category.
Modify General Properties for iSCSI or iSER Adapters on ESXi Hosts If needed, change the default name and alias assigned to the iSER storage adapter vmhba#.
Configure Port Binding for iSCSI or iSER on ESXi You must create network connections to bind the iSER storage adapter vmhba# and the RDMA capable network adapter vmnic#. The process of configuring these connections is called port binding.
Note: iSER does not support NIC teaming. When configuring port binding, use only one RDMA adapter per vSwitch.
Configure Dynamic or Static Discovery for iSCSI and iSER on ESXi Host Set up the dynamic or static discovery for your iSER storage adapter vmhba#. With the dynamic discovery, each time the initiator contacts a specified iSER storage system, it sends the SendTargets request to the system. The iSER system responds by supplying a list of available targets to the initiator. With the static discovery, you manually enter information for the targets.
Set Up CHAP for iSCSI or iSER Storage Adapter If your environment uses the Challenge Handshake Authentication Protocol (CHAP), configure it for your iSER storage adapter vmhba#.
Set Up CHAP for Target You can also configure different CHAP credentials for each discovery address or static target.
Enable Jumbo Frames for Networking If your environment supports Jumbo Frames, enable them for the iSER storage adapter vmhba#.

Install and View an RDMA Capable Network Adapter

ESXi supports RDMA capable network adapters, for example, Mellanox Technologies MT27700 Family ConnectX-4. After you install such adapter on your host, the vSphere Client displays its two components, an RDMA adapter and a physical network adapter.

You can use the vSphere Client to view the RDMA adapter and its corresponding network adapter.

Procedure

  1. In the vSphere Client, navigate to the ESXi host.
  2. Under Networking, click RDMA adapters.
    In this example, the RDMA adapter appears on the list as vmrdma0. The Paired Uplink column displays the network component as the vmnic1 physical network adapter.

    The RDMA adapter appears on the list as vmrdma0. The Paired Uplink column displays the network component as the vmnic1 physical network adapter.

  3. To verify the description of the adapter, select the RDMA adapter from the list, and click the Properties tab.

Results

You can use the vmnic# network component of the adapter for such storage configurations as iSER or NVMe over RDMA. For the iSER configuration steps, see Configure iSER with ESXi. For information about NVMe over RDMA, see Configuring NVMe over RDMA (RoCE v2) on ESXi.

Enable the VMware iSER Adapter

To be able to use the RDMA capable adapter for iSCSI, use the esxcli to enable the VMware iSER storage component. After you enable the component, it appears in the vSphere Client as a vmhba# storage adapter under the VMware iSCSI over RDMA (iSER) Adapter category.

Prerequisites

  • Make sure that your iSCSI storage supports the iSER protocol.
  • Install the RDMA capable adapter on your ESXi host. For information, see Install and View an RDMA Capable Network Adapter.
  • For RDMA capable adapters that support RDMA over Converged Ethernet (RoCE), determine the RoCE version that the adapter uses.
  • Use the RDMA capable switch.
  • Enable flow control on the ESXi host. To enable flow control for the host, use the esxcli system module parameters command. For details, see the VMware knowledge base article at http://kb.vmware.com/kb/1013413.
  • Make sure to configure RDMA switch ports to create lossless connections between the iSER initiator and target.

Procedure

  1. Use the ESXi Shell or vSphere CLI to enable the VMware iSER storage adapter and set its RoCE version.
    1. Enable the iSER storage adapter.
      esxcli rdma iser add
    2. Verify that the iSER adapter has been added.
      esxcli iscsi adapter list
      The output is similar to the following. 
      Adapter Driver State UID Description
------- ------ ------- ------------- -------------------------------------
vmhba64 iser unbound iscsi.vmhba64 VMware iSCSI over RDMA (iSER) Adapter
    3. Specify the RoCE version that iSER uses to connect to the target.
      Use the RoCE version of the RDMA capable adapter. The command you enter is similar to the following:

      esxcli rdma iser params set -a vmhba64 -r 1

      When the command completes, a message similar to the following appears in the VMkernel log.

      vmkernel.0:2020-02-18T18:26:15.949Z cpu6:2100717 opID=45abe37e)iser: iser_set_roce: Setting roce type: 1 for vmhba: vmhba64
vmkernel.0:2020-02-18T18:26:15.949Z cpu6:2100717 opID=45abe37e)iser: iser_set_roce: Setting rdma port: 3260 for vmhba: vmhba64

      If you do not specify the RoCE version, the host defaults to the highest RoCE version the RDMA capable adapter supports.

  2. Use the vSphere Client to display the iSER adapter.
    1. In the vSphere Client, navigate to the ESXi host.
    2. Click the Configure tab.
    3. Under Storage, click Storage Adapters, and review the list of adapters.
      If you enabled the adapter, it appears as a storage vmhba# adapter on the list under the VMware iSCSI over RDMA (iSER) Adapter category.

      The screenshot shows a list of storage adapters and vmhba64 that appears on the list under the VMware iSCSI over RDMA (iSER) Adapter category.

  3. Select the iSER storage vmhba# to review its properties or perform the following tasks.
    Option Description
    Configure port binding for the iSER storage adapter You must create network connections to bind the iSER storage adapter vmhba# and the RDMA capable network adapter vmnic#. The process of configuring these connections is called port binding. For general information about port binding, see Setting Up Network for iSCSI and iSER with ESXi. To configure port binding for iSER, see Configure Port Binding for iSCSI or iSER on ESXi.
    Set up dynamic or static discovery for the iSER storage adapter For information, see Configure Dynamic or Static Discovery for iSCSI and iSER on ESXi Host.
    Configure the Challenge Handshake Authentication Protocol (CHAP) for the iSER storage adapter For information, see Set Up CHAP for iSCSI or iSER Storage Adapter.

What to do next

For more information, see the VMware knowledge base article at https://kb.vmware.com/s/article/79148.