After you configure a vSphere Bitfusion server and client to use PVRDMA for data traffic, the RDMA ports of the network adapter on your vSphere Bitfusion client might display a PORT_DOWN state. You resolve this issue by installing the RDMA kernel modules.

Problem

  • You cannot successfully test the PVRDMA connection between a vSphere Bitfusion server and a client.
  • When you run the ibv_devinfo command, the RDMA ports of your vSphere Bitfusion client are not working.

Cause

After you install the RDMA packages for your operating system, the packages might not install the necessary kernel modules.

Solution

  1. Load the mlx4_ib, ib_umad, rdma_cm, rdma_ucm, and vmw_pvrdma kernel modules.
    This solution is applicable for CentOS, Red Hat Linux, and Ubuntu.
    1. Load the following kernel modules.
      sudo /sbin/modprobe mlx4_ib
      sudo /sbin/modprobe ib_umad
      sudo /sbin/modprobe rdma_cm
      sudo /sbin/modprobe rdma_ucm
    2. Reload the vmw_pvrdma kernel module.
      You must load the vmw_pvrdma kernel module after the other kernel modules.
      Note: In Ubuntu, you must perform this step every time after the virtual machine of your vSphere Bitfusion client boots.
      sudo /sbin/modprobe -r vmw_pvrdma
      sudo /sbin/modprobe vmw_pvrdma
  2. If you are using Ubuntu, run the sudo update-initramfs -k all -u command.
  3. (Optional) To verify that the RDMA ports of your network interface are working, run the ibv_devinfo command.