After you configure a vSphere Bitfusion server and client to use PVRDMA for data traffic, the RDMA ports of the network adapter on your vSphere Bitfusion client might display a PORT_DOWN
state. You resolve this issue by installing the RDMA kernel modules.
Problem
- You cannot successfully test the PVRDMA connection between a vSphere Bitfusion server and a client.
- When you run the ibv_devinfo command, the RDMA ports of your vSphere Bitfusion client are not working.
Cause
After you install the RDMA packages for your operating system, the packages might not install the necessary kernel modules.
Solution
- Load the
mlx4_ib
, ib_umad
, rdma_cm
, rdma_ucm
, and vmw_pvrdma kernel modules.
This solution is applicable for CentOS, Red Hat Linux, and Ubuntu.
- Load the following kernel modules.
sudo /sbin/modprobe mlx4_ib
sudo /sbin/modprobe ib_umad
sudo /sbin/modprobe rdma_cm
sudo /sbin/modprobe rdma_ucm
- Reload the vmw_pvrdma kernel module.
You must load the
vmw_pvrdma kernel module after the other kernel modules.
Note: In Ubuntu, you must perform this step every time after the virtual machine of your
vSphere Bitfusion client boots.
sudo /sbin/modprobe -r vmw_pvrdma
sudo /sbin/modprobe vmw_pvrdma
- If you are using Ubuntu, run the sudo update-initramfs -k all -u command.
- (Optional) To verify that the RDMA ports of your network interface are working, run the ibv_devinfo command.