To enable the SR-IOV mode on KVM, perform the following steps.

Prerequisites

This requires a specific NIC card. The following chipsets are certified by VMware to work with the SD-WAN Gateway and SD-WAN Edge.

  • Intel 82599/82599ES
  • Intel X710/XL710
Note: Before using the Intel X710/XL710 cards in SR-IOV mode on KVM, make sure the supported Firmware and Driver versions specified in the Deployment Prerequisites section are installed correctly.
Note: SR-IOV mode is not supported if the KVM Virtual Edge is deployed with a High-Availability topology. For High-Availability deployments, ensure that SR-IOV is not enabled for that KVM Edge pair.

To enable SR-IOV on KVM:

  1. Enable SR-IOV in BIOS. This will be dependent on your BIOS. Login to the BIOS console and look for SR-IOV Support/DMA. You can verify support on the prompt by checking that Intel has the correct CPU flag. 
    cat /proc/cpuinfo | grep vmx
  2. Add the options on Bboot (in /etc/default/grub). 
    GRUB_CMDLINE_LINUX="intel_iommu=on"
    1. Run the following commands: update-grub and update-initramfs -u.
    2. Reboot
    3. Make sure iommu is enabled. 
      velocloud@KVMperf3:~$ dmesg | grep -i IOMMU
 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 
 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.13.0-107-generic root=/dev/mapper/qa--multiboot--002--vg-root ro intel_iommu=on splash quiet vt.handoff=7 
 [ 0.000000] Intel-IOMMU: enabled
 ….
 velocloud@KVMperf3:~$
  3. Based on the NIC chipset used, add a driver as follows:
    • For the Intel 82599/82599ES cards in SR-IOV mode:
      1. Download and install ixgbe driver from the Intel website.
      2. Configure ixgbe config (tar and sudo make install).
        velocloud@KVMperf1:~$ cat /etc/modprobe.d/ixgbe.conf
      3. If the ixgbe config file does not exist, you must create the file as follows. 
        options ixgbe max_vfs=32,32
options ixgbe allow_unsupported_sfp=1
options ixgbe MDD=0,0
blacklist ixgbevf
      4. Run the update-initramfs -u command and reboot the Server.
      5. Use the modinfo command to verify if the installation is successful. 
        velocloud@KVMperf1:~$ modinfo ixgbe and ip link
 filename: /lib/modules/4.4.0-62-generic/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
 version: 5.0.4
 license: GPL
 description: Intel(R) 10GbE PCI Express Linux Network Driver
 author: Intel Corporation, <[email protected]>
 srcversion: BA7E024DFE57A92C4F1DC93
    • For the Intel X710/XL710 cards in SR-IOV mode:
      1. Download and install i40e driver from the Intel website.
      2. Create the Virtual Functions (VFs).
        echo 4 > /sys/class/net/device name/device/sriov_numvfs
      3. To make the VFs persistent after a reboot, add the command from the previous step to the "/etc/rc.d/rc.local" file.
      4. Deactivate the VF driver.
        echo “blacklist i40evf” >> /etc/modprobe.d/blacklist.conf
      5. Run the update-initramfs -u command and reboot the Server.

Validating SR-IOV (Optional)

You can quickly verify if your host machine has SR-IOV enabled by using the following command:

lspci | grep -i Ethernet

Verify if you have Virtual Functions:

01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function(rev 01)