VMDirectPath I/O allows the guest operating system to access the GPU directly, bypassing the ESXi hypervisor. By using passthrough devices, you can use resources more efficiently and improve the performance of your vSphere Bitfusion environment. Enabling passthrough of the GPU provides a level of performance on vSphere close to that of its native system.

Prerequisites

  • Verify that you have the privileges that you need for the task that you plan to perform.
    • Verify that you have the Virtual machine.Configuration.Add or remove device privilege.
    • Verify that you have the Virtual machine.Configuration.Advanced configuration privilege.
    • If you plan to increase the memory reservation when you edit a virtual machine, verify that you have the Virtual machine.Configuration.Change resource privilege.
    • Verify that you have the Virtual machine.Configuration.Change Memory privilege.
  • Verify that the virtual machine of the vSphere Bitfusion server is powered off.
  • To use DirectPath, verify that Intel Virtualization Technology for Directed I/O (VT-d) or AMD I/O Virtualization Technology (IOMMU) is enabled in the BIOS of the ESXi host.
  • Verify that the GPU PCI devices are connected to the host and marked as available for passthrough. See Enable a GPU for Passthrough.
  • If your ESXi host is configured to boot from a USB device, or if the active coredump partition is configured to be on a USB device or SD cards connected through USB channels, disable the USB controller for passthrough.
    Note: VMware does not support a USB controller passthrough for ESXi hosts that boot from USB devices or SD cards connected through USB channels. A configuration in which the active coredump partition is configured to be on a USB device or SD card connected through USB channels is also not supported. For information, see http://kb.vmware.com/kb/1021345.

Procedure

  1. Add a GPU device.
    1. In the vSphere Client, right-click the vSphere Bitfusion virtual machine in the inventory and select Edit Settings.
    2. On the Virtual Hardware tab, click the Add New Device button.
    3. From the drop-down menu, under Other Devices, select PCI Device.
    4. Expand the New PCI device section and select the access type.
    5. In the New PCI device section, select a GPU from the PCI device drop-down menu.
      Note: By default, the same PCI device address is listed for each new GPU. When adding multiple devices, you must select the PCI addresses of each individual device.
    6. Click OK.
  2. Configure the CPU and memory resources of the ESXi host.
    If the ESXi host is a dedicated vSphere Bitfusion server, set the CPU and memory to their maximum values. If the host machine is not dedicated to vSphere Bitfusion, specify the minimum CPU value as the number of GPUs multiplied by 4, and the minimum memory as 1.5 times that of the aggregated GPU card memory or 32 GB, whichever is higher.
    1. In the vSphere Client, right-click the vSphere Bitfusion virtual machine and select Edit Settings.
    2. Expand the CPU section and edit the resources.
    3. Expand the Memory section and edit the resources.
    4. Click OK.
  3. Adjust the memory mapped I/O (MMIO) size.
    By default, the vSphere Bitfusion installer configures an MMIO size of 256 GB. To calculate how much actual memory you must reserve for MMIO, consider the following MMIO memory calculations for two and three cards, each with 16 GB of memory.
    • 2 x 16 GB = 32. Round 32 GB to the next power of 2, and the memory mapped I/O size that is needed is 64 GB.
    • 3 x 16 GB = 48. Round 48 GB to the next power of 2, and memory mapped I/O size that is needed is 64 GB.
    1. In the vSphere Client, right-click the vSphere Bitfusion virtual machine and select Edit Settings.
    2. Click VM Options, and expand the Advanced section.
    3. Under Configuration Parameters, click Edit Configuration.
    4. In the Configuration Parameters dialog box, locate the parameter pciPassthru.64bitMMIOSizeGB, and enter the MMIO size in gigabytes.
    5. Click OK.
  4. (Optional) Take a snapshot of the virtual machine.
    Snapshots capture the state of the virtual machine at the time you take the snapshot. If an error occurs when you start the virtual machine, you can recover your vSphere Bitfusion installation from the snapshot.
    1. In the vSphere Client, right-click the vSphere Bitfusion virtual machine and select Snapshots > Take Snapshot.
    2. Enter a name and description for the snapshot.
    3. Click Create.
  5. If you are deploying a subsequent vSphere Bitfusion server, enable it.
    Note: When you enable an additional vSphere Bitfusion server, the server recognizes the primary vSphere Bitfusion server you previously deployed. If you do not perform this step before you power on the vSphere Bitfusion virtual machine, your subsequent server becomes primary and the configuration of the vSphere Bitfusion cluster is overwritten.
    1. In the vSphere Client, right-click the virtual machine in the inventory and select Bitfusion > Enable Bitfusion..
    2. In the Enable Bitfusion dialog box, select the For a server, this will allow if to used used as a GPU server radio button and click Enable.
  6. In the vSphere Client, right-click the vSphere Bitfusion virtual machine and select Power > Power On.
    If you are powering on multiple vSphere Bitfusion servers, power them on one at a time. Wait three or more minutes between powering on each server.

Results

After the virtual machine powers on, allow it to run for ten or more minutes before performing any further configuration tasks or operations. During this time the virtual machine registers with vCenter Server and, downloads and installs the NVIDIA driver if you choose this installation option.

What to do next