Passthrough devices provide the means to use resources efficiently and improve the performance of your environment.

In vSphere 8.0 and earlier, passthrough devices were added or removed only when the VM was powered off. The current version of vSphere offers hot-add and hot-remove support for VMDirectPathI/O devices. Hot-add refers to the procedure of adding a passthrough device when the VM is powered on.

The following features are available for hot-add and hot-remove support for VMDirectPathI/O devices.
  • Hot-add of a passthrough enabled device to a VM by vSphere API. You can perform the operation in one of the following ways:
    1. You enable the passthrough for an existing device that is discovered at ESXi boot time and hot-add the device to the VM.
    2. Alternatively, you can physically hot-add a new device to an empty PCIe hot-plug slot of the host, enable passthrough for the newly added device, and then hot-add the device to the VM.
      Note: This feature requires PCIe Native hot plug certification for the server.
  • Hot-remove of a passthrough device from a VM by vSphere API.
  • Surprise hot-remove of passthrough device which is in use by a VM, can be physically pulled out of the device from the host.
    Note: This feature requires PCIe Native surprise hot plug certification for the server.
    Note: Hot-add and hot-remove of VMDirectPath I/O is supported for only NVMe devices. Support for hot-add and hot-remove of VMdirectPathI/O is not available in the vSphere Client.
The following features are not supported for VMs with hot plug of passthrough enabled"
  • Input–output Memory Management Unit (IOMMU) exposed to the guest operating system.
  • CPU hot plug.
  • Memory hot plug.
  • Dynamic Direct path I/O.
  • Support for VMs with the number of vCPUs > 128.
  • Virtual interrupts also known as interrupt posting might be disabled for VMs with the hot plug of the passthrough enabled.
  • Graceful physical hot-removal of passthrough device from the host is not supported.
  • UI support for hot-add and hot-remove.
  • All features that are unavailable with VM Direct Path I/O.

Requirements for Platforms, Devices, and Guest Operating Systems

Server and Device Requirements

  • Ensure that hot-add and hot-remove of VMDirectPath I/O devices feature is supported by your server Original equipment manufacturer (OEM).
  • The server platform firmware must be Unified Extensible Firmware Interface (UEFI).
  • Ensure that the server platform and device conforms to the requirements mentioned in the KB 2142307.
  • The server should be certified for the VM Direct Path I/O.
  • For surprise hot-remove of a passthrough device from a VM the server model should be certified for PCIe Native surprise hot plug.
Note: For more information on server certification, see VMware Compatibility Guide.

Guest Operating Systems Requirements

  • Ensure that the Guest Operating System has support of NVMe hot plug and NVMe surprise hot-removal.
  • Use the latest and stable GOS distribution that contains all the fixes of NVMe driver I/O stack.

The vCenter and ESXi Requirements

The ESXi and the vCenter versions must be 8.0 update 1 or later.

Limits

  • Maximum number of passthrough devices that can be hot-added in a single ReconfigureVM API call is 1.​
  • Maximum number of passthrough devices that can be hot-removed in a single ReconfigureVM API call is 1.​
  • Other VM reconfiguration changes cannot be requested in the same ReconfigureVM API call when hot-add or hot-remove a passthrough device is requested.​
  • Maximum number of NVMe passthrough devices that a VM can support is 32.

Enable Hot-Add and Hot-Remove of VMDirectPath I/O Devices

You can enable hot-add and hot-remove of VMDirectPath I/O devices by preparing your vCenter and the ESXi host.

Prerequisites

Prepare the vCenter and the ESXi host.
  • Deploy the vCenter and stateful install ESXi with the build combination.
  • Set the VMKernel boot parameter.
    set -s maxIntrCookies -v 4096
  • Restart the ESXi host.
  • Deploy the data center and add the host.

Procedure

  1. Browse to a host in the vSphere Client.
  2. On the Configure tab, expand Hardware and click PCI Devices.
  3. Select the NVMe PCI device and press Toggle Passthrough.

What to do next

Prepare and configure the VM.

Prepare and configure VM

Passthrough devices provide the means to use resources efficiently and improve the performance of your environment. You can prepare and configure your virtual machine.

Deploy and configure the VM (UEFI boot)

Procedure

  1. Browse to the virtual machine in the vSphere Client.
    1. To find a virtual machine, select a data center, folder, cluster, resource pool, or host.
    2. Click the VMs tab.
  2. Right-click the virtual machine and select Edit Settings.
  3. Expand the Memory section and set the Reserve all guest memory (All locked).
  4. Click Save.
  5. Set fixedPassthruHotPlugEnabled to TRUE for the VM using vSphere API ReconfigVM_Task() For more information, see https://developer.broadcom.com.
    Note: Search for the term Hotplug using the search bar in https://developer.broadcom.com.
    1. On vCenter, Right click on VM → Compatibility → Upgrade VM Compatibility to vmx-20 and click save.
    2. Set motherboardLayout to ACPI using the vSphere API. For more information, see https://developer.broadcom.com.
      Note: Search for the term Hotplug using the search bar in https://developer.broadcom.com.
  6. (Optional) Add the passthrough enabled PCI devices to the VM.
  7. Power on the VM.
    Note: Verify if the ongoing hot-plug operation has succeeded before proceeding with the next hot-plug operation.
  8. Perform hot-add and hot-remove of passthrough devices using vSphere API. For more information, see https://developer.broadcom.com.
    Note: Search for the term Hotplug using the search bar in https://developer.vmware.com/samples.

What to do next

Verify if the hot plug operation has succeeded.

Verify If the Hot Plug Operation is Successful using vSphere Client

To check if your hot-add, hot-remove, and surprise hot-remove operation has succeeded, you can verify using the vSphere Client.

Procedure

  1. Locate the virtual machine in the vSphere Client.
  2. Select a data center, folder, cluster, resource pool, or host and click the VMs tab.
  3. Click Virtual Machines and click the virtual machine from the list.
  4. In the PCI Devices table, verify the following:
    Option Description

    Verify if the hot-add of a passthrough -enabled device to a VM using vSphere API is successful.

    The hot-added passthrough device is displayed.

    Verify if the hot-remove of a passthrough device from a VM using vSphere API is successful.

    The hot-removed passthrough device is not displayed.

    Verify if the surprise hot-remove (physical hot-remove) of a passthrough device of a VM from the ESXi host is successful.
    • Surprise hot-removed device is not displayed.
    • In the Host View,
      1. Browse to a host in thevSphere Client.
      2. On the Configure tab, expand Hardware, and click PCI Devices.
      3. Verify that the surprise hot-removed device is not displayed.

Verify If the Hot Plug Operation is Successful using VM

To check if your hot-add, hot-remove, and surprise hot-remove operation has succeeded, you can verify using the Virtual Machine.

Procedure

  1. In the Virtul Machine, open an SSH connection to the Guest Operating System.
  2. Log in as the root user.
  3. Run the command lspci.
  4. Verify the results.
    Option Description

    Verify if the hot-add of a passthrough -enabled device to a VM using vSphere API is successful.

    The hot-added passthrough device is displayed.
    Note: The passthrough device might have a different Spotfire Binary Data File (SBDF) address than the one on the host.

    Verify if the hot-remove of a passthrough device from a VM by using vSphere API is successful.

    The hot-removed passthrough device is not displayed.

    Verify if the surprise hot-remove (physical hot-remove) of a passthrough device of a VM from the ESXi host is successful.

    The surprise hot-removed device is not displayed.