vCenter Server performs compatibility checks before it allows migration of running or suspended virtual machines to ensure that the virtual machine is compatible with the target host.

vMotion transfers the running state of a virtual machine between underlying ESXi systems. Live migration requires that the processors of the target host provide the same instructions to the virtual machine after migration that the processors of the source host provided before migration. Clock speed, cache size, and number of cores can differ between source and target processors. However, the processors must come from the same vendor class (AMD or Intel) to be vMotion compatible.

Migrations of suspended virtual machines also require that the virtual machine be able to resume execution on the target host using equivalent instructions.

When you initiate a migration with vMotion or a migration of a suspended virtual machine, the Migrate Virtual Machine wizard checks the destination host for compatibility. If compatibility problems prevent migration, the wizard displays an error message.

The CPU instruction set available to the operating system and to applications running in a virtual machine is determined at the time that a virtual machine is powered on. This CPU feature set is based on the following items:

  • Host CPU family and model
  • Settings in the BIOS that might deactivate CPU features
  • ESXi version running on the host
  • The compatibility setting of the virtual machine
  • The guest operating system of the virtual machine

To improve CPU compatibility between hosts of varying CPU feature sets, some host CPU features can be hidden from the virtual machine by placing the host in an Enhanced vMotion Compatibility (EVC) cluster. For more information about EVC, see About Enhanced vMotion Compatibility.

Note:

You can hide host CPU features from a virtual machine by applying a custom CPU compatibility mask to the virtual machine, but this is not recommended. VMware, in partnership with CPU and hardware vendors, is working to maintain vMotion compatibility across the widest range of processors. For additional information, search the VMware Knowledge Base for the vMotion and CPU Compatibility FAQ.

vSphere vMotion and CPU Compatibility Scenarios

Without the use of EVC, if the vCenter Server's CPU compatibility checks discover any mismatch between user-level features of the hosts, migration is blocked, whether or not the virtual machine itself has access to those features. A mismatch between kernel-level features of the hosts blocks migration only when the virtual machine has access to a feature that the target host does not provide.

vCenter Server's CPU compatibility checks compare the CPU features available on the source host, the subset of features that the virtual machine can access, and the features available on the target host.

User-level features are non-privileged instructions used by virtual machine applications. These include SSE3, SSSE3, SSE4.1, SSE4.2, and AES. Because they are user-level instructions that bypass the virtualization layer, these instructions can cause application instability if mismatched after a migration with vMotion.

Kernel-level features are privileged instructions used by the virtual machine operating system. These include the AMD No eXecute (NX) and the Intel eXecute Disable (XD) security features.

When you attempt to migrate a virtual machine with vMotion, one of the following scenarios applies:

  • The destination host feature set matches the CPU feature set of the virtual machine. CPU compatibility requirements are met, and migration with vMotion proceeds.
  • The CPU feature set of the virtual machine contains features not supported by the destination host. CPU compatibility requirements are not met, and migration with vMotion cannot proceed.
    Note: EVC overcomes such incompatibility by providing a "baseline" feature set for all virtual machines running in a cluster. This baseline feature set hides the differences among the clustered hosts' CPUs from the virtual machines.
  • The destination host supports the feature set of the virtual machine, plus additional user-level features (such as SSE4.1) not found in the feature set of the virtual machine. CPU compatibility requirements are not met, and migration with vMotion cannot proceed.
    Note: This type of incompatibility is ignored for migrations among hosts in EVC clusters.
  • The destination host supports the feature set of the virtual machine, plus additional kernel-level features (such as NX or XD) not found in the feature set of the virtual machine. CPU compatibility requirements are met, and migration with vMotion proceeds. The virtual machine retains its CPU feature set while it remains powered on, allowing it to migrate freely back to the original host. However, if the virtual machine is rebooted, it acquires a new feature set from the new host. This process might cause vMotion incompatibility if you attempt to migrate the virtual machine back to the original host.

CPU Families and Feature Sets Compatibility with vSphere vMotion

Processors are grouped into families. Processors within a given family generally have similar feature sets.

Processor vendors define processor families. You can distinguish different processor versions within the same family by comparing the processors’ model, stepping level, and extended features. Sometimes, processor vendors have introduced significant architectural changes within the same processor family, such as the SSSE3 and SSE4.1 instructions, and NX/XD CPU security features.

By default, vCenter Server identifies mismatches on features accessible to applications as incompatible to guarantee the stability of virtual machines after migrations with vMotion.

Server hardware’s CPU specifications usually indicate whether or not the CPUs contain the features that affect vMotion compatibility.

For more information on identifying Intel processors and their features, see Application Note 485: Intel® Processor Identification and the CPUID Instruction, available from Intel. For more information on identifying AMD processors and their features, see CPUID Specification, available from AMD.

CPU Compatibility Masks and vSphere vMotion

CPU compatibility masks allow customization of the CPU features visible to a virtual machine.

vCenter Server compares the CPU features available to a virtual machine with the CPU features of the destination host to determine whether or not to allow migrations with vMotion.

To guarantee the stability of virtual machines after a migration with vMotion, VMware sets the default values for CPU compatibility masks.

When a choice between CPU compatibility or guest operating system features (such as NX/XD) exists, VMware provides check-box options to configure individual virtual machines. You can access the configuration options through the Advanced Settings option for the CPU of the virtual machine. For more control over the visibility of CPU features, you can edit the CPU compatibility mask of the virtual machine at the bit level.

Caution: Changing the CPU compatibility masks can result in an unsupported configuration. Do not manually change the CPU compatibility masks unless instructed to do so by VMware Support or a VMware Knowledge base article.

CPU compatibility masks cannot prevent virtual machines from accessing masked CPU features in all circumstances. In some circumstances, applications can detect and use masked features even though they are hidden from the guest operating system. In addition, on any host, applications that use unsupported methods of detecting CPU features rather than using the CPUID instruction can access masked features. Virtual machines running applications that use unsupported CPU detection methods might experience stability problems after migration.

Enhanced vMotion Compatibility

You can use the Enhanced vMotion Compatibility (EVC) feature to help ensure vMotion compatibility for the hosts in a cluster. EVC ensures that all hosts in a cluster present the same CPU feature set to virtual machines, even if the actual CPUs on the hosts differ. Using EVC prevents migrations with vMotion from failing because of incompatible CPUs.

Configure EVC from the cluster settings dialog box. When you configure EVC, you configure all host processors in the cluster to present the feature set of a baseline processor. This baseline feature set is called the EVC mode. EVC uses AMD-V Extended Migration technology (for AMD hosts) and Intel FlexMigration technology (for Intel hosts) to mask processor features so that hosts can present the feature set of an earlier generation of processors. The EVC mode must be equivalent to, or a subset of, the feature set of the host with the smallest feature set in the cluster.

EVC masks only those processor features that affect vMotion compatibility. Enabling EVC does not prevent a virtual machine from taking advantage of faster processor speeds, increased numbers of CPU cores, or hardware virtualization support that might be available on newer hosts.

EVC cannot prevent virtual machines from accessing hidden CPU features in all circumstances. Applications that do not follow CPU vendor recommended methods of feature detection might behave unexpectedly in an EVC environment. VMware EVC cannot be supported with ill-behaved applications that do not follow the CPU vendor recommendations. For more information about creating well-behaved applications, search the VMware Knowledge Base for the article Detecting and Using New Features in CPUs.

Starting with vSphere 7.0 Update 1, you can take advantage of the EVC feature for Virtual Shared Graphics Acceleration (vSGA). vSGA allows multiple virtual machines to share GPUs installed on ESXi hosts and leverage the 3D graphics acceleration capabilities.

Enhanced vMotion Compatibility Requirements for Hosts

To improve CPU compatibility between hosts that have varying CPU feature sets, you can hide some host CPU features from the virtual machines by placing the host in an Enhanced vMotion Compatibility (EVC) cluster. Hosts in an EVC cluster and hosts that you add to an existing EVC cluster must meet EVC requirements.

  • Power off all virtual machines in the cluster that are running on hosts with a feature set greater than the EVC mode that you intend to enable. You can also migrate these virtual machines out of the cluster.

  • All hosts in the cluster must meet the following requirements:

Requirements

Description

Supported ESXi version

ESXi6.7 or later.

vCenter Server

The host must be connected to a vCenter Server system.

CPUs

A single vendor, either AMD or Intel.

Advanced CPU features enabled

Enable these CPU features in the BIOS if they are available:

  • Hardware virtualization support (AMD-V or Intel VT)

  • AMD No eXecute(NX)

  • Intel eXecute Disable (XD)

Note:

Hardware vendors sometimes deactivate particular CPU features in the BIOS by default. You might have problems enabling EVC because the EVC compatibility checks detect the absence of features that are expected to be present for a particular CPU. If you cannot enable EVC on a system with a compatible processor, ensure that all features are enabled in the BIOS.

Supported CPUs for the EVC mode that you want to enable

To check EVC support for a specific processor or server model, see the VMware Compatibility Guide at http://www.vmware.com/resources/compatibility/search.php.

Configured for vMotion

See Host Configuration for vSphere vMotion.

How Do You Activate vSphere EVC on an Existing Cluster

Learn how to activate vSphere Enhanced vMotion Compatibility (EVC) on an existing cluster to ensure vMotion CPU compatibility between the hosts in the cluster. Other cluster features such as vSphere DRS and vSphere HA are fully compatible with EVC.

Prerequisites

Verify that the hosts in the cluster meet the requirements listed in Enhanced vMotion Compatibility Requirements for Hosts.

Procedure

  1. Select a cluster in the vSphere inventory.
  2. Power off all the virtual machines on the hosts with feature sets greater than the EVC mode.
    To determine the EVC mode of all virtual machines on the hosts in the cluster, see How to Determine the EVC Mode of Your Virtual Machine. You must power off all powered on virtual machine with no EVC mode or with an EVC mode that is greater than the EVC mode that you plan to enable on the cluster.
  3. Click the Configure tab, select VMware EVC, and click Edit.
  4. Enable EVC for the CPU vendor and feature set appropriate for the hosts in the cluster, and click OK.

How to Change the vSphere EVC Mode for Your Cluster

Learn how to configure vSphere EVC to ensure that virtual machine migrations between hosts in the cluster do not fail because of CPU feature incompatibilities.

Several EVC approaches are available to ensure CPU compatibility:

  • If all the hosts in a cluster are compatible with a newer EVC CPU or Graphics (vSGA) mode, you can change the EVC CPU or Graphics (vSGA) mode of an existing EVC cluster.

  • You can enable EVC for a cluster that does not have EVC enabled.

  • You can raise the EVC mode to expose more CPU features.

  • You can lower the EVC mode to hide CPU features and increase compatibility.

Prerequisites

  • Verify that all hosts in the cluster have supported CPUs for the EVC mode you want to enable. See Knowledge Base article KB 1003212 for a list of supported CPUs.

  • Verify that all hosts in the cluster are connected and registered on vCenter Server. The cluster cannot contain a disconnected host.

  • Virtual machines must be in the following power states, depending on whether you raise or lower the EVC mode.

    EVC Mode

    Virtual Machine Power Action

    Raise the EVC mode to a CPU baseline with more features.

    Running virtual machines can remain powered on. New EVC mode features are not available to the virtual machines until they are powered off and powered back on again. A full power cycling is required. Rebooting the guest operating system or suspending and resuming the virtual machine is not sufficient.

    Tip:

    You can configure a virtual machine to power cycle when its guest OS is rebooted, by setting the vmx.reboot.powerCycle VM parameter to TRUE. When the virtual machine is powered back on, this configuration parameter is removed. To edit the virtual machine configuration settings, you can use the VMware Host Client. See Edit the Configuration File Parameters in the VMware Host Client topic in the vSphere Single Host Management - VMware Host Client guide. See the VMware vSphere Blog post for information about how to set this parameter to a group of virtual machines using VMware PowerCLI.

    Lower the EVC mode to a CPU baseline with fewer features.

    Power off virtual machines if they are powered on and running at a higher EVC mode than the one you intend to enable.
    Note:

    When you take a snapshot of a virtual machine in an EVC cluster, keep in mind the following use cases:

    • If you take a snapshot of a running virtual machine before an EVC downgrade, an attempt to revert to that snapshot after the EVC downgrade fails and the virtual machine is left in a suspended state.
    • If you take a snapshot of a virtual machine before an EVC upgrade, you can revert to that snapshot before or after the EVC upgrade.

    To verify the EVC mode for virtual machines, see How to Determine the EVC Mode of Your Virtual Machine.

Procedure

  1. Select a cluster in the inventory.
  2. Click the Configure tab.
  3. Under Configuration, select VMware EVC, and click Edit.
  4. On the Change EVC Mode page, select whether to activate or deactivate EVC.

    Option

    Description

    Disable EVC

    The EVC feature is deactivated. CPU compatibility is not enforced for the hosts in this cluster.

    Enable EVC for AMD Hosts

    The EVC feature is enabled for AMD hosts.

    Enable EVC for Intel Hosts

    The EVC feature is enabled for Intel hosts.
  5. From the CPU Mode drop-down menu, select the baseline CPU feature set that you want to enable for the cluster.

    If you cannot select the EVC CPU mode, the Compatibility pane displays the reason, and the relevant hosts for each reason.

  6. (Optional) From the Graphics Mode (vSGA) drop-down menu, select a baseline graphics feature set.

    If you cannot select the EVC vSGA mode, the Compatibility pane displays the reason, and the relevant hosts for each reason.

    Option

    Description

    Baseline Graphics

    Applies the Baseline Graphics feature set that includes features through Direct3D 10.1/OpenGL 3.3.

    Note:

    To configure the vSGA mode to apply the Baseline Graphics set that includes features through Direct3D 10.1/OpenGL 3.3, the virtual machine must be compatible with ESXi 6.5 or later.

    D3D 11.0 class features

    Applies the baseline graphics feature set that includes features through Direct3D 11.0/OpenGL 4.3.

    Note:

    When you create a cluster and you enable EVC graphics mode Direct3D 11.0/OpenGL 4.3 class features, you can add only ESXi 8.0 hosts that support Direct3D 11.0/OpenGL 4.3 to the cluster. When you try to add an ESXi 8.0 host that does not support Direct3D 11.0/OpenGL 4.3 to the cluster, the operation fails with an error message.
  7. Click OK.

How to Determine the EVC Mode of Your Virtual Machine

The EVC mode of a virtual machine determines the CPU and graphics features that a host must have in order for the virtual machine to migrate to that host and power on. The EVC mode of a virtual machine is independent from the EVC mode that you configure for the cluster in which the virtual machine runs.

The EVC mode of a virtual machine is determined when the virtual machine powers on. At power-on, the virtual machine also determines the EVC mode of the cluster in which it runs. If the EVC mode of a running virtual machine or the entire EVC cluster is raised, the virtual machine does not change its EVC mode until it is powered off and powered on again. This means that the virtual machine does not use any CPU features exposed by the new EVC mode until the virtual machine is powered off and powered on again.

For example, you create an EVC cluster that contains hosts with Intel processors and you set the EVC mode to Intel "Merom" Generation (Xeon Core 2). When you power on a virtual machine in this cluster, it runs in the Intel Merom Generation (Xeon Core 2) EVC mode. If you raise the EVC mode of the cluster to Intel "Penryn" Generation (Xeon 45 nm Core 2), the virtual machine retains the lower Intel "Merom" Generation (Xeon Core 2) EVC mode. To use the feature set of the higher EVC mode, such as SSE4.1, the virtual machine must be powered off and powered on again.

Procedure

  1. Navigate to a cluster or a host in the vCenter Server inventory.
  2. Click the VMs > Virtual Machines tab.
    A list of all virtual machines in the selected cluster or on the selected host appears.
  3. To verify the status of the CPU mode, check the EVC CPU Mode column.
    1. If the column is not visible, click Manage Columns at the bottom left corner of the Virtual Machines table.
      The Show Columns pop-up appears.
    2. To show the EVC CPU Mode column, locate and select the respective check box from the list.
    The EVC CPU Mode column shows the CPU modes of all virtual machines in the cluster or on the host.
    Important: For each virtual machine, the EVC CPU Mode column displays the EVC mode defined at the virtual machine level.

    However, if you do not configure per-VM EVC for a virtual machine, the virtual machine inherits the EVC mode of its parent cluster or host. As a result, for all virtual machines that do not have per-VM EVC configured, the EVC CPU Mode column displays the inherited EVC mode of the parent host or cluster.

    If the virtual machine is in an EVC cluster, the EVC mode that you see in the EVC CPU Mode column is defined in the following manner.
    • When the virtual machine is powered on, the EVC CPU Mode column displays either the per-VM EVC mode, or the cluster-level EVC mode.
      Per-VM EVC Cluster-Level EVC EVC Mode for the Virtual Machine
      Activated Activated Enabled. The EVC CPU Mode column displays the EVC mode of the virtual machine.
      Deactivated Activated Enabled. The EVC CPU Mode column displays the EVC mode of the EVC cluster.
    • When the virtual machine is powered off, the EVC CPU Mode column displays the per-VM EVC mode. If per-VM EVC is deactivated, the EVC CPU Mode column for the virtual machine is empty.
    When the virtual machine is not in an EVC cluster and per-VM EVC is not configured, the EVC mode that you see in the EVC CPU Mode column is defined in the following manner.
    • When the virtual machine is powered on, the EVC CPU Mode column displays the EVC mode of the parent host.
    • When the virtual machine is powered off, the EVC CPU Mode column is empty.
  4. To verify the status of the graphics mode, check the EVC Graphics Mode (vSGA) column.
    1. If the column is not visible, click Manage Columns at the bottom left corner of the Virtual Machines table.
      The Show Columns pop-up appears.
    2. To show the EVC Graphics Mode (vSGA) column, locate and select the respective check box from the list.
    The EVC Graphics Mode (vSGA) column displays the baseline graphics features set. To view the baseline graphics, you must enable 3D graphics in the virtual machine.

    For information about configuring 3D graphics in a virtual machine, see Configure 3D Graphics and Video Cards in the vSphere Virtual Machine Administration documentation.

vMotion Migration and ESXi Hosts with AMD Processors Without 3DNow!

If hosts in a vSphere cluster have different generations of AMD processors, some with 3DNow! instruction sets and some without, you cannot successfully migrate virtual machines between the hosts. You must use an EVC mode or CPU compatibility mask to hide the instructions.

Newer generations of AMD processors do not include 3DNow! processor instructions. The vCenter Server AMD Opteron Gen. 3 (no 3DNow!) EVC mode masks the 3DNow! instructions from virtual machines. You can apply this EVC mode to EVC clusters containing only AMD Opteron Generation 3 hosts. Applying this mode allows the clusters to maintain vMotion compatibility with AMD Opteron hosts that do not have 3DNow! instructions. Clusters containing AMD Opteron Generation 1 or AMD Opteron Generation 2 hosts cannot be made vMotion-compatible with hosts that do not have 3DNow! instructions.

Prerequisites

Ensure that the cluster contains only hosts with AMD Opteron Generation 3 or newer processors.

Procedure

  • ♦ Enable the AMD Opteron Gen. 3 (no 3DNow!) EVC mode for your EVC cluster.
    The steps to enable the EVC mode differ depending on whether you are creating a cluster or enabling the mode on an existing cluster, and on whether the existing cluster contains powered-on virtual machines.
    Option Description
    Creating a cluster In the New Cluster wizard, enable EVC for AMD hosts and select the AMD Opteron Gen. 3 (no 3DNow!) EVC mode.
    Editing a cluster without powered-on virtual machines In the Cluster Settings dialog box, edit the VMware EVC settings and select the AMD Opteron Gen. 3 (no 3DNow!) EVC mode.
    Editing a cluster with powered-on virtual machines The AMD Opteron Gen. 3 (no 3DNow!) EVC mode cannot be enabled while there are powered-on virtual machines in the cluster.
    1. Power-off any running virtual machines in the cluster, or migrate them out of the cluster using vMotion.

      Migrating the virtual machines out of the cluster with vMotion allows you to delay powering off the virtual machines until a more convenient time.

    2. In the Cluster Settings dialog box, edit the VMware EVC settings and select the AMD Opteron Gen. 3 (no 3DNow!) EVC mode.
    3. If you migrated virtual machines out of the cluster, power them off and cold migrate them back into the cluster.
    4. Power on the virtual machines.

Results

You can now add hosts with AMD processors without 3DNow! instructions to the cluster and preserve vMotion compatibility between the new hosts and the existing hosts in the cluster.

How to View Feature Capabilities Details for Your EVC Cluster

The feature set that is exposed by an EVC cluster corresponds to the feature set of a particular type of processor. Processor feature sets are described by a set of feature flags that you examine using the CPUID instruction.

You can view the CPUID feature flags currently exposed by the hosts in an EVC cluster.

Procedure

  1. Select a cluster in the inventory.
  2. Click the Configure tab.
  3. Select VMware EVC and expand Current Feature Capabilities Details.

Results

This VMware EVC panel displays the CPUID feature flags that EVC enforces for the hosts in this cluster. For information about CPUID feature flags, see the Intel and AMD websites.