Release Date: NOV 23, 2021

Build Details

Download Filename:	ESXi670-202111001.zip
Build:	18828794
Download Size:	478.2 MB
md5sum:	faa7493ac44f766c307bfa1a92f2bc03
sha256checksum:	92fb8d22d012c34195a39e6ccca2f123dfaf6db64fafcead131e9d71ccb2b146
Host Reboot Required:	Yes
Virtual Machine Migration or Shutdown Required:	Yes

Bulletins

Bulletin ID	Category	Severity
ESXi670-202111401-BG	Bugfix	Critical
ESXi670-202111402-BG	Bugfix	Important
ESXi670-202111403-BG	Bugfix	Important
ESXi670-202111404-BG	Bugfix	Important
ESXi670-202111405-BG	Bugfix	Important
ESXi670-202111101-SG	Security	Important
ESXi670-202111102-SG	Security	Important
ESXi670-202111103-SG	Security	Important
ESXi670-202111104-SG	Security	Important

Rollup Bulletin

This rollup bulletin contains the latest VIBs with all the fixes since the initial release of ESXi 6.7.

Bulletin ID	Category	Severity
ESXi670-202111001	Bugfix	Critical

IMPORTANT: For clusters using VMware vSAN, you must first upgrade the vCenter Server system. Upgrading only the ESXi hosts is not supported.
Before an upgrade, always verify in the VMware Product Interoperability Matrix compatible upgrade paths from earlier versions of ESXi, vCenter Server and vSAN to the current version. 

Image Profiles

VMware patch and update releases contain general and critical image profiles. Application of the general release image profile applies to new bug fixes.

Image Profile Name

ESXi-6.7.0-20211104001-standard

ESXi-6.7.0-20211104001-no-tools

ESXi-6.7.0-20211101001s-standard

ESXi-6.7.0-20211101001s-no-tools

For more information about the individual bulletins, see the Product Patches page and the Resolved Issues section.

Patch Download and Installation

The typical way to apply patches to ESXi hosts is by using the VMware vSphere Update Manager. For details, see the About Installing and Administering VMware vSphere Update Manager.
ESXi hosts can be updated by manually downloading the patch ZIP file from VMware Customer Connect. From the Select a Product drop-down menu, select ESXi (Embedded and Installable) and from the Select a Version drop-down menu, select 6.7.0. Install VIBs by using the esxcli software vib update command. Additionally, you can update the system by using the image profile and the esxcli software profile update command.

For more information, see the vSphere Command-Line Interface Concepts and Examples and the vSphere Upgrade Guide.

 

Resolved Issues

The resolved issues are grouped as follows.

• ESXi670-202111401-BG
• ESXi670-202111402-BG
• ESXi670-202111403-BG
• ESXi670-202111404-BG
• ESXi670-202111405-BG
• ESXi670-202111101-SG
• ESXi670-202111102-SG
• ESXi670-202111103-SG
• ESXi670-202111104-SG
• ESXi-6.7.0-20211104001-standard
• ESXi-6.7.0-20211104001-no-tools
• ESXi-6.7.0-20211101001s-standard
• ESXi-6.7.0-20211101001s-no-tools

ESXi670-202111401-BG

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_vsan_6.7.0-3.159.18811818 VMware_bootbank_esx-base_6.7.0-3.159.18828794 VMware_bootbank_esx-update_6.7.0-3.159.18828794 VMware_bootbank_vsanhealth_6.7.0-3.159.18811819
PRs Fixed	2731317, 2727062, 2704831, 2764894, 2720189, 2761940, 2757263, 2752542, 2772048, 2768356, 2741112, 2786864, 2725025, 2776791, 2724166, 2790353, 2787866, 2790502, 2834968, 2832090, 2841163, 2804742, 2732482, 2793380, 2813328, 2792504, 2749902, 2765136, 2697256, 2731139, 2733489, 2755232, 2761131, 2763933, 2766401, 2744211, 2771730, 2800655, 2839716, 2847812, 2726678, 2789302, 2750034, 2739223, 2794562, 2857511, 2847091, 2765679, 2731643, 2774667, 2778008, 2833428, 2835990, 2852099
CVE numbers	N/A

Updates esx-base, esx-update, vsan, and vsanhealth VIBs to resolve the following issues:

• PR 2731317: During an USB passthrough, virtual machines might become unresponsive

  If the USB device attached to an ESXi host has descriptors that are not compliant with the standard USB specifications, the virtual USB stack might fail to pass through a USB device into a virtual machine. As a result, virtual machines become unresponsive, and you must either power off the virtual machine by using an ESXCLI command, or restart the ESXi host.

  This issue is resolved in this release.

• PR 2727062: Editing an advanced options parameter in a host profile and setting a value to false, results in setting the value to true

  When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  This issue is resolved in this release.

• PR 2704831: After vSphere Replication appliance powers on, ESXi hosts intermittently lose connectivity to vCenter Server

  A rare race condition with static map initialization might cause ESXi hosts to temporarily lose connectivity to vCenter Server after the vSphere Replication appliance powers on. However, the hostd service automatically restarts and the ESXi hosts restore connectivity.

  This issue is resolved in this release.

• PR 2764894: The hostd service might fail and ESXi hosts lose connectivity to vCenter Server due to an empty or unset property of a Virtual Infrastructure Management (VIM) API array

  In rare cases, an empty or unset property of a VIM API data array might cause the hostd service to fail. As a result, ESXi hosts lose connectivity to vCenter Server and you must manually reconnect the hosts.

  This issue is resolved in this release.

• PR 2720189: ESXi hosts intermittently fail with a BSVMM_Validate@vmkernel error in the backtrace

  A rare error condition in the VMKernel might cause ESXi hosts to fail when powering on a virtual machine with more than 1 virtual CPU. A BSVMM_Validate@vmkernel error is indicative for the problem.

  This issue is resolved in this release.

• PR 2761940: An ESXi host might fail with a purple diagnostic screen due to locks during a vSphere vMotion operation

  During a vSphere vMotion operation, if the calling context holds any locks, ESXi hosts might fail with a purple diagnostic screen and an error such as PSOD: Panic at bora/vmkernel/core/lock.c:2070.

  This issue is resolved in this release. 

• PR 2757263: ESXi hosts fail with a purple diagnostic screen with a kernel context error, but not in the correct context

  ESXi hosts might intermittently fail with a purple diagnostic screen with an error such as @BlueScreen: VERIFY bora/vmkernel/sched/cpusched.c that suggests a preemption anomaly. However, the VMkernel preemption anomaly detection logic might fail to identify the correct kernel context and show a warning for the wrong context.

  This issue is resolved in this release. The fix covers corner cases that might cause a preemption anomaly, such as when a system world exits while holding a spinlock. The preemption anomaly detection warning remains @BlueScreen: VERIFY bora/vmkernel/sched/cpusched.c, but in the correct context.

• PR 2752542: If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail

  If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  This issue is resolved in this release.

• PR 2772048: ESXi hosts might fail with a purple diagnostic screen due to a memory allocation failure in the vmkapi character device driver

  Some poll requests might exceed the metadata heap of the vmkapi character device driver and cause ESXi hosts to fail with a purple diagnostic screen and an error such as:
  #PF Exception 14 in world 2099138:IPMI Response IP 0x41800b922ab0 addr 0x18 PTEs:0x0;
  In the VMkernel logs, you might see messages such as:
  WARNING: Heap: 3571: Heap VMKAPI-char-metadata already at its maximum size. Cannot expand.
  and
  WARNING: Heap: 4079: Heap_Align(VMKAPI-char-metadata, 40/40 bytes, 8 align) failed. caller: 0x418029922489

  This issue is resolved in this release.

• PR 2741112: Failure of some Intel CPUs to forward Debug Exception (#DB) traps might result in virtual machine triple fault

  In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault. In the vmware.log file, you see an error such as msg.monitorEvent.tripleFault. This issue is not consistent and virtual machines that consistently triple fault or triple fault after boot are impacted by another issue.

  This issue is resolved in this release. The fix forwards all #DB traps from CPUs into the guest operating system, except when the DB trap comes from a debugger that is attached to the virtual machine.

• PR 2786864: An ESXi host fails with a purple diagnostic screen due to a call stack corruption in the ESXi storage stack

  While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  This issue is resolved in this release.

• PR 2725025: You see alarms that a NIC link is down even after the uplink is restored

  If you plug in and out a physical NIC in your vCenter Sever system, after the uplink is restored, you still see an alarm in the vSphere Client or the vSphere Web Client that a NIC link on some ESXi hosts is down. The VOBD daemon might not create the event esx.clear.net.redundancy.restored to remove such alarms, which causes the issue.

  This issue is resolved in this release.

• PR 2776791: ESXi installation by using a ks.cfg installation script and a USB or SD booting device might fail

  In some cases, when multiple USB or SD devices with different file systems are connected to your vCenter Server system, ESXi installation by using a ks.cfg installation script and a USB or SD booting device might fail.

  This issue is resolved in this release. The fix logs an exception and allows the installation to continue, instead of failing.

• PR 2724166: You might see higher than configured rate of automatic unmap operations on a newly mounted VMFS6 volumes

  In the first minutes after mounting a VMFS6 volume, you might see higher than configured rate of automatic unmap operations. For example, if you set the unmap bandwidth to 2000 MB/s, you see unmap operations running at more than 3000 MB/s.

  This issue is resolved in this release.

• PR 2790353: Old data on an independent nonpersistent mode virtual disk remains after a virtual machine reset

  For independent nonpersistent mode virtual disks, all writes are stored in a redo log, which is a temporary file with extension .REDO_XXXXXX in the VM directory. However, a reset of a virtual machine does not clear the old redo log on such disks.

  This issue is resolved in this release.

• PR 2787866: A replay of a pre-recorded support bundle by using the esxtop utility might fail with a segmentation fault

  When you run the esxtop utility by using the ESXi Shell to replay a pre-recorded support bundle, the operation might fail with a segmentation fault.

  This issue is resolved in this release. The fix makes sure all local variables of the esxtop utility are initialized properly.

• PR 2790502: Shutdown or reboot of ESXi hosts takes very long

  Rarely, in certain configurations, the shutdown or reboot of ESXi hosts might stop at the step Shutting down device drivers for a long time, in the order of 20 minutes, but the operation eventually completes.

  This issue is resolved in this release.

• PR 2834968: Virtual machines fail with a message for too many sticky pages

  The virtual network adapter VMXNET Generation 3 (VMXNET3) uses buffers to process rx packets. Such buffers are either pre-pinned or pinned and mapped during runtime. In rare occasions, some buffers might not un-pin and get re-pinned later, resulting in a higher-than-expected pin count of such a buffer. As a result, virtual machines might fail with an error such as MONITOR PANIC: vmk: vcpu-0:Too many sticky pages..

  This issue is resolved in this release.

• PR 2832090: ESXi hosts might fail with a purple diagnostic screen when vSphere Replication is enabled due to a rare lock rank violation

  Due to a rare lock rank violation in the vSphere Replication I/O filter, some ESXi hosts might fail with a purple diagnostic screen when vSphere Replication is enabled. You see an error such as VERIFY bora/vmkernel/main/bh.c:978 on the screen.

  This issue is resolved in this release.

• PR 2841163: You see packet drops for virtual machines with VMware Network Extensibility (NetX) redirection enabled

  In vCenter Server advanced performance charts, you see an increasing number of packet drop count for all virtual machines that have NetX redirection enabled. However, if you disable NetX redirection, the count becomes 0.

  This issue is resolved in this release.

• PR 2804742: A virtual switch might drop IGMPv3 SSM specific queries during snooping

  Virtual switches processing IGMPv3 SSM specific queries might drop some queries when an IP is not included in the list of available ports as a result of the source IP check. 

  This issue is resolved in this release. The fix makes sure a group specific query is not processed like a normal multicast group and does not check the source IP. 

• PR 2732482: You see higher than usual scrub activity on vSAN

  An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  This issue is resolved in this release.

• PR 2793380: ESXi hosts become unresponsive when you power on a virtual machine

  In particular circumstances, when you power on a virtual machine with a corrupted VMware Tools manifest file, the hostd service might fail. As a result, the ESXi host becomes unresponsive. You can backtrace the issue in the hostd dump file that is usually generated in such cases at the time a VM is powered on.

  This issue is resolved in this release.

• PR 2813328: Encrypted virtual machines do not auto power on even when the Start delay option is configured

  After a reboot of an ESXi host, encrypted virtual machines might not auto power on even when Autostart is configured with the Start delay option to set a specific start time of the host. The issue affects only encrypted virtual machines due to a delay in the distribution of keys from standard key providers.

  This issue is resolved in this release. The fix makes sure that the Start delay option for encrypted VMs considers the delay for getting a key after an ESXi host reboot. 

• PR 2792504: Disabling the SLP service might cause failures during operations with host profiles

  When the SLP service is disabled to prevent potential security vulnerabilities, the sfcbd-watchdog service might remain enabled and cause compliance check failures when you perform updates by using a host profile.

  This issue is resolved in this release.

• PR 2749902: vSphere vMotion operations on ESXi 6.7.x hosts with more than 120 virtual machines might get slow or occasionally fail

  When an ESXi 6.7.x host has more than 120 VMs on it, vSphere vMotion operations to that host or from that host might become slow or occasionally fail.

  This issue is resolved in this release.

• PR 2765136: vSphere Client might not meet all current browser security standards

  In certain environments, vSphere Client might not fully protect the underlying websites through modern browser security mechanisms.

  This issue is resolved in this release.

• PR 2697256: ESXi updates and upgrades might fail on Mac Pro servers

  ESXi updates and upgrades might fail on Mac Pro servers with a purple diagnostic screen and a message such as PSOD - NOT IMPLEMENTED bora/vmkernel/hardware/pci/bridge.c:372 PCPU0:2097152/bootstrap. The issue occurs when Mac Pro BIOS does not assign resources to some devices.

  This issue is resolved in this release.

• PR 2731139: Packets with "IPv6 Tunnel Encapsulation Limit option" are lost in traffic between virtual machines with ENS enabled

  Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines with ENS enabled, because of the IPv6 extension header.

  This issue is resolved in this release. The fix is to correctly parse packets with the IPv6 Tunnel Encapsulation Limit option enabled.

• PR 2733489: Taking a snapshot of a large virtual machine on a large VMFS6 datastore might take long

  Resource allocation for a delta disk to create a snapshot of a large virtual machine, with virtual disks equal to or exceeding 1TB, on large VMFS6 datastores of 30TB or more, might take significant time. As a result, the virtual machine might temporarily lose connectivity. The issue affects primarily VMFS6 filesystems.

  This issue is resolved in this release.

• PR 2755232: An ESXi host might not discover devices configured with the VMW_SATP_INV plug-in

  In case of temporary connectivity issues, ESXi hosts might not discover devices configured with the VMW_SATP_INV plug-in after connectivity restores, because SCSI commands that fail during the device discovery stage cause an out-of-memory condition for the plug-in.

  This issue is resolved in this release.

• PR 2761131: If you import virtual machines with a virtual USB device to a vCenter Server system, the virtual machines might fail to power on

  If you import a virtual machine with a virtual USB device that is not supported by vCenter Server, such as a virtual USB camera, to a vCenter Server system, the VM might fail to power on. The start of such virtual machines fails with an error message similar to: PANIC: Unexpected signal: 11. The issue affects mostly VM import operations from VMware Fusion or VMware Workstation systems, which support a wide range of virtual USB devices.

  This issue is resolved in this release. The fix ignores any unsupported virtual USB devices in virtual machines imported to a vCenter Server system.

• PR 2763933: In the vmkernel.log, you see an error for an unexpected result while waiting for pointer block cluster (PBC) to clear

  A non-blocking I/O operation might block a PBC file operation in a VMFS volume. In the vmkernel.log, you see an error such as:
  cpu2:2098100)WARNING: PB3: 5020: Unexpected result (Would block) while waiting for PBC to clear -FD c63 r24
  However, this is expected behavior.

  This issue is resolved in this release. The fix removes the error message in cases when an I/O operation blocks a PBC file operation.

• PR 2766401: When an overlay tunnel is configured from a guest VM with different default VXLAN port than 4789, vmxnet3 might drop packets

  By default, the vmxnet3 driver uses port 4789 when a VM uses ENS and port 8472 when ENS is not enabled. As a result, when an overlay tunnel is configured from a guest VM with different default VXLAN port, vmxnet3 might drop packets.

  This issue is resolved in this release. The fix makes sure vmxnet3 delivers packets when a VM uses either of ports 4789 or 8472. However, the fix works only on VMs with version ESXi 6.7 and later, hardware versions later than 14, and if the overlay tunnel is configured from a guest VM.

• PR 2744211: A static MAC cannot be learned on another VNIC port across a different VLAN

  In some cases, MAC learning does not work as expected and affects some virtual machine operations. For example, cloning a VM with the same MAC address on a different VLAN causes a traffic flood to the cloned VM in the ESXi host where the original VM is present.

  This issue is resolved in this release. The fix makes sure that a static MAC can be learned on another port, on a different VLAN. For example, a static MAC x on VNIC port p1 with on VLAN 1, can be learned on VNIC port p2, VLAN 2.

• PR 2771730: You cannot create a vCenter Host Profiles CIM Indication Subscription due to an out of memory error

  Attempts to create a vCenter Host Profiles CIM Indication Subscription might fail due to a cim-xml parsing request that returns an out of memory error.

  This issue is resolved in this release.

• PR 2800655: You might see packet drops on uplink on heavy traffic

  The packet scheduler that controls network I/O in a vCenter Server system has a fixed length queue that might quickly fill up. As a result, you might see packet drops on uplink on heavy traffic.

  This issue is resolved in this release. The fix makes the Network I/O Control queue size dynamic to allow expansion under certain conditions.

• PR 2839716: You cannot add Intel Ice Lake servers to a Skylake cluster with enabled Enhanced vMotion Compatibility (EVC) mode

  In a vSphere 6.7.x system, you cannot add an Intel Ice Lake server to a Skylake cluster with enabled EVC mode. In the vSphere Web Client or the vSphere Client, you see the following message:
  Host CPU lacks features required by that mode. XSAVE of BND0-BND3 bounds registers (BNDREGS) is unsupported. XSAVE of BNDCFGU and BNDSTATUS registers (BNDCSR) is unsupported.

  This issue is resolved in this release. You can update your ESXi hosts to ESXi 670-202111001, or upgrade to ESXi 7.0.x.

• PR 2847812: Active Directory latency issues might impact performance across the entire vCenter Server system

  The VIM API server deletes the Active Directory cache on certain intervals to check the validity of permissions of Active Directory domain accounts. After the cache delete operation, daily by default, the VIM API server re-populates the cache. However, the cache re-populate operation might be slow on domains with many accounts. As a result, such operations might impact performance across the entire vCenter Server system.

  This issue is resolved in this release. The fix skips the step of re-populating the cache.

• PR 2726678: ESXi hosts might become unresponsive after an abrupt power cycling or reboot

  An abrupt power cycling or reboot of an ESX host might cause a race between subsequent journal replays because other hosts might try to access the same resources. If a race condition happens, the journal replay cannot complete. As a result, virtual machines on a VMFS6 datastore become inaccessible or any operations running on the VMs fail or stop.

  This issue is resolved in this release. The fix enhances local and remote journal replay for VMFS6.

• PR 2789302: Multiple cron instances run after using the reboot_helper.py script to shut down a vSAN cluster

  If you manually run the reboot_helper.py script to shut down a vSAN cluster, the process cleans unused cron jobs registered during the shutdown. However, the script might start a new cron instance instead of restarting the cron daemon after cleaning the unused cron jobs. As a result, multiple cron instances might accumulate and you see cron jobs executed multiple times.

  This issue is resolved in this release.

• PR 2750034: The hostd service might fail with a “Memory exceeds hard limit” error due to a memory leak

  If you run a recursive call on a large datastore, such as DatastoreBrowser::SearchSubFolders, a memory leak condition might cause the hostd service to fail with an out-of-memory error.

  This issue is resolved in this release. The fix prevents the memory leak condition.

• PR 2739223: vSAN health reports certificate errors in the configured KMIP provider

  When you enable encryption for a vSAN cluster with a Key Management Interoperability Protocol (KMIP) provider, the following health check might report status errors: vCenter KMS status.

  This issue is resolved in this release.

• PR 2794562: The recommended size of the core dump partition for an all-flash vSAN cluster might be larger than required

  For all-flash vSAN configurations, you might see larger than required recommended size for the core dump partition. As a result, the core dump configuration check might fail.

  This issue is resolved in this release.

• PR 2857511: Virtual machines appear as inaccessible in the vSphere Client and you might see some downtime for applications

  In rare cases, hardware issues might cause an SQlite DB corruption that makes multiple VMs become inaccessible and lead to some downtime for applications.

  This issue is resolved in this release.

• PR 2847091: Hosts in a vSAN cluster might fail due to a memory block attributes issue

  When a vSAN host has memory pressure, adding disks to a disk group might cause failure of the memory block attributes (blkAttr) component initialization. Without the blkAttr component, commit and flush tasks stop, which causes log entries to build up in the SSD and eventually cause а congestion. As a result, a NMI failure might occur due to the CPU load for processing the large number of log entries.

  This issue is resolved in this release.

• PR 2765679: Cannot add hosts to a vSAN cluster with large cluster support enabled

  In some cases, the host designated as the leader cannot send heartbeat messages to other hosts in a large vSAN cluster. This issue occurs when the leader uses an insufficient size of the TX buffer. The result is that new hosts cannot join the cluster with large cluster support (up to 64 hosts) enabled.

  This issue is resolved in this release.

• PR 2731643: You see memory congestion on vSAN cluster hosts with large number of idle VMs

  On a vSAN cluster with a large number of idle virtual machines, vSAN hosts might experience memory congestion due to higher scrub frequency. 

  This issue is resolved in this release. 

• PR 2774667: vSAN automatic rebalance does not occur even after capacity reaches 80%

  An issue with trim space operations count might cause a large number of pending delete operations in vSAN. If an automatic rebalancing operation triggers, it cannot progress due to such pending delete operations. For example, if you set the rebalancing threshold at 30%, you do not see any  progress on rebalancing even when the storage capacity reaches 80%.

  This issue is resolved in this release.

• PR 2778008: ESXi hosts might fail with a purple diagnostic screen due to a lock spinout on a virtual machine configured to use persistent Memory (PMem) with large number of vCPUs

  Due to a dependency between the management of virtual RAM and virtual NVDIMM within a virtual machine, excessive access to the virtual NVDIMM device might lead to a lock spinout while accessing virtual RAM and cause the ESXi host to fail with a purple diagnostic screen. In the backtrace, you see the following:
  SP_WaitLock
SPLockWork
AsyncRemapPrepareRemapListVM
AsyncRemap_AddOrRemapVM
LPageSelectLPageToDefrag
VmAssistantProcessTasks

  This issue is resolved in this release.

• PR 2833428: ESXi hosts with virtual machines with Latency Sensitivity enabled might randomly become unresponsive due to CPU starvation

  When you enable Latency Sensitivity on virtual machines, some threads of the Likewise Service Manager (lwsmd), which sets CPU affinity explicitly, might compete for CPU resources on such virtual machines. As a result, you might see the ESXi host and the hostd service to become unresponsive.

  This issue is resolved in this release. The fix makes sure lwsmd does not set CPU affinity explicitly.

• PR 2835990: Adding ESXi hosts to an Active Directory domain might take long

  Some LDAP queries that have no specified timeouts might cause a significant delay in domain join operations for adding an ESXi host to an Active Directory domain.

  This issue is resolved in this release. The fix adds a 15 seconds standard timeout with additional logging around the LDAP calls during domain join workflow.

• PR 2852099: When vSphere Replication is enabled on a virtual machine, many other VMs might become unresponsive

  When vSphere Replication is enabled on a virtual machine, you might see higher datastore and in-guest latencies that in certain cases might lead to ESXi hosts becoming unresponsive to vCenter Server. The increased latency comes from vSphere Replication computing MD5 checksums on the I/O completion path, which delays all other I/Os.

  This issue is resolved in this release. The fix offloads the vSphere Replication MD5 calculation from the I/O completion path to a work pool and reduces the amount of outstanding I/O that vSphere Replication issues.

ESXi670-202111402-BG

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794
PRs Fixed	2761169
CVE numbers	N/A

Updates the vmkusb VIB to resolve the following issue:

• PR 2761169: Connection to the /bootbank partition intermittently breaks when you use USB or SD devices

  ESXi670-202111001 adds two fixes to the USB host controller for cases when a USB or SD device lose connectivity to the /bootbank partition:

  1. Adds a parameter usbStorageRegisterDelaySecs.
     The usbStorageRegisterDelaySecs parameter is part of the vmkusb module and it delays to register cached USB storage device within the limit of 0 to 600 seconds. The default delay is 10 seconds. The parametric delay makes sure that if the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the USB or SD device does not get a new path that might break connection to the /bootbank partition or corrupt the VMFS-L LOCKER partition.
  2. Enhances the USB host controller driver to tolerate command timeout failures.
     In cases when a device, such as Dell IDSDM, reconnects by a hardware signal, or a USB device resets due to a heavy workload, the USB host controller driver might make only one retry to restore connectivity. With the fix, the controller makes multiple retries and extends tolerance to timeout failures.

  This issue is resolved in this release. As a best practice, do not set dump partition on USB storage device and do not set USB devices under a heavy workload. For more information, see VMware knowledge base articles 2077516 and 2149257.

ESXi670-202111403-BG

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794
PRs Fixed	2555789
CVE numbers	N/A

Updates the nvme VIB.

ESXi670-202111404-BG

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_brcmfcoe_11.4.1078.26-14vmw.670.3.159.18828794
PRs Fixed	2775375
CVE numbers	N/A

Updates the brcmfcoe VIB to resolve the following issue:

• PR 2775375: ESXi hosts might lose connectivity after brcmfcoe driver upgrade on Hitachi storage arrays

  After an upgrade of the brcmfcoe driver on Hitachi storage arrays, ESXi hosts might fail to boot and lose connectivity.

  This issue is resolved in this release.

ESXi670-202111405-BG

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	No
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794
PRs Fixed	2820768
CVE numbers	N/A

Updates the lsu-smartpqi-plugin​ VIB.

• PR 2820768: OEM smartpqi drivers of version 1.0.3.2323 and earlier fail to blink LEDs or get the physical locations on logical drives

  The OEM smartpqi driver in ESXi 670-202111001 uses a new logical LUN ID assignment rule to become compatible with the drivers of versions later than 1.0.3.2323. As a result, OEM smartpqi drivers of version 1.0.3.2323 and earlier might fail to blink LEDs or get the physical locations on logical drives.

  This issue is resolved in this release. However, if you use an older version of the OEM smartpqi driver, upgrade to a version later than 1.0.3.2323 to avoid the issue.

ESXi670-202111101-SG

Patch Category	Security
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_vsanhealth_6.7.0-3.155.18811783 VMware_bootbank_vsan_6.7.0-3.155.18811780 VMware_bootbank_esx-update_6.7.0-3.155.18812553 VMware_bootbank_esx-base_6.7.0-3.155.18812553
PRs Fixed	2704746, 2765136, 2794798, 2794804, 2794933, 2794943, 2795965, 2797008, 2830222, 2830671
CVE numbers	CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029, CVE-2021-28831

Updates esx-base, esx-update, vsan, and vsanhealth VIBs to resolve the following issues:

• Update of the SQLite database

  The SQLite database is updated to version 3.34.1.

• Update to OpenSSL

  The OpenSSL package is updated to version openssl-1.0.2za.

• Update to cURL

  The cURL library is updated to 7.78.0.

• Update to the OpenSSH

  The OpenSSH is updated to version 8.6p1.

• Update to the libarchive library

  The libarchive library is updated to libarchive - 3.5.1.

• Update to the BusyBox package

  The Busybox package is updated to address CVE-2021-28831.

• Update to the GNU C Library (glibc)

  The glibc library is updated tо address the following CVEs: CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029.

ESXi670-202111102-SG

Patch Category	Security
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_vmkusb_0.1-1vmw.670.3.155.18812553
PRs Fixed	2762034
CVE numbers	N/A

Updates the vmkusb VIB.

ESXi670-202111103-SG

Patch Category	Security
Patch Severity	Important
Host Reboot Required	No
Virtual Machine Migration or Shutdown Required	No
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_locker_tools-light_11.3.5.18557794-18812553
PRs Fixed	2762034, 2785530, 2816541
Related CVE numbers	N/A

This patch updates the tools-light VIB.

• The following VMware Tools ISO images are bundled with ESXi 670-202111001:

  • windows.iso: VMware Tools 11.3.5 supports Windows 7 SP1 or Windows Server 2008 R2 SP1 and later.
  • linux.iso: VMware Tools 10.3.23 ISO image for Linux OS with glibc 2.11 or later.

  The following VMware Tools ISO images are available for download:

  • VMware Tools 11.0.6:
    • windows.iso: for Windows Vista (SP2) and Windows Server 2008 Service Pack 2 (SP2).
       
  • VMware Tools 10.0.12:
    • winPreVista.iso: for Windows 2000, Windows XP, and Windows 2003.
    • linuxPreGLibc25.iso: supports Linux guest operating systems earlier than Red Hat Enterprise Linux (RHEL) 5, SUSE Linux Enterprise Server (SLES) 11, Ubuntu 7.04, and other distributions with glibc version earlier than 2.5.
       
  • solaris.iso: VMware Tools image 10.3.10 for Solaris.
  • darwin.iso: Supports Mac OS X versions 10.11 and later.

  Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

  • VMware Tools 11.3.5 Release Notes
  • Earlier versions of VMware Tools
  • What Every vSphere Admin Must Know About VMware Tools
  • VMware Tools for hosts provisioned with Auto Deploy
  • Updating VMware Tools

ESXi670-202111104-SG

Patch Category	Security
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_cpu-microcode_6.7.0-3.155.18812553
PRs Fixed	2776028
Related CVE numbers	N/A

This patch updates the cpu-microcode VIB.

• The cpu-microcode VIB includes the following Intel microcode:

  Code Name	FMS	Plt ID	MCU Rev	MCU Date	Brand Names
  Clarkdale	0x20652	0x12	0x00000011	5/8/2018	Intel i3/i5 Clarkdale Series; Intel Xeon 34xx Clarkdale Series
  Arrandale	0x20655	0x92	0x00000007	4/23/2018	Intel Core i7-620LE Processor
  Sandy Bridge DT	0x206a7	0x12	0x0000002f	2/17/2019	Intel Xeon E3-1100 Series; Intel Xeon E3-1200 Series; Intel i7-2655-LE Series; Intel i3-2100 Series
  Westmere EP	0x206c2	0x03	0x0000001f	5/8/2018	Intel Xeon 56xx Series; Intel Xeon 36xx Series
  Sandy Bridge EP	0x206d6	0x6d	0x00000621	3/4/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
  Sandy Bridge EP	0x206d7	0x6d	0x0000071a	3/24/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
  Westmere EX	0x206f2	0x05	0x0000003b	5/16/2018	Intel Xeon E7-8800 Series; Intel Xeon E7-4800 Series; Intel Xeon E7-2800 Series
  Ivy Bridge DT	0x306a9	0x12	0x00000021	2/13/2019	Intel i3-3200 Series; Intel i7-3500-LE/UE; Intel i7-3600-QE; Intel Xeon E3-1200-v2 Series; Intel Xeon E3-1100-C-v2 Series; Intel Pentium B925C
  Haswell DT	0x306c3	0x32	0x00000028	11/12/2019	Intel Xeon E3-1200-v3 Series; Intel i7-4700-EQ Series; Intel i5-4500-TE Series; Intel i3-4300 Series
  Ivy Bridge EP	0x306e4	0xed	0x0000042e	3/14/2019	Intel Xeon E5-4600-v2 Series; Intel Xeon E5-2600-v2 Series; Intel Xeon E5-2400-v2 Series; Intel Xeon E5-1600-v2 Series; Intel Xeon E5-1400-v2 Series
  Ivy Bridge EX	0x306e7	0xed	0x00000715	3/14/2019	Intel Xeon E7-8800/4800/2800-v2 Series
  Haswell EP	0x306f2	0x6f	0x00000046	1/27/2021	Intel Xeon E5-4600-v3 Series; Intel Xeon E5-2600-v3 Series; Intel Xeon E5-2400-v3 Series; Intel Xeon E5-1600-v3 Series; Intel Xeon E5-1400-v3 Series
  Haswell EX	0x306f4	0x80	0x00000019	2/5/2021	Intel Xeon E7-8800/4800-v3 Series
  Broadwell H	0x40671	0x22	0x00000022	11/12/2019	Intel Core i7-5700EQ; Intel Xeon E3-1200-v4 Series
  Avoton	0x406d8	0x01	0x0000012d	9/16/2019	Intel Atom C2300 Series; Intel Atom C2500 Series; Intel Atom C2700 Series
  Broadwell EP/EX	0x406f1	0xef	0x0b00003e	2/6/2021	Intel Xeon E7-8800/4800-v4 Series; Intel Xeon E5-4600-v4 Series; Intel Xeon E5-2600-v4 Series; Intel Xeon E5-1600-v4 Series
  Skylake SP	0x50654	0xb7	0x02006b06	3/8/2021	Intel Xeon Platinum 8100 Series; Intel Xeon Gold 6100/5100, Silver 4100, Bronze 3100 Series; Intel Xeon D-2100 Series; Intel Xeon D-1600 Series; Intel Xeon W-3100 Series; Intel Xeon W-2100 Series
  Cascade Lake B-0	0x50656	0xbf	0x04003103	4/20/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
  Cascade Lake	0x50657	0xbf	0x05003103	4/8/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
  Cooper Lake	0x5065b	0xbf	0x07002302	4/23/2021	Intel Xeon Platinum 8300 Series; Intel Xeon Gold 6300/5300
  Broadwell DE	0x50662	0x10	0x0000001c	6/17/2019	Intel Xeon D-1500 Series
  Broadwell DE	0x50663	0x10	0x0700001b	2/4/2021	Intel Xeon D-1500 Series
  Broadwell DE	0x50664	0x10	0x0f000019	2/4/2021	Intel Xeon D-1500 Series
  Broadwell NS	0x50665	0x10	0x0e000012	2/4/2021	Intel Xeon D-1600 Series
  Skylake H/S	0x506e3	0x36	0x000000ea	1/25/2021	Intel Xeon E3-1500-v5 Series; Intel Xeon E3-1200-v5 Series
  Denverton	0x506f1	0x01	0x00000034	10/23/2020	Intel Atom C3000 Series
  Ice Lake SP	0x606a6	0x87	0x0d0002a0	4/25/2021	Intel Xeon Silver 4300 Series; Intel Xeon Gold 6300/5300 Series; Intel Xeon Platinum 8300 Series
  Snow Ridge	0x80665	0x01	0x0b00000f	2/17/2021	Intel Atom P5000 Series
  Kaby Lake H/S/X	0x906e9	0x2a	0x000000ea	1/5/2021	Intel Xeon E3-1200-v6 Series; Intel Xeon E3-1500-v6 Series
  Coffee Lake	0x906ea	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series; Intel Xeon E-2200 Series (4 or 6 core)
  Coffee Lake	0x906eb	0x02	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
  Coffee Lake	0x906ec	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
  Coffee Lake Refresh	0x906ed	0x22	0x000000ea	1/5/2021	Intel Xeon E-2200 Series (8 core)
  Rocket Lake S	0xa0671	0x02	0x00000040	4/11/2021	Intel Xeon E-2300 Series

ESXi-6.7.0-20211104001-standard

Profile Name	ESXi-6.7.0-20211104001-standard
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	November 23, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsan_6.7.0-3.159.18811818 VMware_bootbank_esx-base_6.7.0-3.159.18828794 VMware_bootbank_esx-update_6.7.0-3.159.18828794 VMware_bootbank_vsanhealth_6.7.0-3.159.18811819 VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794 VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794 VMW_bootbank_brcmfcoe_11.4.1078.26-14vmw.670.3.159.18828794 VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794
PRs Fixed	2731317, 2727062, 2704831, 2764894, 2720189, 2761940, 2757263, 2752542, 2772048, 2768356, 2741112, 2786864, 2725025, 2776791, 2724166, 2790353, 2787866, 2790502, 2834968, 2832090, 2841163, 2804742, 2732482, 2793380, 2813328, 2792504, 2749902, 2765136, 2697256, 2731139, 2733489, 2755232, 2761131, 2763933, 2766401, 2744211, 2771730, 2800655, 2839716, 2847812, 2726678, 2789302, 2750034, 2739223, 2794562, 2857511, 2847091, 2765679, 2731643, 2774667, 2778008, 2833428, 2761169, 2775375, 2820768, 2835990, 2852099
Related CVE numbers	N/A

• This patch resolves the following issues:

  • If the USB device attached to an ESXi host has descriptors that are not compliant with the standard USB specifications, the virtual USB stack might fail to pass through a USB device into a virtual machine. As a result, virtual machines become unresponsive, and you must either power off the virtual machine by using an ESXCLI command, or restart the ESXi host.

  • When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  • A rare race condition with static map initialization might cause ESXi hosts to temporarily lose connectivity to vCenter Server after the vSphere Replication appliance powers on. However, the hostd service automatically restarts and the ESXi hosts restore connectivity.

  • In rare cases, an empty or unset property of a VIM API data array might cause the hostd service to fail. As a result, ESXi hosts lose connectivity to vCenter Server and you must manually reconnect the hosts.

  • A rare error condition in the VMKernel might cause ESXi hosts to fail when powering on a virtual machine with more than 1 virtual CPU. A BSVMM_Validate@vmkernel error is indicative for the problem.

  • During a vSphere vMotion operation, if the calling context holds any locks, ESXi hosts might fail with a purple diagnostic screen and an error such as PSOD: Panic at bora/vmkernel/core/lock.c:2070.

  • ESXi hosts might intermittently fail with a purple diagnostic screen with an error such as @BlueScreen: VERIFY bora/vmkernel/sched/cpusched.c that suggests a preemption anomaly. However, the VMkernel preemption anomaly detection logic might fail to identify the correct kernel context and show a warning for the wrong context.

  • If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  • Some poll requests might exceed the metadata heap of the vmkapi character device driver and cause ESXi hosts to fail with a purple diagnostic screen and an error such as:
    #PF Exception 14 in world 2099138:IPMI Response IP 0x41800b922ab0 addr 0x18 PTEs:0x0;
    In the VMkernel logs, you might see messages such as:
    WARNING: Heap: 3571: Heap VMKAPI-char-metadata already at its maximum size. Cannot expand.
    and
    WARNING: Heap: 4079: Heap_Align(VMKAPI-char-metadata, 40/40 bytes, 8 align) failed. caller: 0x418029922489

  • In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault. In the vmware.log file, you see an error such as msg.monitorEvent.tripleFault. This issue is not consistent and virtual machines that consistently triple fault or triple fault after boot are impacted by another issue.

  • While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  • If you plug in and out a physical NIC in your vCenter Sever system, after the uplink is restored, you still see an alarm in the vmkernel.log of some ESXi hosts that a NIC link is down. The VOBD daemon might not create the event esx.clear.net.redundancy.restored to remove such alarms, which causes the issue.

  • In some cases, when multiple USB or SD devices with different file systems are connected to your vCenter Server system, ESXi installation by using a ks.cfg installation script and a USB or SD booting device might fail.

  • In the first minutes after mounting a VMFS6 volume, you might see higher than configured rate of automatic unmap operations. For example, if you set the unmap bandwidth to 2000 MB/s, you see unmap operations running at more than 3000 MB/s.

  • For independent nonpersistent mode virtual disks, all writes are stored in a redo log, which is a temporary file with extension .REDO_XXXXXX in the VM directory. However, a reset of a virtual machine does not clear the old redo log on such disks.

  • When you run the esxtop utility by using the ESXi Shell to replay a pre-recorded support bundle, the operation might fail with a segmentation fault.

  • Rarely, in certain configurations, the shutdown or reboot of ESXi hosts might stop at the step Shutting down device drivers for a long time, in the order of 20 minutes, but the operation eventually completes.

  • The virtual network adapter VMXNET Generation 3 (VMXNET3) uses buffers to process rx packets. Such buffers are either pre-pinned or pinned and mapped during runtime. In rare occasions, some buffers might not un-pin and get re-pinned later, resulting in a higher-than-expected pin count of such a buffer. As a result, virtual machines might fail with an error such as MONITOR PANIC: vmk: vcpu-0:Too many sticky pages..

  • Due to a rare lock rank violation in the vSphere Replication I/O filter, some ESXi hosts might fail with a purple diagnostic screen when vSphere Replication is enabled. You see an error such as VERIFY bora/vmkernel/main/bh.c:978 on the screen.

  • In vCenter Server advanced performance charts, you see an increasing number of packet drop count for all virtual machines that have NetX redirection enabled. However, if you disable NetX redirection, the count becomes 0.

  • Virtual switches processing IGMPv3 SSM specific queries might drop some queries when an IP is not included in the list of available ports as a result of the source IP check. 

  • An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  • In particular circumstances, when you power on a virtual machine with a corrupted VMware Tools manifest file, the hostd service might fail. As a result, the ESXi host becomes unresponsive. You can backtrace the issue in the hostd dump file that is usually generated in such cases at the time a VM is powered on.

  • After a reboot of an ESXi host, encrypted virtual machines might not auto power on even when Autostart is configured with the Start delay option to set a specific start time of the host. The issue affects only encrypted virtual machines due to a delay in the distribution of keys from standard key providers.

  • When the SLP service is disabled to prevent potential security vulnerabilities, the sfcbd-watchdog service might remain enabled and cause compliance check failures when you perform updates by using a host profile.

  • When an ESXi 6.7.x host has more than 120 VMs on it, vSphere vMotion operations to that host or from that host might become slow or occasionally fail.

  • In certain environments, vSphere Client might not fully protect the underlying websites through modern browser security mechanisms.

  • ESXi updates and upgrades might fail on Mac Pro servers with a purple diagnostic screen and a message such as PSOD - NOT IMPLEMENTED bora/vmkernel/hardware/pci/bridge.c:372 PCPU0:2097152/bootstrap. The issue occurs when Mac Pro BIOS does not assign resources to some devices.

  • Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines with ENS enabled, because of the IPv6 extension header.

  • Resource allocation for a delta disk to create a snapshot of a large virtual machine, with virtual disks equal to or exceeding 1TB, on large VMFS6 datastores of 30TB or more, might take significant time. As a result, the virtual machine might temporarily lose connectivity. The issue affects primarily VMFS6 filesystems.

  • In case of temporary connectivity issues, ESXi hosts might not discover devices configured with the VMW_SATP_INV plug-in after connectivity restores, because SCSI commands that fail during the device discovery stage cause an out-of-memory condition for the plug-in.

  • If you import a virtual machine with a virtual USB device that is not supported by vCenter Server, such as a virtual USB camera, to a vCenter Server system, the VM might fail to power on. The start of such virtual machines fails with an error message similar to: PANIC: Unexpected signal: 11. The issue affects mostly VM import operations from VMware Fusion or VMware Workstation systems, which support a wide range of virtual USB devices.

  • A non-blocking I/O operation might block a PBC file operation in a VMFS volume. In the vmkernel.log, you see an error such as:
    cpu2:2098100)WARNING: PB3: 5020: Unexpected result (Would block) while waiting for PBC to clear -FD c63 r24
    However, this is expected behavior.

  • By default, the vmxnet3 driver uses port 4789 when a VM uses ENS and port 8472 when ENS is not enabled. As a result, when an overlay tunnel is configured from a guest VM with different default VXLAN port, vmxnet3 might drop packets.

  • In some cases, MAC learning does not work as expected and affects some virtual machine operations. For example, cloning a VM with the same MAC address on a different VLAN causes a traffic flood to the cloned VM in the ESXi host where the original VM is present.

  • Attempts to create a vCenter Host Profiles CIM Indication Subscription might fail due to a cim-xml parsing request that returns an out of memory error.

  • The packet scheduler that controls network I/O in a vCenter Server system has a fixed length queue that might quickly fill up. As a result, you might see packet drops on uplink on heavy traffic.

  • In a vSphere 6.7.x system, you cannot add an Intel Ice Lake server to a Skylake cluster with enabled EVC mode. In the vSphere Web Client or the vSphere Client, you see the following message:
    Host CPU lacks features required by that mode. XSAVE of BND0-BND3 bounds registers (BNDREGS) is unsupported. XSAVE of BNDCFGU and BNDSTATUS registers (BNDCSR) is unsupported.

  • The VIM API server deletes the Active Directory cache on certain intervals to check the validity of permissions of Active Directory domain accounts. After the cache delete operation, daily by default, the VIM API server re-populates the cache. However, the cache re-populate operation might be slow on domains with many accounts. As a result, such operations might impact performance across the entire vCenter Server system.

  • An abrupt power cycling or reboot of an ESX host might cause a race between subsequent journal replays because other hosts might try to access the same resources. If a race condition happens, the journal replay cannot complete. As a result, virtual machines on a VMFS6 datastore become inaccessible or any operations running on the VMs fail or stop.

  • If you manually run the reboot_helper.py script to shut down a vSAN cluster, the process cleans unused cron jobs registered during the shutdown. However, the script might start a new cron instance instead of restarting the cron daemon after cleaning the unused cron jobs. As a result, multiple cron instances might accumulate and you see cron jobs executed multiple times.

  • If you run a recursive call on a large datastore, such as DatastoreBrowser::SearchSubFolders, a memory leak condition might cause the hostd service to fail with an out-of-memory error.

  • When you enable encryption for a vSAN cluster with a Key Management Interoperability Protocol (KMIP) provider, the following health check might report status errors: vCenter KMS status.

  • For all-flash vSAN configurations, you might see larger than required recommended size for the core dump partition. As a result, the core dump configuration check might fail.

  • In rare cases, hardware issues might cause an SQlite DB corruption that makes multiple VMs become inaccessible and lead to some downtime for applications.

  • When a vSAN host has memory pressure, adding disks to a disk group might cause failure of the memory block attributes (blkAttr) component initialization. Without the blkAttr component, commit and flush tasks stop, which causes log entries to build up in the SSD and eventually cause а congestion. As a result, a NMI failure might occur due to the CPU load for processing the large number of log entries.

  • In some cases, the host designated as the leader cannot send heartbeat messages to other hosts in a large vSAN cluster. This issue occurs when the leader uses an insufficient size of the TX buffer. The result is that new hosts cannot join the cluster with large cluster support (up to 64 hosts) enabled.

  • On a vSAN cluster with a large number of idle virtual machines, vSAN hosts might experience memory congestion due to higher scrub frequency. 

  • An issue with trim space operations count might cause a large number of pending delete operations in vSAN. If an automatic rebalancing operation triggers, it cannot progress due to such pending delete operations. For example, if you set the rebalancing threshold at 30%, you do not see any  progress on rebalancing even when the storage capacity reaches 80%.

  • Due to a dependency between the management of virtual RAM and virtual NVDIMM within a virtual machine, excessive access to the virtual NVDIMM device might lead to a lock spinout while accessing virtual RAM and cause the ESXi host to fail with a purple diagnostic screen. In the backtrace, you see the following:
    SP_WaitLock
SPLockWork
AsyncRemapPrepareRemapListVM
AsyncRemap_AddOrRemapVM
LPageSelectLPageToDefrag
VmAssistantProcessTasks

  • When you enable Latency Sensitivity on virtual machines, some threads of the Likewise Service Manager (lwsmd), which sets CPU affinity explicitly, might compete for CPU resources on such virtual machines. As a result, you might see the ESXi host and the hostd service to become unresponsive.

  • ESXi670-202111001 adds two fixes to the USB host controller for cases when a USB or SD device lose connectivity to the /bootbank partition:

    1. Adds a parameter usbStorageRegisterDelaySecs.
       The usbStorageRegisterDelaySecs parameter is part of the vmkusb module and it delays to register cached USB storage device within the limit of 0 to 600 seconds. The default delay is 10 seconds. The parametric delay makes sure that if the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the USB or SD device does not get a new path that might break connection to the /bootbank partition or corrupt the VMFS-L LOCKER partition.
    2. Enhances the USB host controller driver to tolerate command timeout failures.
       In cases when a device, such as Dell IDSDM, reconnects by a hardware signal, or a USB device resets due to a heavy workload, the USB host controller driver might make only one retry to restore connectivity. With the fix, the controller makes multiple retries and extends tolerance to timeout failures.
  • Some LDAP queries that have no specified timeouts might cause a significant delay in domain join operations for adding an ESXi host to an Active Directory domain.

  • After an upgrade of the brcmfcoe driver on Hitachi storage arrays, ESXi hosts might fail to boot and lose connectivity.

  • The OEM smartpqi driver in ESXi 670-202111001 uses a new logical LUN ID assignment rule to become compatible with the drivers of versions later than 1.0.3.2323. As a result, OEM smartpqi drivers of version 1.0.3.2323 and earlier might fail to blink LEDs or get the physical locations on logical drives.

  • When vSphere Replication is enabled on a virtual machine, you might see higher datastore and in-guest latencies that in certain cases might lead to ESXi hosts becoming unresponsive to vCenter Server. The increased latency comes from vSphere Replication computing MD5 checksums on the I/O completion path, which delays all other I/Os.

ESXi-6.7.0-20211104001-no-tools

Profile Name	ESXi-6.7.0-20211104001-no-tools
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	November 23, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsan_6.7.0-3.159.18811818 VMware_bootbank_esx-base_6.7.0-3.159.18828794 VMware_bootbank_esx-update_6.7.0-3.159.18828794 VMware_bootbank_vsanhealth_6.7.0-3.159.18811819 VMW_bootbank_vmkusb_0.1-4vmw.670.3.159.18828794 VMW_bootbank_nvme_1.2.2.28-5vmw.670.3.159.18828794 VMW_bootbank_brcmfcoe_11.4.1078.26-14vmw.670.3.159.18828794 VMware_bootbank_lsu-smartpqi-plugin_1.0.0-4vmw.670.3.159.18828794
PRs Fixed	2731317, 2727062, 2704831, 2764894, 2720189, 2761940, 2757263, 2752542, 2772048, 2768356, 2741112, 2786864, 2725025, 2776791, 2724166, 2790353, 2787866, 2790502, 2834968, 2832090, 2841163, 2804742, 2732482, 2793380, 2813328, 2792504, 2749902, 2765136, 2697256, 2731139, 2733489, 2755232, 2761131, 2763933, 2766401, 2744211, 2771730, 2800655, 2839716, 2847812, 2726678, 2789302, 2750034, 2739223, 2794562, 2857511, 2847091, 2765679, 2731643, 2774667, 2778008, 2833428, 2761169, 2775375, 2820768, 2835990, 2852099
Related CVE numbers	N/A

• This patch resolves the following issues:

  • If the USB device attached to an ESXi host has descriptors that are not compliant with the standard USB specifications, the virtual USB stack might fail to pass through a USB device into a virtual machine. As a result, virtual machines become unresponsive, and you must either power off the virtual machine by using an ESXCLI command, or restart the ESXi host.

  • When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  • A rare race condition with static map initialization might cause ESXi hosts to temporarily lose connectivity to vCenter Server after the vSphere Replication appliance powers on. However, the hostd service automatically restarts and the ESXi hosts restore connectivity.

  • In rare cases, an empty or unset property of a VIM API data array might cause the hostd service to fail. As a result, ESXi hosts lose connectivity to vCenter Server and you must manually reconnect the hosts.

  • A rare error condition in the VMKernel might cause ESXi hosts to fail when powering on a virtual machine with more than 1 virtual CPU. A BSVMM_Validate@vmkernel error is indicative for the problem.

  • During a vSphere vMotion operation, if the calling context holds any locks, ESXi hosts might fail with a purple diagnostic screen and an error such as PSOD: Panic at bora/vmkernel/core/lock.c:2070.

  • ESXi hosts might intermittently fail with a purple diagnostic screen with an error such as @BlueScreen: VERIFY bora/vmkernel/sched/cpusched.c that suggests a preemption anomaly. However, the VMkernel preemption anomaly detection logic might fail to identify the correct kernel context and show a warning for the wrong context.

  • If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  • Some poll requests might exceed the metadata heap of the vmkapi character device driver and cause ESXi hosts to fail with a purple diagnostic screen and an error such as:
    #PF Exception 14 in world 2099138:IPMI Response IP 0x41800b922ab0 addr 0x18 PTEs:0x0;
    In the VMkernel logs, you might see messages such as:
    WARNING: Heap: 3571: Heap VMKAPI-char-metadata already at its maximum size. Cannot expand.
    and
    WARNING: Heap: 4079: Heap_Align(VMKAPI-char-metadata, 40/40 bytes, 8 align) failed. caller: 0x418029922489

  • In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault. In the vmware.log file, you see an error such as msg.monitorEvent.tripleFault. This issue is not consistent and virtual machines that consistently triple fault or triple fault after boot are impacted by another issue.

  • While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  • If you plug in and out a physical NIC in your vCenter Sever system, after the uplink is restored, you still see an alarm in the vmkernel.log of some ESXi hosts that a NIC link is down. The VOBD daemon might not create the event esx.clear.net.redundancy.restored to remove such alarms, which causes the issue.

  • In some cases, when multiple USB or SD devices with different file systems are connected to your vCenter Server system, ESXi installation by using a ks.cfg installation script and a USB or SD booting device might fail.

  • In the first minutes after mounting a VMFS6 volume, you might see higher than configured rate of automatic unmap operations. For example, if you set the unmap bandwidth to 2000 MB/s, you see unmap operations running at more than 3000 MB/s.

  • For independent nonpersistent mode virtual disks, all writes are stored in a redo log, which is a temporary file with extension .REDO_XXXXXX in the VM directory. However, a reset of a virtual machine does not clear the old redo log on such disks.

  • When you run the esxtop utility by using the ESXi Shell to replay a pre-recorded support bundle, the operation might fail with a segmentation fault.

  • Rarely, in certain configurations, the shutdown or reboot of ESXi hosts might stop at the step Shutting down device drivers for a long time, in the order of 20 minutes, but the operation eventually completes.

  • The virtual network adapter VMXNET Generation 3 (VMXNET3) uses buffers to process rx packets. Such buffers are either pre-pinned or pinned and mapped during runtime. In rare occasions, some buffers might not un-pin and get re-pinned later, resulting in a higher-than-expected pin count of such a buffer. As a result, virtual machines might fail with an error such as MONITOR PANIC: vmk: vcpu-0:Too many sticky pages..

  • Due to a rare lock rank violation in the vSphere Replication I/O filter, some ESXi hosts might fail with a purple diagnostic screen when vSphere Replication is enabled. You see an error such as VERIFY bora/vmkernel/main/bh.c:978 on the screen.

  • In vCenter Server advanced performance charts, you see an increasing number of packet drop count for all virtual machines that have NetX redirection enabled. However, if you disable NetX redirection, the count becomes 0.

  • Virtual switches processing IGMPv3 SSM specific queries might drop some queries when an IP is not included in the list of available ports as a result of the source IP check. 

  • An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  • In particular circumstances, when you power on a virtual machine with a corrupted VMware Tools manifest file, the hostd service might fail. As a result, the ESXi host becomes unresponsive. You can backtrace the issue in the hostd dump file that is usually generated in such cases at the time a VM is powered on.

  • After a reboot of an ESXi host, encrypted virtual machines might not auto power on even when Autostart is configured with the Start delay option to set a specific start time of the host. The issue affects only encrypted virtual machines due to a delay in the distribution of keys from standard key providers.

  • When the SLP service is disabled to prevent potential security vulnerabilities, the sfcbd-watchdog service might remain enabled and cause compliance check failures when you perform updates by using a host profile.

  • When an ESXi 6.7.x host has more than 120 VMs on it, vSphere vMotion operations to that host or from that host might become slow or occasionally fail.

  • In certain environments, vSphere Client might not fully protect the underlying websites through modern browser security mechanisms.

  • ESXi updates and upgrades might fail on Mac Pro servers with a purple diagnostic screen and a message such as PSOD - NOT IMPLEMENTED bora/vmkernel/hardware/pci/bridge.c:372 PCPU0:2097152/bootstrap. The issue occurs when Mac Pro BIOS does not assign resources to some devices.

  • Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines with ENS enabled, because of the IPv6 extension header.

  • Resource allocation for a delta disk to create a snapshot of a large virtual machine, with virtual disks equal to or exceeding 1TB, on large VMFS6 datastores of 30TB or more, might take significant time. As a result, the virtual machine might temporarily lose connectivity. The issue affects primarily VMFS6 filesystems.

  • In case of temporary connectivity issues, ESXi hosts might not discover devices configured with the VMW_SATP_INV plug-in after connectivity restores, because SCSI commands that fail during the device discovery stage cause an out-of-memory condition for the plug-in.

  • If you import a virtual machine with a virtual USB device that is not supported by vCenter Server, such as a virtual USB camera, to a vCenter Server system, the VM might fail to power on. The start of such virtual machines fails with an error message similar to: PANIC: Unexpected signal: 11. The issue affects mostly VM import operations from VMware Fusion or VMware Workstation systems, which support a wide range of virtual USB devices.

  • A non-blocking I/O operation might block a PBC file operation in a VMFS volume. In the vmkernel.log, you see an error such as:
    cpu2:2098100)WARNING: PB3: 5020: Unexpected result (Would block) while waiting for PBC to clear -FD c63 r24
    However, this is expected behavior.

  • By default, the vmxnet3 driver uses port 4789 when a VM uses ENS and port 8472 when ENS is not enabled. As a result, when an overlay tunnel is configured from a guest VM with different default VXLAN port, vmxnet3 might drop packets.

  • In some cases, MAC learning does not work as expected and affects some virtual machine operations. For example, cloning a VM with the same MAC address on a different VLAN causes a traffic flood to the cloned VM in the ESXi host where the original VM is present.

  • Attempts to create a vCenter Host Profiles CIM Indication Subscription might fail due to a cim-xml parsing request that returns an out of memory error.

  • The packet scheduler that controls network I/O in a vCenter Server system has a fixed length queue that might quickly fill up. As a result, you might see packet drops on uplink on heavy traffic.

  • In a vSphere 6.7.x system, you cannot add an Intel Ice Lake server to a Skylake cluster with enabled EVC mode. In the vSphere Web Client or the vSphere Client, you see the following message:
    Host CPU lacks features required by that mode. XSAVE of BND0-BND3 bounds registers (BNDREGS) is unsupported. XSAVE of BNDCFGU and BNDSTATUS registers (BNDCSR) is unsupported.

  • The VIM API server deletes the Active Directory cache on certain intervals to check the validity of permissions of Active Directory domain accounts. After the cache delete operation, daily by default, the VIM API server re-populates the cache. However, the cache re-populate operation might be slow on domains with many accounts. As a result, such operations might impact performance across the entire vCenter Server system.

  • An abrupt power cycling or reboot of an ESX host might cause a race between subsequent journal replays because other hosts might try to access the same resources. If a race condition happens, the journal replay cannot complete. As a result, virtual machines on a VMFS6 datastore become inaccessible or any operations running on the VMs fail or stop.

  • If you manually run the reboot_helper.py script to shut down a vSAN cluster, the process cleans unused cron jobs registered during the shutdown. However, the script might start a new cron instance instead of restarting the cron daemon after cleaning the unused cron jobs. As a result, multiple cron instances might accumulate and you see cron jobs executed multiple times.

  • If you run a recursive call on a large datastore, such as DatastoreBrowser::SearchSubFolders, a memory leak condition might cause the hostd service to fail with an out-of-memory error.

  • When you enable encryption for a vSAN cluster with a Key Management Interoperability Protocol (KMIP) provider, the following health check might report status errors: vCenter KMS status.

  • For all-flash vSAN configurations, you might see larger than required recommended size for the core dump partition. As a result, the core dump configuration check might fail.

  • In rare cases, hardware issues might cause an SQlite DB corruption that makes multiple VMs become inaccessible and lead to some downtime for applications.

  • When a vSAN host has memory pressure, adding disks to a disk group might cause failure of the memory block attributes (blkAttr) component initialization. Without the blkAttr component, commit and flush tasks stop, which causes log entries to build up in the SSD and eventually cause а congestion. As a result, a NMI failure might occur due to the CPU load for processing the large number of log entries.

  • In some cases, the host designated as the leader cannot send heartbeat messages to other hosts in a large vSAN cluster. This issue occurs when the leader uses an insufficient size of the TX buffer. The result is that new hosts cannot join the cluster with large cluster support (up to 64 hosts) enabled.

  • On a vSAN cluster with a large number of idle virtual machines, vSAN hosts might experience memory congestion due to higher scrub frequency. 

  • An issue with trim space operations count might cause a large number of pending delete operations in vSAN. If an automatic rebalancing operation triggers, it cannot progress due to such pending delete operations. For example, if you set the rebalancing threshold at 30%, you do not see any  progress on rebalancing even when the storage capacity reaches 80%.

  • Due to a dependency between the management of virtual RAM and virtual NVDIMM within a virtual machine, excessive access to the virtual NVDIMM device might lead to a lock spinout while accessing virtual RAM and cause the ESXi host to fail with a purple diagnostic screen. In the backtrace, you see the following:
    SP_WaitLock
SPLockWork
AsyncRemapPrepareRemapListVM
AsyncRemap_AddOrRemapVM
LPageSelectLPageToDefrag
VmAssistantProcessTasks

  • When you enable Latency Sensitivity on virtual machines, some threads of the Likewise Service Manager (lwsmd), which sets CPU affinity explicitly, might compete for CPU resources on such virtual machines. As a result, you might see the ESXi host and the hostd service to become unresponsive.

  • ESXi670-202111001 adds two fixes to the USB host controller for cases when a USB or SD device lose connectivity to the /bootbank partition:

    1. Adds a parameter usbStorageRegisterDelaySecs.
       The usbStorageRegisterDelaySecs parameter is part of the vmkusb module and it delays to register cached USB storage device within the limit of 0 to 600 seconds. The default delay is 10 seconds. The parametric delay makes sure that if the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the USB or SD device does not get a new path that might break connection to the /bootbank partition or corrupt the VMFS-L LOCKER partition.
    2. Enhances the USB host controller driver to tolerate command timeout failures.
       In cases when a device, such as Dell IDSDM, reconnects by a hardware signal, or a USB device resets due to a heavy workload, the USB host controller driver might make only one retry to restore connectivity. With the fix, the controller makes multiple retries and extends tolerance to timeout failures.
  • Some LDAP queries that have no specified timeouts might cause a significant delay in domain join operations for adding an ESXi host to an Active Directory domain.

  • After an upgrade of the brcmfcoe driver on Hitachi storage arrays, ESXi hosts might fail to boot and lose connectivity.

  • The OEM smartpqi driver in ESXi 670-202111001 uses a new logical LUN ID assignment rule to become compatible with the drivers of versions later than 1.0.3.2323. As a result, OEM smartpqi drivers of version 1.0.3.2323 and earlier might fail to blink LEDs or get the physical locations on logical drives.

  • When vSphere Replication is enabled on a virtual machine, you might see higher datastore and in-guest latencies that in certain cases might lead to ESXi hosts becoming unresponsive to vCenter Server. The increased latency comes from vSphere Replication computing MD5 checksums on the I/O completion path, which delays all other I/Os.

ESXi-6.7.0-20211101001s-standard

Profile Name	ESXi-6.7.0-20211101001s-standard
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	November 23, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsanhealth_6.7.0-3.155.18811783 VMware_bootbank_vsan_6.7.0-3.155.18811780 VMware_bootbank_esx-update_6.7.0-3.155.18812553 VMware_bootbank_esx-base_6.7.0-3.155.18812553 VMW_bootbank_vmkusb_0.1-1vmw.670.3.155.18812553 VMware_locker_tools-light_11.3.5.18557794-18812553 VMware_bootbank_cpu-microcode_6.7.0-3.155.18812553
PRs Fixed	2704746, 2765136, 2794798, 2794804, 2794933, 2794943, 2795965, 2797008, 2830222, 2830671, 2762034, 2762034, 2785530, 2816541, 2776028
Related CVE numbers	CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029, CVE-2021-28831

• This patch resolves the following issues:

  • The SQLite database is updated to version 3.34.1.

  • The OpenSSL package is updated to version openssl-1.0.2za.

  • The cURL library is updated to 7.78.0.

  • The OpenSSH is updated to version 8.6p1.

  • The libarchive library is updated to libarchive - 3.5.1.

  • The Busybox package is updated to address CVE-2021-28831.

  • The glibc library is updated tо address the following CVEs: CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029.

  • The following VMware Tools ISO images are bundled with ESXi 670-202111001:

    • windows.iso: VMware Tools 11.3.5 supports Windows 7 SP1 or Windows Server 2008 R2 SP1 and later.
    • linux.iso: VMware Tools 10.3.23 ISO image for Linux OS with glibc 2.11 or later.

    The following VMware Tools ISO images are available for download:

    • VMware Tools 11.0.6:
      • windows.iso: for Windows Vista (SP2) and Windows Server 2008 Service Pack 2 (SP2).
         
    • VMware Tools 10.0.12:
      • winPreVista.iso: for Windows 2000, Windows XP, and Windows 2003.
      • linuxPreGLibc25.iso: supports Linux guest operating systems earlier than Red Hat Enterprise Linux (RHEL) 5, SUSE Linux Enterprise Server (SLES) 11, Ubuntu 7.04, and other distributions with glibc version earlier than 2.5.
         
    • solaris.iso: VMware Tools image 10.3.10 for Solaris.
    • darwin.iso: Supports Mac OS X versions 10.11 and later.

    Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

    • VMware Tools 11.3.5 Release Notes
    • Earlier versions of VMware Tools
    • What Every vSphere Admin Must Know About VMware Tools
    • VMware Tools for hosts provisioned with Auto Deploy
    • Updating VMware Tools

  • This patch updates the cpu-microcode VIB.

  • Code Name	FMS	Plt ID	MCU Rev	MCU Date	Brand Names
    Clarkdale	0x20652	0x12	0x00000011	5/8/2018	Intel i3/i5 Clarkdale Series; Intel Xeon 34xx Clarkdale Series
    Arrandale	0x20655	0x92	0x00000007	4/23/2018	Intel Core i7-620LE Processor
    Sandy Bridge DT	0x206a7	0x12	0x0000002f	2/17/2019	Intel Xeon E3-1100 Series; Intel Xeon E3-1200 Series; Intel i7-2655-LE Series; Intel i3-2100 Series
    Westmere EP	0x206c2	0x03	0x0000001f	5/8/2018	Intel Xeon 56xx Series; Intel Xeon 36xx Series
    Sandy Bridge EP	0x206d6	0x6d	0x00000621	3/4/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
    Sandy Bridge EP	0x206d7	0x6d	0x0000071a	3/24/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
    Westmere EX	0x206f2	0x05	0x0000003b	5/16/2018	Intel Xeon E7-8800 Series; Intel Xeon E7-4800 Series; Intel Xeon E7-2800 Series
    Ivy Bridge DT	0x306a9	0x12	0x00000021	2/13/2019	Intel i3-3200 Series; Intel i7-3500-LE/UE; Intel i7-3600-QE; Intel Xeon E3-1200-v2 Series; Intel Xeon E3-1100-C-v2 Series; Intel Pentium B925C
    Haswell DT	0x306c3	0x32	0x00000028	11/12/2019	Intel Xeon E3-1200-v3 Series; Intel i7-4700-EQ Series; Intel i5-4500-TE Series; Intel i3-4300 Series
    Ivy Bridge EP	0x306e4	0xed	0x0000042e	3/14/2019	Intel Xeon E5-4600-v2 Series; Intel Xeon E5-2600-v2 Series; Intel Xeon E5-2400-v2 Series; Intel Xeon E5-1600-v2 Series; Intel Xeon E5-1400-v2 Series
    Ivy Bridge EX	0x306e7	0xed	0x00000715	3/14/2019	Intel Xeon E7-8800/4800/2800-v2 Series
    Haswell EP	0x306f2	0x6f	0x00000046	1/27/2021	Intel Xeon E5-4600-v3 Series; Intel Xeon E5-2600-v3 Series; Intel Xeon E5-2400-v3 Series; Intel Xeon E5-1600-v3 Series; Intel Xeon E5-1400-v3 Series
    Haswell EX	0x306f4	0x80	0x00000019	2/5/2021	Intel Xeon E7-8800/4800-v3 Series
    Broadwell H	0x40671	0x22	0x00000022	11/12/2019	Intel Core i7-5700EQ; Intel Xeon E3-1200-v4 Series
    Avoton	0x406d8	0x01	0x0000012d	9/16/2019	Intel Atom C2300 Series; Intel Atom C2500 Series; Intel Atom C2700 Series
    Broadwell EP/EX	0x406f1	0xef	0x0b00003e	2/6/2021	Intel Xeon E7-8800/4800-v4 Series; Intel Xeon E5-4600-v4 Series; Intel Xeon E5-2600-v4 Series; Intel Xeon E5-1600-v4 Series
    Skylake SP	0x50654	0xb7	0x02006b06	3/8/2021	Intel Xeon Platinum 8100 Series; Intel Xeon Gold 6100/5100, Silver 4100, Bronze 3100 Series; Intel Xeon D-2100 Series; Intel Xeon D-1600 Series; Intel Xeon W-3100 Series; Intel Xeon W-2100 Series
    Cascade Lake B-0	0x50656	0xbf	0x04003103	4/20/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
    Cascade Lake	0x50657	0xbf	0x05003103	4/8/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
    Cooper Lake	0x5065b	0xbf	0x07002302	4/23/2021	Intel Xeon Platinum 8300 Series; Intel Xeon Gold 6300/5300
    Broadwell DE	0x50662	0x10	0x0000001c	6/17/2019	Intel Xeon D-1500 Series
    Broadwell DE	0x50663	0x10	0x0700001b	2/4/2021	Intel Xeon D-1500 Series
    Broadwell DE	0x50664	0x10	0x0f000019	2/4/2021	Intel Xeon D-1500 Series
    Broadwell NS	0x50665	0x10	0x0e000012	2/4/2021	Intel Xeon D-1600 Series
    Skylake H/S	0x506e3	0x36	0x000000ea	1/25/2021	Intel Xeon E3-1500-v5 Series; Intel Xeon E3-1200-v5 Series
    Denverton	0x506f1	0x01	0x00000034	10/23/2020	Intel Atom C3000 Series
    Ice Lake SP	0x606a6	0x87	0x0d0002a0	4/25/2021	Intel Xeon Silver 4300 Series; Intel Xeon Gold 6300/5300 Series; Intel Xeon Platinum 8300 Series
    Snow Ridge	0x80665	0x01	0x0b00000f	2/17/2021	Intel Atom P5000 Series
    Kaby Lake H/S/X	0x906e9	0x2a	0x000000ea	1/5/2021	Intel Xeon E3-1200-v6 Series; Intel Xeon E3-1500-v6 Series
    Coffee Lake	0x906ea	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series; Intel Xeon E-2200 Series (4 or 6 core)
    Coffee Lake	0x906eb	0x02	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
    Coffee Lake	0x906ec	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
    Coffee Lake Refresh	0x906ed	0x22	0x000000ea	1/5/2021	Intel Xeon E-2200 Series (8 core)
    Rocket Lake S	0xa0671	0x02	0x00000040	4/11/2021	Intel Xeon E-2300 Series

ESXi-6.7.0-20211101001s-no-tools

Profile Name	ESXi-6.7.0-20211101001s-no-tools
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	November 23, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsanhealth_6.7.0-3.155.18811783 VMware_bootbank_vsan_6.7.0-3.155.18811780 VMware_bootbank_esx-update_6.7.0-3.155.18812553 VMware_bootbank_esx-base_6.7.0-3.155.18812553 VMW_bootbank_vmkusb_0.1-1vmw.670.3.155.18812553 VMware_bootbank_cpu-microcode_6.7.0-3.155.18812553
PRs Fixed	2704746, 2765136, 2794798, 2794804, 2794933, 2794943, 2795965, 2797008, 2830222, 2830671, 2762034, 2762034, 2785530, 2816541, 2776028
Related CVE numbers	CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029, CVE-2021-28831

• This patch updates the following issues:

  • The SQLite database is updated to version 3.34.1.

  • The OpenSSL package is updated to version openssl-1.0.2za.

  • The cURL library is updated to 7.78.0.

  • The OpenSSH is updated to version 8.6p1.

  • The libarchive library is updated to libarchive - 3.5.1.

  • The Busybox package is updated to address CVE-2021-28831.

  • The glibc library is updated tо address the following CVEs: CVE-2015-5180, CVE-2015-8777, CVE-2015-8982, CVE-2016-3706, CVE-2017-1000366, CVE-2018-1000001, CVE-2018-19591, CVE-2019-19126, CVE-2020-10029.

  • The following VMware Tools ISO images are bundled with ESXi 670-202111001:

    • windows.iso: VMware Tools 11.3.5 supports Windows 7 SP1 or Windows Server 2008 R2 SP1 and later.
    • linux.iso: VMware Tools 10.3.23 ISO image for Linux OS with glibc 2.11 or later.

    The following VMware Tools ISO images are available for download:

    • VMware Tools 11.0.6:
      • windows.iso: for Windows Vista (SP2) and Windows Server 2008 Service Pack 2 (SP2).
         
    • VMware Tools 10.0.12:
      • winPreVista.iso: for Windows 2000, Windows XP, and Windows 2003.
      • linuxPreGLibc25.iso: supports Linux guest operating systems earlier than Red Hat Enterprise Linux (RHEL) 5, SUSE Linux Enterprise Server (SLES) 11, Ubuntu 7.04, and other distributions with glibc version earlier than 2.5.
         
    • solaris.iso: VMware Tools image 10.3.10 for Solaris.
    • darwin.iso: Supports Mac OS X versions 10.11 and later.

    Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

    • VMware Tools 11.3.5 Release Notes
    • Earlier versions of VMware Tools
    • What Every vSphere Admin Must Know About VMware Tools
    • VMware Tools for hosts provisioned with Auto Deploy
    • Updating VMware Tools

  • This patch updates the cpu-microcode VIB.

  • Code Name	FMS	Plt ID	MCU Rev	MCU Date	Brand Names
    Clarkdale	0x20652	0x12	0x00000011	5/8/2018	Intel i3/i5 Clarkdale Series; Intel Xeon 34xx Clarkdale Series
    Arrandale	0x20655	0x92	0x00000007	4/23/2018	Intel Core i7-620LE Processor
    Sandy Bridge DT	0x206a7	0x12	0x0000002f	2/17/2019	Intel Xeon E3-1100 Series; Intel Xeon E3-1200 Series; Intel i7-2655-LE Series; Intel i3-2100 Series
    Westmere EP	0x206c2	0x03	0x0000001f	5/8/2018	Intel Xeon 56xx Series; Intel Xeon 36xx Series
    Sandy Bridge EP	0x206d6	0x6d	0x00000621	3/4/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
    Sandy Bridge EP	0x206d7	0x6d	0x0000071a	3/24/2020	Intel Pentium 1400 Series; Intel Xeon E5-1400 Series; Intel Xeon E5-1600 Series; Intel Xeon E5-2400 Series; Intel Xeon E5-2600 Series; Intel Xeon E5-4600 Series
    Westmere EX	0x206f2	0x05	0x0000003b	5/16/2018	Intel Xeon E7-8800 Series; Intel Xeon E7-4800 Series; Intel Xeon E7-2800 Series
    Ivy Bridge DT	0x306a9	0x12	0x00000021	2/13/2019	Intel i3-3200 Series; Intel i7-3500-LE/UE; Intel i7-3600-QE; Intel Xeon E3-1200-v2 Series; Intel Xeon E3-1100-C-v2 Series; Intel Pentium B925C
    Haswell DT	0x306c3	0x32	0x00000028	11/12/2019	Intel Xeon E3-1200-v3 Series; Intel i7-4700-EQ Series; Intel i5-4500-TE Series; Intel i3-4300 Series
    Ivy Bridge EP	0x306e4	0xed	0x0000042e	3/14/2019	Intel Xeon E5-4600-v2 Series; Intel Xeon E5-2600-v2 Series; Intel Xeon E5-2400-v2 Series; Intel Xeon E5-1600-v2 Series; Intel Xeon E5-1400-v2 Series
    Ivy Bridge EX	0x306e7	0xed	0x00000715	3/14/2019	Intel Xeon E7-8800/4800/2800-v2 Series
    Haswell EP	0x306f2	0x6f	0x00000046	1/27/2021	Intel Xeon E5-4600-v3 Series; Intel Xeon E5-2600-v3 Series; Intel Xeon E5-2400-v3 Series; Intel Xeon E5-1600-v3 Series; Intel Xeon E5-1400-v3 Series
    Haswell EX	0x306f4	0x80	0x00000019	2/5/2021	Intel Xeon E7-8800/4800-v3 Series
    Broadwell H	0x40671	0x22	0x00000022	11/12/2019	Intel Core i7-5700EQ; Intel Xeon E3-1200-v4 Series
    Avoton	0x406d8	0x01	0x0000012d	9/16/2019	Intel Atom C2300 Series; Intel Atom C2500 Series; Intel Atom C2700 Series
    Broadwell EP/EX	0x406f1	0xef	0x0b00003e	2/6/2021	Intel Xeon E7-8800/4800-v4 Series; Intel Xeon E5-4600-v4 Series; Intel Xeon E5-2600-v4 Series; Intel Xeon E5-1600-v4 Series
    Skylake SP	0x50654	0xb7	0x02006b06	3/8/2021	Intel Xeon Platinum 8100 Series; Intel Xeon Gold 6100/5100, Silver 4100, Bronze 3100 Series; Intel Xeon D-2100 Series; Intel Xeon D-1600 Series; Intel Xeon W-3100 Series; Intel Xeon W-2100 Series
    Cascade Lake B-0	0x50656	0xbf	0x04003103	4/20/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
    Cascade Lake	0x50657	0xbf	0x05003103	4/8/2021	Intel Xeon Platinum 9200/8200 Series; Intel Xeon Gold 6200/5200; Intel Xeon Silver 4200/Bronze 3200; Intel Xeon W-3200
    Cooper Lake	0x5065b	0xbf	0x07002302	4/23/2021	Intel Xeon Platinum 8300 Series; Intel Xeon Gold 6300/5300
    Broadwell DE	0x50662	0x10	0x0000001c	6/17/2019	Intel Xeon D-1500 Series
    Broadwell DE	0x50663	0x10	0x0700001b	2/4/2021	Intel Xeon D-1500 Series
    Broadwell DE	0x50664	0x10	0x0f000019	2/4/2021	Intel Xeon D-1500 Series
    Broadwell NS	0x50665	0x10	0x0e000012	2/4/2021	Intel Xeon D-1600 Series
    Skylake H/S	0x506e3	0x36	0x000000ea	1/25/2021	Intel Xeon E3-1500-v5 Series; Intel Xeon E3-1200-v5 Series
    Denverton	0x506f1	0x01	0x00000034	10/23/2020	Intel Atom C3000 Series
    Ice Lake SP	0x606a6	0x87	0x0d0002a0	4/25/2021	Intel Xeon Silver 4300 Series; Intel Xeon Gold 6300/5300 Series; Intel Xeon Platinum 8300 Series
    Snow Ridge	0x80665	0x01	0x0b00000f	2/17/2021	Intel Atom P5000 Series
    Kaby Lake H/S/X	0x906e9	0x2a	0x000000ea	1/5/2021	Intel Xeon E3-1200-v6 Series; Intel Xeon E3-1500-v6 Series
    Coffee Lake	0x906ea	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series; Intel Xeon E-2200 Series (4 or 6 core)
    Coffee Lake	0x906eb	0x02	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
    Coffee Lake	0x906ec	0x22	0x000000ea	1/5/2021	Intel Xeon E-2100 Series
    Coffee Lake Refresh	0x906ed	0x22	0x000000ea	1/5/2021	Intel Xeon E-2200 Series (8 core)
    Rocket Lake S	0xa0671	0x02	0x00000040	4/11/2021	Intel Xeon E-2300 Series

Known Issues

The known issues are grouped as follows.

• Miscellaneous Issues
• Known Issues from Prior Releases

Miscellaneous Issues

• The sensord daemon fails to report ESXi host hardware status

  A logic error in the IPMI SDR validation might cause sensord to fail to identify a source for power supply information. As a result, when you run the command vsish -e get /power/hostStats, you might not see any output.

  Workaround: None

• The ESXi SNMP service intermittently stops working and ESXi hosts become unresponsive

  If your environment has IPv6 disabled, the ESXi SNMP agent might stop responding. As a result, ESXi hosts also become unresponsive. In the VMkernel logs, you see multiple messages such as Admission failure in path: snmpd/snmpd.3955682/uw.3955682.

  Workaround: Manually restart the SNMP service or use a cron job to periodically restart the service.

• Stateful ESXi installation might fail on hosts with a hardware iSCSI disk connected to a Emulex OneConnect OCe11100 or 14000 NIC device

  An issue with the IMA elx-esx-libelxima plug-in might cause the hostd service to fail and statefull ESXi installation cannot complete on hosts with hardware iSCSI disk connected to a Emulex OneConnect OCe11100 or 14000 NIC device. After a network boot, the ESXi server does not reboot, but you see no errors in the vSphere Client or vSphere Web Client.

  Workaround: Install a vendor-provided async version of the elx-esx-libelxima plug-in.

• The Windows guest OS of a virtual machine configured with a virtual NVDIMM of size less than 16MB might fail while initializing a new disk

  If you configure a Windows virtual machine with a NVDIMM of size less than 16MB, when you try to initialize a new disk, you might see either the guest OS failing with a blue diagnostic screen or an error message in a pop-up window in the Disk Management screen. The blue diagnostic screen issue occurs in Windows 10, Windows Server 2022, and Windows 11 v21H2 guest operating systems. 

  Workaround: Increase the size of the virtual NVDIMM to 16MB or larger.

Known Issues from Prior Releases

To view a list of previous known issues, click here.

The earlier known issues are grouped as follows.

• CLI Issues
• Internationalization Issues
• Tools Issues
• Installation, Upgrade, and Migration Issues
• Security Features Issues
• Networking Issues
• Storage Issues
• Backup and Restore Issues
• vCenter Server Appliance, vCenter Server, vSphere Web Client, and vSphere Client Issues
• Virtual Machine Management Issues
• vSphere HA and Fault Tolerance Issues
• Auto Deploy and Image Builder Issues
• Miscellaneous Issues

CLI Issues

• Views might switch from the appliance shell to the Direct Console User Interface during an upgrade of the vCenter Server Appliance by using the CLI installer

  During an upgrade of the vCenter Server Appliance by using the CLI installer, views might switch intermittently from the appliance shell to the Direct Console User Interface. Restarts of the applmgmt service during updates causes the issue.

  Workaround: Switch to appliance shell tty to monitor the progress. 

Internationalization Issues

• A VMkernel network using an NSX logical switch might fail for stateless hosts if you register vCenter Server with non-ASCII characters on VMware NSX Manager

  If you register vCenter Server to an NSX Manager with a password containing characters from the extended ASCII codes between 128 and 255, or non-ASCII characters, a VMkernel network using an NSX logical switch might be lost after deploying a stateless host.

  Workaround: Register vCenter Server to an NSX Manager with a password containing only ASCII characters.

• An ESXi host might stop responding if you add a vSphere Distributed Switch named with a string containing tens of non-ASCII characters to a physical adapter in a hyper-converged infrastructure (HCI) cluster

  If you name a VDS with a string containing more than 40 characters from the extended ASCII codes between 128 and 255, or more than 26 non-ASCII characters, ESXi hosts might stop responding when you attempt to add the VDS to a physical adapter during the configuration of a hyper-converged infrastructure (HCI) cluster.

  Workaround: Use strings with less than 40 characters from the extended ASCII codes and 26 non-ASCII characters when naming a VDS.

Tools Issues

• The OVF Tool might fail to verify an SSL thumbprint if you use CLI

  If you set the SSL thumbprint value by using CLI, the OVF Tool might fail to verify the thumbprint. The issue is not monitored if you use the Direct Console User Interface (DCUI).

  Workaround: Use any of the following alternatives:

  • In the DCUI, specify the thumbprint in the section of ssl_certificate_verification.
  • In the DCUI, specify to ignore certificate thumbprint for ESXi by putting ssl_certificate_verification verification_mode to False.
  • Ignore all certificate thumbprints globally by using the command-line parameter: --no-ssl-certificate-verification.
  • Wait for the CLI prompt to accept the thumbprint that it receives from the source. 

Installation, Upgrade, and Migration Issues

• ESXi installation or upgrade fail due to memory corruption on HPE ProLiant - DL380/360 Gen 9 Servers

  The issue occurs on HPE ProLiant - DL380/360 Gen 9 Servers that have a Smart Array P440ar storage controller.

  Workaround: Set the server BIOS mode to UEFI before you install or upgrade ESXi.

• After an ESXi upgrade to version 6.7 and a subsequent rollback to version 6.5 or earlier, you might experience failures with error messages

  You might see failures and error messages when you perform one of the following on your ESXi host after reverting to 6.5 or earlier versions:

  • Install patches and VIBs on the host
    Error message: [DependencyError] VIB VMware_locker_tools-light requires esx-version >= 6.6.0
  • Install or upgrade VMware Tools on VMs
    Error message: Unable to install VMware Tools.

  After the ESXi rollback from version 6.7, the new tools-light VIB does not revert to the earlier version. As a result, the VIB becomes incompatible with the rolled back ESXi host causing these issues.

  Workaround: Perform the following to fix this problem.

  SSH to the host and run one of these commands:

  esxcli software vib install -v /path/to/tools-light.vib

  or

  esxcli software vib install -d /path/to/depot/zip -n tools-light

  Where the vib and zip are of the currently running ESXi version.

  Note: For VMs that already have new VMware Tools installed, you do not have to revert VMware Tools back when ESXi host is rolled back.

• Special characters backslash (\) or double-quote (") used in passwords causes installation pre-check to fail

  If the special characters backslash (\) or double quote (") are used in ESXi, vCenter Single Sign-On, or operating system password fields during the vCenter Server Appliance Installation templates, the installation pre-check fails with the following error:

  Error message: com.vmware.vcsa.installer.template.cli_argument_validation: Invalid \escape: line ## column ## (char ###)

  Workaround: If you include special characters backslash (\) or double quote (") in the passwords for ESXi, operating systems, or Single-Sign-On, the special characters need to be escaped. For example, the password pass\word should be escaped as pass\\word.

• Windows vCenter Server 6.7 installer fails when non-ASCII characters are present in password

  The Windows vCenter Server 6.7 installer fails when the Single Sign-on password contains non-ASCII characters for Chinese, Japanese, Korean, and Taiwanese locales. 

  Workaround: Ensure that the Single Sign-on password contains ASCII characters only for Chinese, Japanese, Korean, and Taiwanese locales. In case your vCenter Server system is installed on an external database, both the user and password for vCenter Server login must not include non-ASCII characters for Simplified Chinese, Japanese, Korean, Traditional Chinese, French, German and Spanish locales.

• Cannot log in to vSphere Appliance Management Interface if the colon character (:) is part of vCenter Server root password

  During the vCenter Server Appliance UI installation (Set up appliance VM page of Stage 1), if you include the colon character (:) as part of the vCenter Server root password, logging into the vSphere Appliance Management Interface (https://vc_ip:5480) fails and you are unable to login. The password might be accepted by the password rule check during the setup, but login fails.

  Workaround: Do not use the colon character (:) to set the vCenter Server root password in the vCenter Server Appliance UI (Set up appliance VM of Stage 1).

• vCenter Server Appliance installation fails when the backslash character (\) is included in the vCenter Single Sign-On password

  During the vCenter Server Appliance UI installation (SSO setup page of Stage 2), if you include the backslash character (\) as part of the vCenter Single Sign-On password, the installation fails with the error Analytics Service registration with Component Manager failed. The password might be accepted by the password rule check, but installation fails.

  Workaround: Do not use the backslash character (\) to set the vCenter Single Sign-On password in the vCenter Server Appliance UI installer (SSO setup page of Stage 2)

• Scripted ESXi installation fails on HP ProLiant Gen 9 Servers with an error 

  When you perform a scripted ESXi installation on an HP ProLiant Gen 9 Server under the following conditions: 

  • The Embedded User Partition option is enabled in the BIOS. 
  • You use multiple USB drives during installation: one USB drive contains the ks.cfg file, and the others USB drive is not formatted and usable.

  The installation fails with the error message Partitions not initialized.

  Workaround:

  1. Disable the Embedded User Partition option in the server BIOS.
  2. Format the unformatted USB drive with a file system or unplug it from the server.
• Upgrading vCenter Server 6.5 for Windows to vCenter Server 6.7 might fail if the vSphere Authentication Proxy service is active

  If the vSphere Authentication Proxy service is active while you perform an upgrade from vCenter Server 6.5 for Windows to vCenter Server 6.7, the operation might fail during the pre-check. You might see an error similar to:

  The following non-configurable port(s) are already in use:
2016, 7475, 7476
Stop the process(es) that use these port(s).

  Workaround: Stop the vSphere Authentication Proxy service. You can restart the service after the successful upgrade to vCenter Server 6.7.

• Patching to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7 might fail when vCenter Server High Availability is active

  Patching to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7 might fail when vCenter Server High Availability is active due to a DB schema change. For more information, see VMware knowledge base article 55938.

  Workaround: To patch your system to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7, you must remove vCenter Server High Availability and delete passive and witness nodes. After the upgrade, you must re-create your vCenter Server High Availability clusters.

• Windows vCenter Server 6.0.x or 6.5.x upgrade to vCenter Server 6.7 fails if vCenter Server contains non-ASCII or high-ASCII named 5.5 host profiles

  When a source Windows vCenter Server 6.0.x or 6.5.x contains vCenter Server 5.5.x host profiles named with non-ASCII or high-ASCII characters, UpgradeRunner fails to start during the upgrade pre-check process.

  Workaround: Before upgrading Windows vCenter Server 6.0.x or 6.5.x to vCenter Server 6.7, upgrade the ESXi 5.5.x with the non-ASCII or high-ASCII named host profiles to ESXi 6.0.x or 6.5.x, then update the host profile from the upgraded host by clicking Copy setting from the hosts.

• Upgrade to vCenter Server Appliance 6.7 Update 1 from vCenter Server Appliance 6.5 Update 2 and later, using custom HTTP and HTTPS ports, might fail

  Upgrades from vCenter Server Appliance 6.5 Update 2 and later, using custom HTTP and HTTPS ports, to vCenter Server Appliance 6.7 Update 1 might fail. You might see the issue regardless if you use the GUI or CLI installer.

  Workaround: None

• Converging an external Platform Services Controller to a vCenter Server might fail if the Platform Services Controller uses a custom HTTPS port

  You might fail to converge an external Platform Services Controller to a vCenter Server system, if the vCenter Server system is configured with the default HTTPS port, 443, and the Platform Services Controller node is configured with a custom value for the HTTPS port. The operation fails in the firstboot stage due to convergence issues.

  Workaround: Change the HTTPS port value to the default value, 443, for Platform Services Controller nodes before running vCenter External to Embedded Convergence tool. You can run the following commands to do the same:

  1. /usr/lib/vmware-vmafd/bin/vmafd-cli set-dc-port --server-name localhost --dc-port 443
  2. /usr/lib/vmware-vmafd/bin/vmafd-cli set-rhttpproxy-port --server-name localhost --rhttpproxy-port 443
• After an upgrade of a vCenter Server system to version 6.7 Update 2, pre-upgrade First Class Disks (FCD) might not be listed in the Global Catalog

  During an upgrade of your vCenter Server system to version 6.7 Update 2, the FCD Global Catalog might pick an ESXi host that is not yet updated to invoke a sync and the sync fails. As a result, the listVStorageObjectForSpec API might not return all FCDs created prior to the upgrade.

  Workaround: After you upgrade all ESXi hosts in the inventory, start syncDatastore API with fullSync set to true.

• If you edit the etc/issue file, vCenter Server Appliance upgrade by using the Virtual Appliance Management Infrastructure user interface fails

  If you edit the etc/issue file while you upgrade a vCenter Server Appliance by using the Virtual Appliance Management Infrastructure user interface, for example to create a Custom Login Banner, and you do not split the text in new lines, the upgrade fails. In the interface, you see an error such as:
  Got exception while trying to save metadata to a file: Unexpected content in /etc/issue file.

  Workaround: Restore the contents of the /etc/issue file before an upgrade by using the Virtual Appliance Management Infrastructure user interface. 

• After an upgrade to vCenter Server 6.7 Update 3l and later, you might see existing or new HPE VASA providers in disconnected state

  If the inventory of your vCenter Server system has vSphere Virtual Volumes supported by either of HPE 3PAR StoreServ or HPE Primera VASA providers, you might see the providers get into a disconnected state after an upgrade to vCenter Server 6.7 Update 3l or later. The issue affects 3PAR 3.3.1 MU5 storage, but not 3PAR 3.3.1 MU3 storage. 

  Workaround: Upgrade to vCenter Server 7.0 Update 1c and later. For upgrade compatibility, see VMware knowledge base article 67077.
  Alternatively, you can restore your system to a backup prior to vCenter Server 6.7 Update 3l.
  If you are not already using HPE 3PAR 3.3.1 MU5 VASA provider, postpone the VASA provider upgrade to HPE 3PAR 3.3.1 MU5 until HPE resolves the issue. For more information, see VMware knowledge base article 83038.

• You cannot run the camregister command with the -x option if the vCenter Single Sign-On password contains non-ASCII characters

  When you run the camregister command with the -x file option, for example, to register the vSphere Authentication Proxy, the process fails with an access denied error when the vCenter Single Sign-On password contains non-ASCII characters.

  Workaround: Either set up the vCenter Single Sign-On password with ASCII characters, or use the –p password option when you run the camregister command to enter the vCenter Single Sign-On password that contains non-ASCII characters.

• The Bash shell and SSH login are disabled after upgrading to vCenter Server 6.7

  After upgrading to vCenter Server 6.7, you are not able to access the vCenter Server Appliance using either the Bash shell or SSH login.

  Workaround:

  1. After successfully upgrading to vCenter Server 6.7, log in to the vCenter Server Appliance Management Interface. In a Web browser, go to: https://appliance_ip_address_or_fqdn:5480
  2. Log in as root.
     The default root password is the password you set while deploying the vCenter Server Appliance.
  3. Click Access, and click Edit.
  4. Edit the access settings for the Bash shell and SSH login. 
     When enabling Bash shell access to the vCenter Server Appliance, enter the number of minutes to keep access enabled.
  5. Click OK to save the settings.
• Management node migration is blocked if vCenter Server for Windows 6.0 is installed on Windows Server 2008 R2 without previously enabling Transport Layer Security 1.2

  This issue occurs if you are migrating vCenter Server for Windows 6.0 using an external Platform Services Controller (an MxN topology) on Windows Server 2008 R2. After migrating the external Platform Services Controller, when you run Migration Assistant on the Management node it fails, reporting that it cannot retrieve the Platform Services Controller version. This error occurs because Windows Server 2008 R2 does not support Transport Layer Security (TLS) 1.2 by default, which is the default TLS protocol for Platform Services Controller 6.7.

  Workaround: Enable TLS 1.2 for Windows Server 2008 R2.1.

  1. Navigate to the registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols
  2. Create a new folder and label it TLS 1.2.
  3. Create two new keys with the TLS 1.2 folder, and name the keys Client and Server.
  4. Under the Client key, create two DWORD (32-bit) values, and name them DisabledByDefault and Enabled.
  5. Under the Server key, create two DWORD (32-bit) values, and name them DisabledByDefault and Enabled.
  6. Ensure that the Value field is set to 0 and that the Base is Hexadecimal for DisabledByDefault.
  7. Ensure that the Value field is set to 1 and that the Base is Hexadecimal for Enabled.
  8. Reboot the Windows Server 2008 R2 computer.

  For more information on using TLS 1.2 with Windows Server 2008 R2, refer to the operating system vendor's documentation.

• vCenter Server containing host profiles with version less than 6.0 fails during upgrade to version 6.7

  vCenter Server 6.7 does not support host profiles with version less than 6.0. To upgrade to vCenter Server 6.7, you must first upgrade the host profiles to version 6.0 or later, if you have any of the following components:

  • ESXi host(s) version - 5.1 or 5.5
  • vCenter server version - 6.0 or 6.5
  • Host profiles version - 5.1 or 5.5

  Workaround: See KB 52932

• After upgrading to vCenter Server 6.7, any edits to the ESXi host's /etc/ssh/sshd_config file are discarded, and the file is restored to the vCenter Server 6.7 default configuration

  Due to changes in the default values in the /etc/ssh/sshd_config file, the vCenter Server 6.7 upgrade replaces any manual edits to this configuration file with the default configuration. This change was necessary as some prior settings (for example, permitted ciphers) are no longer compatible with current ESXi behavior, and prevented SSHD (SSH daemon) from starting correctly.

  CAUTION: Editing /etc/ssh/sshd_config is not recommended. SSHD is disabled by default, and the preferred method for editing the system configuration is through the VIM API (including the ESXi Host Client interface) or ESXCLI.

  Workaround: If edits to /etc/ssh/sshd_config are needed, you can apply them after successfully completing the vCenter Server 6.7 upgrade. The default configuration file now contains a version number. Preserve the version number to avoid overwriting the file.

  For further information on editing the /etc/ssh/sshd_config file, see the following Knowledge Base articles:

  • For information on enabling public/private key authentication, see Knowledge Base article KB 1002866
  • For information on changing the default SSHD configuration, see Knowledge Base article KB 1020530

Security Features Issues

• Virtualization Based Security (VBS) on vSphere in Windows Guest OSs RS1, RS2 and RS3 require HyperV to be enabled in the Guest OS.

  Virtualization Based Security (VBS) on vSphere in Windows Guest OSs RS1, RS2 and RS3 require HyperV to be enabled in the Guest OS.

  Workaround: Enable Hyper-V Platform on Windows Server 2016. In the Server Manager, under Local Server select Manage -> Add Roles and Features Wizard and under Role-based or feature-based installation select Hyper-V from the server pool and specify the server roles. Choose defaults for Server Roles, Features, Hyper-V, Virtual Switches, Migration and Default Stores. Reboot the host.

  Enable Hyper-V on Windows 10: Browse to Control Panel -> Programs -> Turn Windows features on or off. Check the Hyper-V Platform which includes the Hyper-V Hypervisor and Hyper-V Services. Uncheck Hyper-V Management Tools. Click OK. Reboot the host.

• The TLS configuration utility in vCenter Server 6.7.x does not install on certain versions of ESXi

  The TLS Configuration utility does not install on certain early versions of ESXi due to the replacement of expired digital signing certificate and key. 

  Workaround: The utility is signed with the same certificate and key as ESX VIBs. For more information on the impacted ESXi versions, see VMware knowledge base article 76555.

• Remote HTTPS servers might not send the HTTP Strict-Transport-Security response header (HSTS) ports 5480 and 5580

  In some environments, remote HTTPS servers running on ports 5480 and 5580 might not return HSTS.

  Workaround: None

Networking Issues

• Hostprofile PeerDNS flags do not work in some scenarios

  If PeerDNS for IPv4 is enabled for a vmknic on a stateless host that has an associated host profile, the iPv6PeerDNS might appear with a different state in the extracted host profile after the host reboots.

  Workaround: None.

• When you upgrade vSphere Distributed Switches to version 6.6, you might encounter a few known issues

  During upgrade, the connected virtual machines might experience packet loss for a few seconds.

  Workaround: If you have multiple vSphere Distributed Switches that need to be upgraded to version 6.6, upgrade the switches sequentially.

  Schedule the upgrade of vSphere Distributed Switches during a maintenance window, set DRS mode to manual, and do not apply DRS recommendations for the duration of the upgrade.

  For more details about known issues and solutions, see KB 52621

• VM fails to power on when Network I/O Control is enabled and all active uplinks are down

  A VM fails to power on when Network I/O Control is enabled and the following conditions are met:

  • The VM is connected to a distributed port group on a vSphere distributed switch
  • The VM is configured with bandwidth allocation reservation and the VM's network adapter (vNIC) has a reservation configured
  • The distributed port group teaming policy is set to Failover
  • All active uplinks on the distributed switch are down. In this case, vSphere DRS cannot use the standby uplinks and the VM fails to power on.

  Workaround: Move the available standby adapters to the active adapters list in the teaming policy of the distributed port group.

• Network flapping on a NIC that uses qfle3f driver might cause ESXi host to crash

  The qfle3f driver might cause the ESXi host to crash (PSOD) when the physical NIC that uses the qfle3f driver experiences frequent link status flapping every 1-2 seconds.

  Workaround: Make sure that network flapping does not occur. If the link status flapping interval is more than 10 seconds, the qfle3f driver does not cause ESXi to crash. For more information, see KB 2008093.

• Port Mirror traffic packets of ERSPAN Type III fail to be recognized by packet analyzers

  A wrong bit that is incorrectly introduced in ERSPAN Type III packet header causes all ERSPAN Type III packets to appear corrupt in packet analyzers.

  Workaround: Use GRE or ERSPAN Type II packets, if your traffic analyzer supports these types. 

• DNS configuration esxcli commands are not supported on non-default TCP/IP stacks

  DNS configuration of non-default TCP/IP stacks is not supported. Commands such as esxcli network ip dns server add -N vmotion -s 10.11.12.13 do not work.

  Workaround: Do not use DNS configuration esxcli commands on non-default TCP/IP stacks.

• Compliance check fails with an error when applying a host profile with enabled default IPv4 gateway for vmknic interface

  When applying a host profile with enabled default IPv4 gateway for vmknic interface, the setting is populated with "0.0.0.0" and does not match the host info, resulting with the following error:

  IPv4 vmknic gateway configuration doesn't match the specification

  Workaround:

  1. Edit the host profile settings.
  2. Navigate to Networking configuration > Host virtual nic or Host portgroup > (name of the vSphere Distributed Switch or name of portgroup) > IP address settings.
  3. From the Default gateway Vmkernal Network Adapter (IPv4) drop-down menu, select Choose a default IPv4 gateway for the vmknic and enter the Vmknic Default IPv4 gateway.
• Intel Fortville series NICs cannot receive Geneve encapsulation packets with option length bigger than 255 bytes

  If you configure Geneve encapsulation with option length bigger than 255 bytes, the packets are not received correctly on Intel Fortville NICs X710, XL710, and XXV710.

  Workaround: Disable hardware VLAN stripping on these NICs by running the following command:

  esxcli network nic software set --untagging=1 -n vmnicX.

• RSPAN_SRC mirror session fails after migration

  When a VM connected to a port assigned for RSPAN_SRC mirror session is migrated to another host, and there is no required pNic on the destination network of the destination host, then the RSPAN_SRC mirror session fails to configure on the port. This causes the port connection to fail failure but the vMotion migration process succeeds.

  Workaround: To restore port connection failure, complete either one of the following:

  • Remove the failed port and add a new port.
  • Disable the port and enable it.

  The mirror session fails to configure, but the port connection is restored.

Storage Issues

• NFS datastores intermittently become read-only

  A host's NFS datastores may become read-only when the NFS vmknic temporarily loses its IP address or after a stateless hosts reboot.

  Workaround: You can unmount and remount the datastores to regain connectivity through the NFS vmknic. You can also set the NFS datastore write permission to both the IP address of the NFS vmknic and the IP address of the Management vmknic.

• When editing a VM's storage policies, selecting Host-local PMem Storage Policy fails with an error 

  In the Edit VM Storage Policies dialog, if you select Host-local PMem Storage Policy from the dropdown menu and click OK, the task fails with one of these errors:

  The operation is not supported on the object.

  or

  Incompatible device backing specified for device '0'"Detailed

  Workaround: You cannot apply the Host-local PMem Storage Policy to VM home. For a virtual disk, you can use the migration wizard to migrate the virtual disk and apply the Host-local PMem Storage Policy.

• Datastores might appear as inaccessible after ESXi hosts in a cluster recover from a permanent device loss state

  This issue might occur in the environment where the hosts in the cluster share a large number of datastore, for example, 512 to 1000 datastores.
  After the hosts in the cluster recover from the permanent device loss condition, the datastores are mounted successfully at the host level. However, in vCenter Server, several datastores might continue to appear as inaccessible for a number of hosts.

  Workaround: On the hosts that show inaccessible datastores in the vCenter Server view, perform the Rescan Storage operation from vCenter Server.

• Incorrect behavior of the backingObjectId and SnapshotInfo fields of VStorageObjectResult

  In non-vSAN datastores, the backingObjectId and SnapshotInfo fields of VStorageObjectResult for a First Class Disk are always set to null.
  In vSAN datastores, when you create a snapshot of a First Class Disk, backingObjectId and SnapshotInfo fields of VStorageObjectResult for the First Class Disk are populated. If the First Class Disk has multiple snapshots, deleting the latest snapshot updates the backingObjectId and SnapshotInfo fields, but deleting older snapshots does not update the fields.

  Workaround: None.

• Migration of a virtual machine from a VMFS3 datastore to VMFS5 fails in a mixed ESXi 6.5 and 6.7 host environment

  If you have a mixed host environment, you cannot migrate a virtual machine from a VMFS3 datastore connected to an ESXi 6.5 host to a VMFS5 datastore on an ESXi 6.7 host.

  Workaround: Upgrade the VMFS3 datastore to VMFS5 to be able to migrate the VM to the ESXi 6.7 host.

• Warning message about a VMFS3 datastore remains unchanged after you upgrade the VMFS3 datastore using the CLI

  Typically, you use the CLI to upgrade the VMFS3 datastore that failed to upgrade during an ESXi upgrade. The VMFS3 datastore might fail to upgrade due to several reasons including the following:

  • No space is available on the VMFS3 datastore.
  • One of the extents on the spanned datastore is offline.

  After you fix the reason of the failure and upgrade the VMFS3 datastore to VMFS5 using the CLI, the host continues to detect the VMFS3 datastore and reports the following error:

  Deprecated VMFS (ver 3) volumes found. Upgrading such volumes to VMFS (ver5) is mandatory for continued availability on vSphere 6.7 host.

  Workaround: To remove the error message, restart hostd using the /etc/init.d/hostd restart command or reboot the host.

• The Mellanox ConnectX-4/ConnectX-5 native ESXi driver might exhibit performance degradation when its Default Queue Receive Side Scaling (DRSS) feature is turned on

  Receive Side Scaling (RSS) technology distributes incoming network traffic across several hardware-based receive queues, allowing inbound traffic to be processed by multiple CPUs. In Default Queue Receive Side Scaling (DRSS) mode, the entire device is in RSS mode. The driver presents a single logical queue to OS and is backed by several hardware queues.

  The native nmlx5_core driver for the Mellanox ConnectX-4 and ConnectX-5 adapter cards enables the DRSS functionality by default. While DRSS helps to improve performance for many workloads, it could lead to possible performance degradation with certain multi-VM and multi-vCPU workloads.

  Workaround: If significant performance degradation is observed, you can disable the DRSS functionality.

  1. Run the esxcli system module parameters set -m nmlx5_core -p DRSS=0 RSS=0 command.
  2. Reboot the host.
• Datastore name does not extract to the Coredump File setting in the host profile

  When you extract a host profile, the Datastore name field is empty in the Coredump File setting of the host profile. Issue appears when using esxcli command to set coredump.

  Workaround:

  1. Extract a host profile from an ESXi host.
  2. Edit the host profile settings and navigate to General System Settings > Core Dump Configuration > Coredump File.
  3. Select Create the Coredump file with an explicit datastore and size option and enter the Datastore name, where you want the Coredump File to reside.
• Native software FCoE adapters configured on an ESXi host might disappear when the host is rebooted

  After you successfully enable the native software FCoE adapter (vmhba) supported by the vmkfcoe driver and then reboot the host, the adapter might disappear from the list of adapters. This might occur when you use Cavium QLogic 57810 or QLogic 57840 CNAs supported by the qfle3 driver.

  Workaround: To recover the vmkfcoe adapter, perform these steps:

  1. Run the esxcli storage core adapter list command to make sure that the adapter is missing from the list.
  2. Verify the vSwitch configuration on vmnic associated with the missing FCoE adapter.
  3. Run the following command to discover the FCoE vmhba:
     • On a fabric setup:
       #esxcli fcoe nic discover -n vmnic_number
     • On a VN2VN setup:
       #esxcli fcoe nic discover -n vmnic_number
• Attempts to create a VMFS datastore on an ESXi 6.7 host might fail in certain software FCoE environments

  Your attempts to create the VMFS datastore fail if you use the following configuration:

  • Native software FCoE adapters configured on an ESXi 6.7 host.
  • Cavium QLogic 57810 or 57840 CNAs.
  • Cisco FCoE switch connected directly to an FCoE port on a storage array from the Dell EMC VNX5300 or VNX5700 series.

  Workaround: None.

  As an alternative, you can switch to the following end-to-end configuration:
  ESXi host > Cisco FCoE switch > FC switch > storage array from the DELL EMC VNX5300 and VNX5700 series.

Backup and Restore Issues

• Windows Explorer displays some backups with unicode differently from how browsers and file system paths show them

  Some backups containing unicode display differently in the Windows Explorer file system folder than they do in browsers and file system paths.

  Workaround: Using http, https, or ftp, you can browse backups with your web browser instead of going to the storage folder locations through Windows Explorer.

vCenter Server Appliance, vCenter Server, vSphere Web Client, and vSphere Client Issues

• The time synchronization mode setting is not retained when upgrading vCenter Server Appliance

  If NTP time synchronization is disabled on a source vCenter Server Appliance, and you perform an upgrade to vCenter Server Appliance 6.7, after the upgrade has successfully completed NTP time synchronization will be enabled on the newly upgraded appliance.

  Workaround:

  1. After successfully upgrading to vCenter Server Appliance 6.7, log into the vCenter Server Appliance Management Interface as root.

     The default root password is the password you set while deploying the vCenter Server Appliance.

      https://IP_or_FQDN_of_appliance:5480

  2. In the vCenter Server Appliance Management Interface, click Time.
  3. In the Time Synchronization pane, click Edit.
  4. From the Mode drop-down menu, select Disabled.

     The newly upgraded vCenter Server Appliance 6.7 will no longer use NTP time synchronization, and will instead use the system time zone settings.

• Login to vSphere Web Client with Windows session authentication fails on Firefox browsers of version 54 or later

  If you use Firefox of version 54 or later to log in to the vSphere Web Client, and you use your Windows session for authentication, the VMware Enhanced Authentication Plugin might fail to populate your user name and to log you in.

  Workaround: If you are using Windows session authentication to log in to the vSphere Web Client, use one of the following browsers: Internet Explorer, Chrome, or Firefox of version 53 and earlier.

• vCenter hardware health alarm notifications are not triggered in some instances

  When multiple sensors in the same category on an ESXi host are tripped within a time span of less than five minutes, traps are not received and email notifications are not sent.

  Workaround: None. You can check the hardware sensors section for any alerts.

• The vSphere Client and vSphere Web Client might not reflect update from vCenter Server 6.7 to vCenter Server 6.7 Update 1 for vCenter Server for Windows

  If you update vCenter Server for Windows from vCenter Server 6.7 to vCenter Server 6.7 Update 1, the build number details for vpxd in the Summary tab of both the vSphere Client and vSphere Web Client might not reflect the update and show version 6.7.0.

  Workaround: None.

• When using the VCSA Installer Time Sync option, you must connect the target ESX to the NTP server in the Time & Date Setting from the ESX Management

  If you want to select Time Sync with NTP server from the VCSA Installer->Stage2->Appliance configuration->Time Sync option (ESX/NTP server), you also need to have the target ESX already connected to NTP server in the Time&Date Setting from the ESX Management, otherwise it'll fail in installation.

  Workaround:

  1. Set the Time Sync option in stage2->Appliance configuration to sync with ESX
  2. Set the Time Sync option in stage2->Appliance configuration to sync with NTP Servers, make sure both the ESX and VC are set to connect to NTP servers.
• When you monitor Windows vCenter Server health, an error message appears

  Health service is not available for Windows vCenter Server. If you select the vCenter Server, and click Monitor > Health, an error message appears:

  Unable to query vSAN health information. Check vSphere Client logs for details.

  This problem can occur after you upgrade the Windows vCenter Server from release 6.0 Update 1 or 6.0 Update 2 to release 6.7. You can ignore this message.

  Workaround: None. Users can access vSAN health information through the vCenter Server Appliance.

• vCenter hardware health alarms do not function with earlier ESXi versions

  If ESXi version 6.5 Update 1 or earlier is added to vCenter 6.7, hardware health related alarms will not be generated when hardware events occur such as high CPU temperatures, FAN failures, and voltage fluctuations.

  Workaround: None.

• vCenter Server stops working in some cases when using vmodl to edit or expand a disk

  When you configure a VM disk in a Storage DRS-enabled cluster using the latest vmodl, vCenter Server stops working. A previous workaround using an earlier vmodl no longer works and will also cause vCenter Server to stop working.

  Workaround: None

• vCenter Server for Windows migration to vCenter Server Appliance fails with error

  When you migrate vCenter Server for Windows 6.0.x or 6.5.x to vCenter Server Appliance 6.7, the migration might fail during the data export stage with the error: The compressed zip folder is invalid or corrupted.

  Workaround: You must zip the data export folder manually and follow these steps:

  1. In the source system, create an environment variable MA_INTERACTIVE_MODE.
  2. Go to Computer > Properties > Advanced system settings > Environment Variables > System Variables > New.
  3. Enter "MA_INTERACTIVE_MODE" as variable name with value 0 or 1.
  4. Start the VMware Migration Assistant and provide your password.
  5. Start the Migration from the client machine. The migration will pause, and the Migration Assistant console will display the message To continue the migration, create the export.zip file manually from the export data (include export folder).
  6. NOTE: Do not press any keys or tabs on the Migration Assistant console.
  7. Go to the %appdata%\vmware\migration-assistant folder.
  8. Delete the export.zip created by the Migration Assistant.
  9. To continue the migration, manually create the export.zip file from the export folder.
  10. Return to the Migration Assistant console. Type Y and press Enter.
• Discrepancy between the build number in VAMI and the build number in the vSphere Client

  In vSphere 6.7, the VAMI summary tab displays the ISO build for the vCenter Server and vCenter Server Appliance products. The vSphere Client summary tab displays the build for the vCenter product, which is a component within the vCenter Server product.

  Workaround: None

• vCenter Server Appliance 6.7 displays an error message in the Available Update section of the vCenter Server Appliance Management Interface (VAMI)

  The Available Update section of the vCenter Server Appliance Management Interface (VAMI) displays the following error message:

  Check the URL and try again.

  This message is generated when the vCenter Server Appliance searches for and fails to find a patch or update. No functionality is impacted by this issue. This issue will be resolved with the release of the first patch for vSphere 6.7.

  Workaround: None. No functionality is impacted by this issue.

Virtual Machine Management Issues

• Name of the virtual machine in the inventory changes to its path name

  This issue might occur when a datastore where the VM resides enters the All Paths Down state and becomes inaccessible. When hostd is loading or reloading VM state, it is unable to read the VM's name and returns the VM path instead. For example, /vmfs/volumes/123456xxxxxxcc/cs-00.111.222.333.

  Workaround: After you resolve the storage issue, the virtual machine reloads, and its name is displayed again.

• You must select the "Secure boot" Platform Security Level when enabling VBS in a Guest OS on AMD systems

  On AMD systems, vSphere virtual machines do not provide a vIOMMU. Since vIOMMU is required for DMA protection, AMD users cannot select "Secure Boot and DMA protection" in the Windows Group Policy Editor when they "Turn on Virtualization Based Security". Instead select "Secure boot." If you select the wrong option it will cause VBS services to be silently disabled by Windows.

  Workaround: Select "Secure boot" Platform Security Level in a Guest OS on AMD systems.

• You cannot hot add memory and CPU for Windows VMs when Virtualization Based Security (VBS) is enabled within Windows

  Virtualization Based Security (VBS) is a new feature introduced in Windows 10 and Windows Server 2016. vSphere supports running Windows with VBS enabled starting in the vSphere 6.7 release. However, Hot add of memory and CPU will not operate for Windows VMs when Virtualization Based Security (VBS) is enabled.

  Workaround: Power-off the VM, change memory or CPU settings and power-on the VM.

• Snapshot tree of a linked-clone VM might be incomplete after a vSAN network recovery from a failure

  A vSAN network failure might impact accessibility of vSAN objects and VMs. After a network recovery, the vSAN objects regain accessibility. The hostd service reloads the VM state from storage to recover VMs. However, for a linked-clone VM, hostd might not detect that the parent VM namespace has recovered its accessibility. This results in the VM remaining in inaccessible state and VM snapshot information not being displayed in vCenter Server.

  Workaround: Unregister the VM, then re-register it to force the hostd to reload the VM state. Snapshot information will be loaded from storage.

• The Virtual Appliance Management Interface might display a 0- message or a blank page during patching from vCenter Server 6.7 to later versions

  The Virtual Appliance Management Interface might display a 0- message or a blank page during patching from vCenter Server 6.7 to later versions, if calls from the interface fail to reach the back end applmgmt service. You might also see the message Unable to get historical data import status. Check Server Status.

  Workaround: These are not failure messages. Refresh the browser and log in to the Virtual Appliance Management Interface again once the reboot of appliance in the back end is complete.

• The Ready to Complete page of the Register Virtual Machine wizard displays only one horizontal line

  The Ready to Complete page of the Register Virtual Machine wizard might display content similar to one horizontal line due to a rendering issue. This issue does not affect the workflow of the wizard.

  Workaround: None

• VMware Service Lifecycle Manager restarts the vpxd service every couple of days

  If you enable Storage I/O Control for a datastore with a large number of connected ESXi hosts, a big amount of I/O events might pile up for handling by the vpxd service. In such a case, the VMware Service Lifecycle Manager might restart vpxd.

  Workaround: Disable Storage I/O Control for datastores with a large amount of connected ESXi hosts or reduce the number of connected hosts.

• An OVF Virtual Appliance fails to start in the vSphere Client

  The vSphere Client does not support selecting vService extensions in the Deploy OVF Template wizard. As a result, if an OVF virtual appliance uses vService extensions and you use the vSphere Client to deploy the OVF file, the deployment succeeds, but the virtual appliance fails to start.

  Workaround: Use the vSphere Web Client to deploy OVF virtual appliances that use vService extensions. 

vSphere HA and Fault Tolerance Issues

• When you configure Proactive HA in Manual/MixedMode in vSphere 6.7 RC build you are prompted twice to apply DRS recommendations    

  When you configure Proactive HA in Manual/MixedMode in vSphere 6.7 RC build and a red health update is sent from the Proactive HA provider plug-in, you are prompted twice to apply the recommendations under Cluster -> Monitor -> vSphere DRS -> Recommendations. The first prompt is to enter the host into maintenance mode. The second prompt is to migrate all VMs on a host entering maintenance mode. In vSphere 6.5, these two steps are presented as a single recommendation for entering maintenance mode, which lists all VMs to be migrated.

  Workaround: There is no impact to work flow or results. You must apply the recommendations twice. If you are using automated scripts, you must modify the scripts to include the additional step.

• Lazy import upgrade interaction when VCHA is not configured

  The VCHA feature is available as part of  6.5 release. As of 6.5, a VCHA cluster cannot be upgraded while preserving the VCHA configuration. The recommended approach for upgrade is to first remove the VCHA configuration either through vSphere Client or by calling a destroy VCHA API. So for lazy import upgrade workflow without VCHA configuration, there is no interaction with VCHA.

  Do not configure a fresh VCHA setup while lazy import is in progress. The VCHA setup requires cloning the Active VM as Passive/Witness VM. As a result of an ongoing lazy import, the amount of data that needs to be cloned is large and may lead to performance issues.

  Workaround: None.

• You cannot add ESXi hosts running vSphere Fault Tolerance workloads to a vCenter Server system by using the vSphere Client

  Attempts to add ESXi hosts running vSphere Fault Tolerance workloads to a vCenter Server system by using the vSphere Client might fail with the error Cannot add a host with virtual machines that have Fault Tolerance turned on as a stand-alone host.

  Workaround: As alternatives, you can:

  1. Schedule a task to add the host and execute it immediately.
     1. In the vSphere Client, navigate to Configure > Scheduled tasks for a selected cluster.
     2. Select New scheduled task > Add Host.
     3. Schedule a time to run the task.
     4. Add a host and run the task.
     5. Delete the task after the host is added.
  2. Use the vSphere Web Client to add the host. Login to the vSphere Web Client and execute the standard add host workflow.
  3. Turn off the fault tolerance virtual machines temporarily, add the host to the new vCenter Server system, and then turn it on back again.
• vCenter Server High Availability cluster configuration by using an NSX-T logical switch might fail

  Configuration of a vCenter Server High Availability cluster by using an NSX-T logical switch might fail with the error Failed to connect peer node.

  Workaround: Configure vCenter Server High Availability clusters by using a vSphere Distributed Switch.

Auto Deploy and Image Builder Issues

• Reboot of an ESXi stateless host resets the numRxQueue value of the host

  When an ESXi host provisioned with vSphere Auto Deploy reboots, it loses the previously set numRxQueue value. The Host Profiles feature does not support saving the numRxQueue value after the host reboots.

  Workaround: After the ESXi stateless host reboots:

  1. Remove the vmknic from the host.
  2. Create a vmknic on the host with the expected numRxQueue value.
• After caching on a drive, if the server is in the UEFI mode, a boot from cache does not succeed unless you explicitly select the device to boot from the UEFI boot manager

  In case of Stateless Caching, after the ESXi image is cached on a 512n, 512e, USB, or 4Kn target disk, the ESXi stateless boot from autodeploy might fail on a system reboot. This occurs if autodeploy service is down.

  The system attempts to search for the cached ESXi image on the disk, next in the boot order. If the ESXi cached image is found, the host is booted from it. In legacy BIOS, this feature works without problems. However, in the UEFI mode of the BIOS, the next device with the cached image might not be found. As a result, the host cannot boot from the image even if the image is present on the disk.

  Workaround: If autodeploy service is down, on the system reboot, manually select the disk with the cached image from the UEFI Boot Manager.

• If an ESXi host machine uses UEFI firmware, you cannot use VMware vSphere Auto Deploy in a VLAN network

  You might not be able to use vSphere Auto Deploy to deploy an ESXi image over a VLAN network environment when the ESXi host machine uses UEFI firmware. UEFI Secure boot fails at the ipxe binary loading stage because the implementation of VLAN support of some UEFI firmware vendors might not fully support iPXE.

  Workaround: None

• A stateless ESXi host boot time might take 20 minutes or more

  The booting of a stateless ESXi host with 1,000 configured datastores might require 20 minutes or more.

  Workaround: None.

Miscellaneous Issues

• ESXi might fail during reboot with VMs running on the iSCSI LUNs claimed by the qfle3i driver

  ESXi might fail during reboot with VMs running on the iSCSI LUNs claimed by the qfle3i driver if you attempt to reboot the server with VMs in the running I/O state.

  Workaround: First power off VMs and then reboot the ESXi host.

• VXLAN stateless hardware offloads are not supported with Guest OS TCP traffic over IPv6 on UCS VIC 13xx adapters

  You may experience issues with VXLAN encapsulated TCP traffic over IPv6 on Cisco UCS VIC 13xx adapters configured to use the VXLAN stateless hardware offload feature. For VXLAN deployments involving Guest OS TCP traffic over IPV6, TCP packets subject to TSO are not processed correctly by the Cisco UCS VIC 13xx adapters, which causes traffic disruption. The stateless offloads are not performed correctly. From a TCP protocol standpoint this may cause incorrect packet checksums being reported to the ESXi software stack, which may lead to incorrect TCP protocol processing in the Guest OS.

  Workaround: To resolve this issue, disable the VXLAN stateless offload feature on the Cisco UCS VIC 13xx adapters for VXLAN encapsulated TCP traffic over IPV6. To disable the VXLAN stateless offload feature in UCS Manager, disable the Virtual Extensible LAN field in the Ethernet Adapter Policy. To disable the VXLAN stateless offload feature in the CIMC of a Cisco C-Series UCS server, uncheck Enable VXLAN field in the Ethernet Interfaces vNIC properties section.

• Significant time might be required to list a large number of unresolved VMFS volumes using the batch QueryUnresolvedVmfsVolume API

  ESXi provides the batch QueryUnresolvedVmfsVolume API, so that you can query and list unresolved VMFS volumes or LUN snapshots. You can then use other batch APIs to perform operations, such as resignaturing specific unresolved VMFS volumes. By default, when the API QueryUnresolvedVmfsVolume is invoked on a host, the system performs an additional filesystem liveness check for all unresolved volumes that are found.  The liveness check detects whether the specified LUN is mounted on other hosts, whether an active VMFS heartbeat is in progress, or if there is any filesystem activity. This operation is time consuming and requires at least 16 seconds per LUN. As a result, when your environment has a large number of snapshot LUNs, the query and listing operation might take significant time.

  Workaround: To decrease the time of the query operation, you can disable the filesystem liveness check. 

  1. Log in to your host as root.
  2. Open the configuration file for hostd using a text editor. The configuration file is located in /etc/vmware/hostd/config.xml under plugins/hostsvc/storage node.
  3. Add the checkLiveFSUnresolvedVolume parameter and set its value to FALSE. Use the following syntax:

     <checkLiveFSUnresolvedVolume>FALSE</checkLiveFSUnresolvedVolume>

  As an alternative, you can set the ESXi Advanced option VMFS.UnresolvedVolumeLiveCheck to FALSE in the vSphere Client.

• Importing a .csv file overwrites user input during host customization.

  User input in the Customize hosts pane is overwritten by the import process and the values from the .csv file.

  Workaround: Import the .csv file before adding manual changes in the Customize hosts pane.

• The vCenter Server Convergence Tool might fail to convert an external Platform Services Controller to an embedded Platform Services Controller due to conflicting IP and FQDN

  If you have configured an external Platform Services Controller with an IP address as an optional FQDN field during the deployment, the vCenter Server Convergence Tool might fail to convert the external Platform Services Controller to an embedded Platform Services Controller because of a name conflict.

  Workaround: Do not use the vCenter Server Convergence Tool for Platform Services Controllers installed with an IP address as an alternative or addition to the FQDN address.

• If you repoint and reconfigure the setup of a Platform Services Controller, the restore process might fail

  If you repoint and reconfigure the setup of a Platform Services Controller, the restore process might fail due to a stale service ID entry.

  Workaround: Follow the steps in VMware knowledge base article 2131327 to clean up the stale service ID before proceeding with restore.

• Attempts to log in to a vCenter Server system after an upgrade to vCenter Server 6.7 might fail with a credentials validation error

  After an upgrade of your system to vCenter Server 6.7, if you try to log in to the system by using either the vSphere Web Client or vSphere Client, and a security token or smartcard, the login might fail with an error Unable to validate the submitted credential.

  Workaround: Remove and re-add the identity source. For more information, see Add or Edit a vCenter Single Sign-On Identity Source. 

• An Enhanced vMotion Compatibility (EVC) cluster might show new CPU IDs such as IBPB even if you revert an ESXi host to an older version

  If you revert an ESXi host to an older version of ESXi, an EVC cluster might expose new CPU IDs, such as IBRS, STIBP and IBPB, even though the host does not have any of the features.

  Workaround: This issue is resolved in this release. However, a host that does not meet the requirements of an EVC cluster does not automatically reconnect and you must remove it from the cluster.

• Some vCenter Server plug-ins might not correctly render the dark theme mode in the vSphere Client

  If you change color schemes in the vSphere Client to display the interface in a dark theme, some vCenter Server plug-ins might not correctly render the mode.

  Workaround: None

• If you enable per-VM EVC, virtual machines might fail to power on

  If you install or use for upgrade only VMware vCenter Server 6.7 Update 1, but do not apply ESXi 6.7 Update 1, and if you configure or reconfigure per-VM EVC, virtual machines on unpatched hosts might fail to power on. You might also see the issue if you enable cluster-level EVC and even one of the hosts in the cluster is not patched with the latest update. The new CPU IDs of that cluster might not be available on the cluster. In such a cluster, if you configure or reconfigure per-VM EVC, virtual machines might fail to power on.

  Workaround: Before you configure or reconfigure per-VM EVC, upgrade all the standalone ESXi hosts, as well as hosts inside a cluster, to the latest update for hypervisor-assisted guest mitigation for guest operating systems.

• Edits to the DNS settings might cause deletion of the IPv6 loopback address from the /etc/resolv.conf and /etc/systemd/resolved.conf files

  Edits to the DNS settings by using either the Appliance Management Interface, appliance shell, or the vSphere Web Client, might cause deletion of the IPv6 loopback address from the/etc/resolv.conf and /etc/systemd/resolved.conf files.

  Workaround: To avoid deletion of the IPv6 loopback address, edit the resolv.conf files by using the Bash shell:

  1. In the /etc/resolv.conf file, set the following parameters:

     nameserver: ::1
     nameserver: <dnsserver 1>
     nameserver: <dnsserver 2>

  2. In the /etc/systemd/resolved.conf file, set the following parameters:

     [Resolve]
     LLMNR=false
     DNS=::1 <dnsserver 1> <dnsserver 2>

• The SSH service might be disabled after an external Platform Services Controller converts to an embedded Platform Services Controller

  If you convert an external Platform Services Controller to an embedded Platform Services Controller, the SSH service might be disabled based on Active Directory policies and restrictions.

  Workaround: Manually enable the SSH service after the conversion is complete.

• After upgrade to ESXi 6.7, networking workloads on Intel 10GbE NICs cause higher CPU utilization

  If you run certain types of networking workloads on an upgraded ESXi 6.7 host, you might see a higher CPU utilization under the following conditions:

  • The NICs on the ESXi host are from the Intel 82599EB or X540 families
  • The workload involves multiple VMs that run simultaneously and each VM is configured with multiple vCPUs
  • Before the upgrade to ESXi 6.7, the VMKLinux ixgbe driver was used

  Workaround: Revert to the legacy VMKLinux ixgbe driver:

  1. Connect to the ESXi host and run the following command:
     # esxcli system module set -e false -m ixgben
  2. Reboot the host.

  Note: The legacy VMKLinux ixgbe inbox driver version 3.7.x does not support Intel X550 NICs. Use the VMKLinux ixgbe async driver version 4.x with Intel X550 NICs.

• Initial install of DELL CIM VIB might fail to respond

  After you install a third-party CIM VIB it might fail to respond.

  Workaround: To fix this issue, enter the following two commands to restart sfcbd:

  1. esxcli system wbem set --enable false
  2. esxcli system wbem set --enable true

To collapse the list of previous known issues, click here.