ESXi 7.0 Update 2c | 24 AUG 2021 | Build 18426014 Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

• What's New
• Earlier Releases of ESXi 7.0
• Patches Contained in this Release
• Product Support Notices
• Resolved Issues
• Known Issues

What's New

• ESXi 7.0 Update 2c delivers bug and security fixes documented in the Resolved Issues section.

Earlier Releases of ESXi 7.0

New features, resolved, and known issues of ESXi are described in the release notes for each release. Release notes for earlier releases of ESXi 7.0 are:

• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 2a
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 2
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 1d
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 1c
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 1b
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 1a
• VMware ESXi 7.0, Patch Release ESXi 7.0 Update 1
• VMware ESXi 7.0, Patch Release ESXi 7.0b

For internationalization, compatibility, and open source components, see the VMware vSphere 7.0 Release Notes.

For more information on ESXi versions that support upgrade to ESXi 7.0 Update 2c, refer to VMware knowledge base article 67077.

Patches Contained in This Release

This release of ESXi 7.0 Update 2c delivers the following patches:

Build Details

Download Filename:	VMware-ESXi-7.0U2c-18426014-depot
Build:	18426014
Download Size:	580.8 MB
md5sum:	a6b2d1e3a1a071b8d93a55d7a7fc0b63
sha1checksum:	829da0330f765b4ae46c2fb5b90a8b60f90e4e5b
Host Reboot Required:	Yes
Virtual Machine Migration or Shutdown Required:	Yes

Components

Component	Bulletin	Category	Severity
ESXi Component - core ESXi VIBs	ESXi_7.0.2-0.20.18426014	Bugfix	Critical
Emulex FC Driver	Broadcom-ELX-lpfc_12.8.298.3-2vmw.702.0.20.18426014	Bugfix	Critical
Broadcom NetXtreme-E Network and ROCE/RDMA Drivers for VMware ESXi	Broadcom-bnxt-Net-RoCE_216.0.0.0-1vmw.702.0.20.18426014	Bugfix	Important
QLogic FastLinQ 10/25/40/50/100 GbE Ethernet and RoCE/RDMA Drivers for VMware ESXi	MRVL-E4-CNA-Driver-Bundle_1.0.0.0-1vmw.702.0.20.18426014	Bugfix	Important
ESXi Install/Upgrade Component	esx-update_7.0.2-0.20.18426014	Bugfix	Critical
Network driver for Intel(R) X710/XL710/XXV710/X722 Adapters	Intel-i40en_1.8.1.137-1vmw.702.0.20.18426014	Bugfix	Critical
USB Driver	VMware-vmkusb_0.1-4vmw.702.0.20.18426014	Bugfix	Critical
ESXi Component - core ESXi VIBs	ESXi_7.0.2-0.15.18295176	Security	Critical
USB Driver	VMware-vmkusb_0.1-1vmw.702.0.15.18295176	Security	Moderate
ESXi Tools Component	VMware-VM-Tools_11.2.6.17901274-18295176	Security	Critical
ESXi Install/Upgrade Component	esx-update_7.0.2-0.15.18295176	Security	Critical

IMPORTANT:

• To download the ESXi 7.0 Update 2c patch offline depot ZIP file from VMware Customer Connect, you must navigate to Products and Accounts > Product Patches. From the Select a Product drop-down menu, select ESXi (Embedded and Installable) and from the Select a Version drop-down menu, select 7.0.
• Starting with vSphere 7.0, VMware uses components for packaging VIBs along with bulletins. The ESXi and esx-update bulletins are dependent on each other. Always include both in a single ESXi host patch baseline or include the rollup bulletin in the baseline to avoid failure during host patching. 
• When patching ESXi hosts by using VMware Update Manager from a version prior to ESXi 7.0 Update 2, it is strongly recommended to use the rollup bulletin in the patch baseline. If you cannot use the rollup bulletin, be sure to include all of the following packages in the patching baseline. If the following packages are not included in the baseline, the update operation fails:
  • VMware-vmkusb_0.1-1vmw.701.0.0.16850804 or higher
  • VMware-vmkata_0.1-1vmw.701.0.0.16850804 or higher
  • VMware-vmkfcoe_1.0.0.2-1vmw.701.0.0.16850804 or higher
  • VMware-NVMeoF-RDMA_1.0.1.2-1vmw.701.0.0.16850804 or higher

Rollup Bulletin

This rollup bulletin contains the latest VIBs with all the fixes after the initial release of ESXi 7.0.

Bulletin ID	Category	Severity	Details
ESXi-7.0U2c-18426014	Bugfix	Critical	Bugfix and Security
ESXi-7.0U2sc-18295176	Security	Critical	Security only

Image Profiles

VMware patch and update releases contain general and critical image profiles. Application of the general release image profile applies to new bug fixes.

Image Profile Name

ESXi-7.0U2c-18426014-standard

ESXi-7.0U2c-18426014-no-tools

ESXi-7.0U2sc-18295176-standard

ESXi-7.0U2sc-18295176-no-tools

ESXi Image

Name and Version	Release Date	Category	Detail
ESXi70U2c-18426014	08/24/2021	Bugfix	Bugfix and Security image
ESXi70U2sc-18295176	08/24/2021	Security	Security only image

For information about the individual components and bulletins, see the Product Patches page and the Resolved Issues section.

Patch Download and Installation

In vSphere 7.0.x, the Update Manager plug-in, used for administering vSphere Update Manager, is replaced with the Lifecycle Manager plug-in. Administrative operations for vSphere Update Manager are still available under the Lifecycle Manager plug-in, along with new capabilities for vSphere Lifecycle Manager.
The typical way to apply patches to ESXi 7.0.x hosts is by using the vSphere Lifecycle Manager. For details, see About vSphere Lifecycle Manager and vSphere Lifecycle Manager Baselines and Images.
You can also update ESXi hosts without using the Lifecycle Manager plug-in, and use an image profile instead. To do this, you must manually download the patch offline bundle ZIP file from the Product Patches page and use the esxcli software profile update command.
For more information, see the Upgrading Hosts by Using ESXCLI Commands and the VMware ESXi Upgrade guide.

Product Support Notices

• Discontinuation of Trusted Platform Module (TPM) 1.2 in a future major vSphere release: VMware intends in a future major vSphere release to discontinue support of TPM 1.2 and associated features such as TPM 1.2 with TXT. To get full use of vSphere features, you can use TPM 2.0 instead of TPM 1.2. Support for TPM 1.2 continues in all vSphere 7.0.x releases, updates, and patches. However, you will not see a deprecation warning for TPM 1.2 during installation or updates of 7.0.x releases.

• Service Location Protocol (SLP) service is disabled by default​: Starting from ESX 7.0 Update 2c, the SLP service is disabled by default to prevent potential security vulnerabilities. The SLP service is also automatically disabled after upgrades to ESX 7.0 Update 2c. You can manually enable the SLP service by using the command esxcli system slp set --enable true. For more information, see VMware knowledge base article 76372.

Resolved Issues

The resolved issues are grouped as follows.

• ESXi_7.0.2-0.20.18426014
• esx-update_7.0.2-0.20.18426014
• Broadcom-bnxt-Net-RoCE_216.0.0.0-1vmw.702.0.20.18426014
• MRVL-E4-CNA-Driver-Bundle_1.0.0.0-1vmw.702.0.20.18426014
• Intel-i40en_1.8.1.137-1vmw.702.0.20.18426014
• Broadcom-ELX-lpfc_12.8.298.3-2vmw.702.0.20.18426014
• VMware-vmkusb_0.1-4vmw.702.0.20.18426014
• ESXi_7.0.2-0.15.18295176
• esx-update_7.0.2-0.15.18295176
• VMware-VM-Tools_11.2.6.17901274-18295176
• VMware-vmkusb_0.1-1vmw.702.0.15.18295176
• ESXi-7.0U2c-18426014-standard
• ESXi-7.0U2c-18426014-no-tools
• ESXi-7.0U2sc-18295176-standard
• ESXi-7.0U2sc-18295176-no-tools
• ESXi Image - ESXi70U2c-18426014
• ESXi image - ESXi70U2sc-18295176

ESXi_7.0.2-0.20.18426014

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_vsanhealth_7.0.2-0.20.18426014 VMware_bootbank_vsan_7.0.2-0.20.18426014 VMware_bootbank_gc_7.0.2-0.20.18426014 VMware_bootbank_esx-xserver_7.0.2-0.20.18426014 VMware_bootbank_clusterstore_7.0.2-0.20.18426014 VMware_bootbank_vdfs_7.0.2-0.20.18426014 VMware_bootbank_native-misc-drivers_7.0.2-0.20.18426014 VMware_bootbank_cpu-microcode_7.0.2-0.20.18426014 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.20.18426014 VMware_bootbank_esx-base_7.0.2-0.20.18426014
PRs Fixed	2717698, 2731446, 2515171, 2731306, 2751564, 2718934, 2755056, 2725886, 2725886, 2753231, 2728910, 2741111, 2759343, 2721379, 2751277, 2708326, 2755977, 2737934, 2777001, 2760932, 2731263, 2731142, 2760267, 2749688, 2763986, 2765534, 2766127, 2777003, 2778793, 2755807, 2760081, 2731644, 2749815, 2749962, 2738238, 2799408
CVE numbers	N/A

The ESXi and esx-update bulletins are dependent on each other. Always include both in a single ESXi host patch baseline or include the rollup bulletin in the baseline to avoid failure during host patching.

Updates the crx, vsanhealth, vsan, gc, esx-xserver, clusterstore, vdfs, native-misc-drivers, cpu-microcode, esx-dvfilter-generic-fastpath, and esx-base VIBs to resolve the following issues:

• PR 2717698: An ESXi host might fail with a purple diagnostic screen due to a rare race condition in the qedentv driver

  A rare race condition in the qedentv driver might cause an ESXi host to fail with a purple diagnostic screen. The issue occurs when an Rx complete interrupt arrives just after a General Services Interface (GSI) queue pair (QP) is destroyed, for example during a qedentv driver unload or a system shut down. In such a case, the qedentv driver might access an already freed QP address that leads to a PF exception. The issue might occur in ESXi hosts that are connected to a busy physical switch with heavy unsolicited GSI traffic. In the backtrace, you see messages such as:

  cpu4:2107287)0x45389609bcb0:[0x42001d3e6f72]qedrntv_ll2_rx_cb@(qedrntv)#<None>+0x1be stack: 0x45b8f00a7740, 0x1e146d040, 0x432d65738d40, 0x0, 0x
2021-02-11T03:31:53.882Z cpu4:2107287)0x45389609bd50:[0x42001d421d2a]ecore_ll2_rxq_completion@(qedrntv)#<None>+0x2ab stack: 0x432bc20020ed, 0x4c1e74ef0, 0x432bc2002000,
2021-02-11T03:31:53.967Z cpu4:2107287)0x45389609bdf0:[0x42001d1296d0]ecore_int_sp_dpc@(qedentv)#<None>+0x331 stack: 0x0, 0x42001c3bfb6b, 0x76f1e5c0, 0x2000097, 0x14c2002
2021-02-11T03:31:54.039Z cpu4:2107287)0x45389609be60:[0x42001c0db867]IntrCookieBH@vmkernel#nover+0x17c stack: 0x45389609be80, 0x40992f09ba, 0x43007a436690, 0x43007a43669
2021-02-11T03:31:54.116Z cpu4:2107287)0x45389609bef0:[0x42001c0be6b0]BH_Check@vmkernel#nover+0x121 stack: 0x98ba, 0x33e72f6f6e20, 0x0, 0x8000000000000000, 0x430000000001
2021-02-11T03:31:54.187Z cpu4:2107287)0x45389609bf70:[0x42001c28370c]NetPollWorldCallback@vmkernel#nover+0x129 stack: 0x61, 0x42001d0e0000, 0x42001c283770, 0x0, 0x0
2021-02-11T03:31:54.256Z cpu4:2107287)0x45389609bfe0:[0x42001c380bad]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001c0c2b44, 0x0, 0x0, 0x0
2021-02-11T03:31:54.319Z cpu4:2107287)0x45389609c000:[0x42001c0c2b43]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
2021-02-11T03:31:54.424Z cpu4:2107287)^[[45m^[[33;1mVMware ESXi 7.0.2 [Releasebuild-17435195 x86_64]^[[0m
#PF Exception 14 in world 2107287:vmnic7-pollW IP 0x42001d3e6f72 addr 0x1c

  This issue is resolved in this release.

• PR 2731446: While vSphere Replication appliance powers on, the hostd service might fail and cause temporary unresponsiveness of ESXi hosts to vCenter Server

  In rare cases, while the vSphere Replication appliance powers on, the hostd service might fail and cause temporary unresponsiveness of ESXi hosts to vCenter Server. The hostd service restarts automatically and connectivity restores.

  This issue is resolved in this release.

• PR 2515171: Editing an advanced options parameter in a host profile and setting a value to false, results in setting the value to true

  When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  This issue is resolved in this release.

• PR 2731306: ESXi hosts become unresponsive to vCenter Server with repeating messages for admission failures

  ESXi hosts might become unresponsive to vCenter Server even when the hosts are accessible and running. The issue occurs in vendor images with customized drivers where ImageConfigManager consumes more RAM than allocated. As a result, repeated failures of ImageConfigManager cause ESXi hosts to disconnect from vCenter Server.
  In the vmkernel logs, you see repeating errors such as:
  Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.<pid1>:python.<pid2>:uw.<pid2>.
  In the hostd logs, you see messages such as: Task Created : haTask-ha-host-vim.host.ImageConfigManager.fetchSoftwarePackages-<sequence>, followed by: ForkExec(/usr/bin/sh) <pid1>, where <pid1> identifies an ImageConfigManager process.

  This issue is resolved in this release.

• PR 2751564: If you lower the value of the DiskMaxIOSize advanced config option, ESXi hosts I/O operations might fail

  If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  This issue is resolved in this release.

• PR 2718934: Power on or power off operations of virtual machines take too long

  In certain conditions, the power on or power off operations of virtual machines might take as long as 20 minutes. The issue occurs when the underlying storage of the VMs runs a background operation such as HBA rescan or APD events, and some locks are continuously held.

  The issue is resolved in this release. The fix removes unnecessary locks on storage resources and introduces a fast mutex to enhance VM power operation performance.

• PR 2755056: ESXi hosts fail with purple diagnostic screen due to Vital Product Data (VPD) page response size difference

  In rare cases, when the VPD page response size from the target ESXi host is different on different paths to the host, ESXi might write more bytes than the allocated length for a given path. As a result, the ESXi host fails with a purple diagnostic screen and a message such as:
  Panic Message: @BlueScreen: PANIC bora/vmkernel/main/dlmalloc.c:4878 - Corruption in dlmalloc

  This issue is resolved in this release. The fix tracks the size of the VPD pages using a separate field instead of depending on the VPD response header.

• PR 2725886: Rare lock contention might cause some virtual machine operations to fail

  A rare lock contention might cause virtual machine operations such as failover or migration to fail.
  Locking failure messages look similar to:
  vol 'VM_SYS_11', lock at 480321536: [Req mode 1] Checking liveness:
type 10c00003 offset 480321536 v 1940464, hb offset 3801088
  or
  Res3: 2496: Rank violation threshold reached: cid 0xc1d00002, resType 2, cnum 5 vol VM_SYS_11

  This issue is resolved in this release.

• PR 2766036: In the vSphere Client, you see incorrect numbers for the provisioned space of virtual machines

  In complicated workflows that result in creating multiple virtual machines, when you navigate to VMs > Virtual Machines, you might see huge numbers in the Provisioned Space column for some VMs. For example, 2.95 TB instead of the actual provisioned space of 25 GB.

  This issue is resolved in this release.

• PR 2753231: An ESXi host fails with a purple diagnostic screen due to a call stack corruption in the ESXI storage stack

  While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  This issue is resolved in this release.

• PR 2728910: ESX hosts might fail with a purple diagnostic screen when virtual machines power on

  In very rare cases, VMFS resource clusters that are included in a journal transaction might not be locked. As a result, during power on of virtual machines, multiple ESXi hosts might fail with a purple diagnostic screen due to exception 14 in the VMFS layer. A typical message and stack trace seen the vmkernel log is as follows:
  @BlueScreen: #PF Exception 14 in world 2097684:VSCSI Emulat IP 0x418014d06fca addr 0x0
  PTEs:0x16a47a027;0x600faa8007;0x0;
  2020-06-24T17:35:57.073Z cpu29:2097684)Code start: 0x418013c00000 VMK uptime: 0:01:01:20.555
2020-06-24T17:35:57.073Z cpu29:2097684)0x451a50a1baa0:[0x418014d06fca]Res6MemUnlockTxnRCList@esx#nover+0x176 stack: 0x1 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb10:[0x418014c7cdb6]J3_DeleteTransaction@esx#nover+0x33f stack: 0xbad0003
2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb40:[0x418014c7db10]J3_AbortTransaction@esx#nover+0x105 stack: 0x0 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb80:[0x418014cbb752]Fil3_FinalizePunchFileHoleTxnVMFS6@esx#nover+0x16f stack: 0x430fe950e1f0
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bbd0:[0x418014c7252b]Fil3UpdateBlocks@esx#nover+0x348 stack: 0x451a50a1bc78 2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bce0:[0x418014c731dc]Fil3_PunchFileHoleWithRetry@esx#nover+0x89 stack: 0x451a50a1bec8
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bd90:[0x418014c739a5]Fil3_FileBlockUnmap@esx#nover+0x50e stack: 0x230eb5 2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be40:[0x418013c4c551]FSSVec_FileBlockUnmap@vmkernel#nover+0x6e stack: 0x230eb5
2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be90:[0x418013fb87b1]VSCSI_ExecFSSUnmap@vmkernel#nover+0x8e stack: 0x0 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf00:[0x418013fb71cb]VSCSIDoEmulHelperIO@vmkernel#nover+0x2c stack: 0x430145fbe070
2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf30:[0x418013ceadfa]HelperQueueFunc@vmkernel#nover+0x157 stack: 0x430aa05c9618 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bfe0:[0x418013f0eaa2]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0
2020-06-24T17:35:57.083Z cpu29:2097684)base fs=0x0 gs=0x418047400000 Kgs=0x0

  This issue is resolved in this release.

• PR 2741111: Failure of some Intel CPUs to forward Debug Exception (#DB) traps might result in virtual machine triple fault

  In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault.

  This issue is resolved in this release. The fix forwards all #DB traps from CPUs into the guest operating system, except when the DB trap comes from a debugger that is attached to the virtual machine.

• PR 2759343: The sfcbd service does not start when you restart ESXi management agents from the Direct Console User Interface (DCUI)

  If you restart ESXi management agents from DCUI, the sfcbd service does not start. You must manually start the SFCB service.

  This issue is resolved in this release.

• PR 2721379: The hostd service logs a warning for zero power supply sensors every 2 minutes

  The hostd service logs the message BuildPowerSupplySensorList found 0 power supply sensors every 2 minutes. The issue occurs when you upgrade to or fresh install ESXi 7.0 Update 1 on an ESXi host.

  This issue is resolved in this release.

• PR 2751277: ESXi hosts might fail with a purple diagnostic screen due to a null pointer reference error

  In rare cases, the kernel memory allocator might return a NULL pointer that might not be correctly dereferenced. As a result, the ESXi host might fail with a purple diagnostic screen with an error such as #GP Exception 13 in world 2422594:vmm:overCom @ 0x42003091ede4. In the backtrace, you see errors similar to:
  #1 Util_ZeroPageTable
#2 Util_ZeroPageTableMPN
#3 VmMemPfUnmapped
#4 VmMemPfInt
#5 VmMemPfGetMapping
#6 VmMemPf
#7 VmMemPfLockPageInt
#8 VMMVMKCall_Call
#9 VMKVMM_ArchEnterVMKernel

  This issue is resolved in this release.

• PR 2708326: If an NVMe device is hot added and hot removed in a short interval, the ESXi host might fail with a purple diagnostic screen

  If an NVMe device is hot added and hot removed in a short interval, the NVMe driver might fail to initialize the NVMe controller due to a command timeout. As a result, the driver might access memory that is already freed in a cleanup process. In the backtrace, you see a message such as WARNING: NVMEDEV: NVMEInitializeController:4045: Failed to get controller identify data, status: Timeout.
  Eventually, the ESXi host might fail with a purple diagnostic screen with an error similar to #PF Exception ... in world ...:vmkdevmgr.

  This issue is resolved in this release.

• PR 2755977: ESXi hosts might fail to provision thick VMDK or create a swap file during a virtual machine power on

  When an ESXi host does not have free large file blocks (LFBs) to allocate, the host fulfills thick VMDK provisioning or the creation of a swap file with small file blocks (SFBs). However, in rare cases, the host might fail to allocate SFBs as well. As a result, thick VMDK provisioning or the creation of a swap file fails. When virtual machines try to power on, you see an error in the vmkernel logs such as:
   vmkernel.1:2021-01-08T19:13:28.341Z cpu20:2965260)Fil3: 10269: Max no space retries (10) exceeded for caller Fil3_SetFileLength (status 'No space left on device')

  This issue is resolved in this release.

• PR 2737934: If you use very large VMFS6 datastores, you might see repeating CPU lockup errors, or ESXi hosts might intermittently fail

  When you use very large VMFS6 datastores, the process of allocating file blocks for thin VMDKs in a resource cluster might cause a CPU lockup. As a result, you see CPU lockup messages and in rare cases, ESXi hosts might fail. In the backtrace, you see messages such as:
  2021-04-07T02:18:08.730Z cpu31:2345212)WARNING: Heartbeat: 849: PCPU 10 didn't have a heartbeat for 7 seconds; *may* be locked up.
  The associated backtrace involving Res6AffMgr functions looks similar to:
  2021-01-06T02:16:16.073Z cpu0:2265741)ALERT: NMI: 694: NMI IPI: RIPOFF(base):RBP:CS [0x121d74d(0x42001a000000):0x50d0:0xf48] (Src 0x1, CPU0)
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b320:[0x42001b21d74c]Res6AffMgrComputeSortIndices@esx#nover+0x369 stack: 0x43119125a000
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b3b0:[0x42001b224e46]Res6AffMgrGetCluster@esx#nover+0xb1f stack: 0xd 2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b4b0:[0x42001b226491]Res6AffMgr_AllocResourcesInt@esx#nover+0x40a stack: 0x4311910f4890
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b680:[0x42001b2270e2]Res6AffMgr_AllocResources@esx#nover+0x1b stack: 0x0

  This issue is resolved in this release. For more information, see VMware knowledge base article 83473.

• PR 2777001: ESXi hosts with large physical CPU count might fail with a purple diagnostic screen during or after an upgrade to ESXi 7.0 Update 2

  During or after an upgrade to ESXi 7.0 Update 2, the ESX storage layer might not allocate sufficient memory resources for ESXi hosts with a large physical CPU count and many storage devices or paths connected to the hosts. As a result, such ESXi hosts might fail with a purple diagnostic screen.

  This issue is resolved in this release. The fix corrects the ESX storage layer memory allocation.

• PR 2760932: Virtual machines with hardware version 18 might intermittently fail during vSphere vMotion operations

  Virtual machines with hardware version 18 and VMware Tools 11.2.0 or later might fail due to a virtual graphics device issue on the destination side during a vSphere vMotion or Storage vMotion.
  In the vmkernel.log you see a line such as:
  PF failed to handle a fault on mmInfo at va 0x114f5ae0: Out of memory. Terminating...

  This issue is resolved in this release.

• PR 2731263: After updating to hardware version 18, you see a guest OS kernel panic in virtual machines

  After updating to hardware version 18, some guest OS, such as CentOS 7, on AMD CPUs might fail with kernel panic when you boot the virtual machine. You see the kernel panic message when you open a web console for the virtual machine.

  This issue is resolved in this release.

• PR 2731142: Packets with IPv6 Tunnel Encapsulation Limit option are lost in traffic between virtual machines

  Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines, because of the IPv6 extension header.

  This issue is resolved in this release. The fix parses packets with the IPv6 Tunnel Encapsulation Limit option correctly.

• PR 2760267: In vmkernel logs, you see multiple warnings indicating a switch to preferred path XXX in STANDBY state for device XXX

  When virtual media, such as a CD drive, is connected to your vSphere environment by using an Integrated Dell Remote Access Controller (iDRAC), and no media file is mapped to the drive, in the vmkernel log you might see multiple warning messages such as:
  vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L1 in STANDBY state for device mpx.vmhba32:C0:T0:L1.WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L1 for device mpx.vmhba32:C0:T0:L1 to activate. This may lead to path thrashing.

  This issue is resolved in this release.

• PR 2749688: vCenter Server might intermittently remove internal NSX-controlled distributed virtual ports from ESXi hosts

  During 24-hour sync workflows of the vSphere Distributed Switch, vCenter Server might intermittently remove NSX-controlled distributed virtual ports such as the Service Plane Forwarding (SPF) port, from ESXi hosts. As a result, vSphere vMotion operations of virtual machines might fail. You see an error such as Could not connect SPF port : Not found in the logs.

  This issue is resolved in this release.

• PR 2763986: If you configure the scratch partition on the root directory of a VMFS volume, upgrade to ESXi 7.0.x fails and you might lose data on the VMFS volume

  ESXi 7.0 introduced the VMFS-L based ESX-OSData partition, which takes on the role of the legacy scratch partition, locker partition for VMware Tools, and core dump destination. If you configure the scratch partition on the root directory of a VMFS volume during an upgrade to ESXi 7.0.x, the system tries to convert the VMFS files into VMFS-L format to match the OS-Data partition requirements. As a result, the OS-Data partition might overwrite the VMFS volume and cause data loss.

  This issue is resolved in this release. The fix adds a check if the filesystem is VFAT before converting it into OS-DATA. For more information, see VMware knowledge base article 83647.

• PR 2765534: Communication Channel health status of ESXi hosts in NSX Data Center for vSphere clusters is down after upgrade to ESXi 7.0 Update 2a

  Due to a log handling issue, after an upgrade to ESXi 7.0 Update 2a, when you check the Communication Channel health status you see all ESXi hosts with status down, because NSX Manager fails to connect to the Firewall Agent.

  This issue is resolved in this release. If you already face the issue, stop the vsfwd service and retry the upgrade operation.

• PR 2766127: If you upgrade ESXi hosts by using vSphere Quick Boot in an FCoE environment, the physical server might fail with a purple diagnostic screen

  If an ESXi host of version 7.0 Update 2 is installed on a FCoE LUN and uses UEFI boot mode, when you try to upgrade the host by using vSphere QuickBoot, the physical server might fail with a purple diagnostic screen because of a memory error.

  This issue is resolved in this release.

• PR 2777003: If you use a USB as a boot device for ESXi 7.0 Update 2a, ESXi hosts might become unresponsive, and you see host not-responding and boot bank is not found alerts

  USB devices have a small queue depth and due to a race condition in the ESXi storage stack, some I/O operations might not get to the device. Such I/Os queue in the ESXi storage stack and ultimately time out. As a result, ESXi hosts become unresponsive.
  In the vSphere Client, you see alerts such as Alert: /bootbank not to be found at path '/bootbank' and Host not-responding.
  In vmkernel logs, you see errors such as:
  2021-04-12T04:47:44.940Z cpu0:2097441)ScsiPath: 8058: Cancelled Cmd(0x45b92ea3fd40) 0xa0, cmdId.initiator=0x4538c859b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-12T04:48:50.527Z cpu2:2097440)ScsiDeviceIO: 4315: Cmd(0x45b92ea76d40) 0x28, cmdId.initiator=0x4305f74cc780 CmdSN 0x1279 from world 2099370 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-12T04:48:50.527Z cpu2:2097440)Queued:4

  This issue is resolved in this release.

• PR 2778793: Boot time of ESXi hosts slows down after upgrade to ESXi Update 2a

  You might see a slowdown of up to 60 min in the booting time of ESXi hosts after an upgrade to ESXi Update 2a.
  The issue does not affect updates to ESXi Update 2a from earlier versions of ESXi 7.0.x.
  The issue occurs only when you use a vSphere Lifecycle Manager image or baseline, or the esxcli software profile update command to perform the upgrade operation. If you use an ISO image or a scripted installation, you do not encounter the problem.
  The issue is most likely to affect iSCSI configurations, but is related to the ESXi algorithm for boot bank detection, not to slow external storage targets.

  This issue is resolved in this release. The fix increases the tolerance limits when a boot.cfg file or storage are not promptly provided.

• PR 2755807: vSphere vMotion operations on an ESXi host fail with an out of memory error

  Due to a memory leak in the portset heap, vSphere vMotion operations on an ESXi host might fail with an out of memory error. In the vSphere Client, you see an error such as:
  Failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 1 39.
Failed with error status: Out of memory.

  This issue is resolved in this release.

• PR 2760081: vSAN Build Recommendation Engine Health warning after upgrade to vSAN 7.0 Update 2

  If vCenter Server cannot resolve vcsa.vmware.com to an IP address, the vSAN release catalog cannot be uploaded. The following vSAN heath check displays a warning: vSAN Build Recommendation Engine Health.
  You might see an error message similar to the following:
  2021-04-12T11:40:59.435Z ERROR vsan-mgmt[32221] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z WARNING vsan-mgmt[32221] [VsanVumCoreUtil::CheckVumServiceStatus opID=noOpId] Failed to connect VUM service: Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumIntegration::VumStatusCallback opID=noOpId] VUM is not installed or not responsive ​
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumSystemUtil::HasConfigIssue opID=noOpId] Config issue health.test.vum.vumDisabled exists on entity None

  This issue is resolved in this release.

• PR 2731644: You see higher than usual scrub activity on vSAN

  An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  This issue is resolved in this release.

• PR 2749815: ESXi hosts fail with a purple diagnostic screen when a resync is aborted with active guest writes

  When a resync runs in parallel with guest writes on objects owned by an ESXi host, the host might fail with a purple diagnostic screen.

  This issue is resolved in this release.

• PR 2749962: You see a syslog error Failed to write header on rotate

  When the cmmdstimemachineDump.log file reaches a max limit of 10 MB, rotation happens and a new cmmdstimemachineDump file is created. However, during the rotation, in the vmsyslogd.err logs you might see an error such as:
  2021-04-08T03:15:53.719Z vmsyslog.loggers.file : ERROR ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: '/var/run/cmmdstimemachineDumpHeader.txt'

  This issue is resolved in this release.

• PR 2738238: Object Format Change task takes long to respond

  In some cases, the cluster monitoring, membership, and directory service (CMMDS) does not provide correct input to the Object Format Change task and the task takes long to respond.

  This issue is resolved in this release.

• PR 2799408: Upgrading a stretched cluster to version ESXi 7.0 Update 2 or later might cause multiple ESX hosts to fail with a purple diagnostic screen

  In rare scenarios, if the witness host is replaced during the upgrade process and a Disk Format Conversion task runs shortly after the replacement, multiple ESX hosts on a stretched cluster might fail with a purple diagnostic screen.

  This issue is resolved in this release.

• PR 2949777: If the LargeBAR setting of a vmxnet3 device is enabled on ESX 7.0 Update 3 and later, virtual machines might lose connectivity

  The LargeBAR setting that extends the Base Address Register (BAR) on a vmxnet3 device supports the Uniform Passthrough (UPT). However, UPT is not supported on ESX 7.0 Update 3 and later, and if the vmxnet3 driver is downgraded to a version earlier than 7.0, and the LargeBAR setting is enabled, virtual machines might lose connectivity.

  This issue is resolved in this release.

esx-update_7.0.2-0.20.18426014

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_loadesx_7.0.2-0.20.18426014 VMware_bootbank_esx-update_7.0.2-0.20.18426014
PRs Fixed	2722263, 2812906
CVE numbers	N/A

Updates the loadesx and esx-update VIBs to resolve the following issue:

• PR 2722263: You cannot remove Components in a customized ESXi image after upgrade to ESXi 7.0 Update 2a

  If you create a custom 7.0 Update 2a image by using the ESXi Packaging Kit, override some of the Components in the base image, for example Intel-ixgben:1.8.9.0-1OEM.700.1.0.15525992, and then upgrade your ESXi 7.0.x hosts, you cannot remove the custom Component after the upgrade. Also, some VIBs from the custom Components might be missing.

  This issue is resolved in this release.

• PR 2812906: Metadata of VMFS datastores might get corrupted in certain ESXi deployment scenarios due to duplicating System Universally Unique Identifiers (UUIDs)

  Certain deployment scenarios, such as cloning ESXi boot banks, might lead to partially of fully identical UUIDs of multiple ESXi hosts. Since UUIDs are used for VMFS heartbeat and journal operations, when you have duplicating UUIDS, after installation or upgrade operations multiple ESXi hosts might attempt to access metadata regions on the same VMFS datastore. As a result, you might see metadata corruption in some VMFS datastores.
  In the vmkernel logs, you see messages such as:
  vmkernel 608: VMFS volume DCS_HCFA_TEDS_Regulated_10032/5ba2905b-ac11166d-b145-0025b5920a02 on naa.60060e8012a34f005040a34f00000d64:1 has been detected corrupted.
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 319: FS3RCMeta 2245 200 21 105 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 326: 0 0 0 0 0 0 0 0 0 0 0 7 0 254 1 4 0 248 3 96 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 332: 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 338: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

  Other impacted deployment scenarios are reuse of hardware profiles, for example Blade systems that use the same MAC to deploy a new instance, and moving boot disks between servers.
  

  Workaround: None. For more information, see VMware knowledge base articles 84280 and 84349.

Broadcom-bnxt-Net-RoCE_216.0.0.0-1vmw.702.0.20.18426014

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_bnxtroce_216.0.58.0-20vmw.702.0.20.18426014 VMW_bootbank_bnxtnet_216.0.50.0-34vmw.702.0.20.18426014
PRs Fixed	N/A
CVE numbers	N/A

Updates the bnxtroce VIB.

MRVL-E4-CNA-Driver-Bundle_1.0.0.0-1vmw.702.0.20.18426014

Patch Category	Bugfix
Patch Severity	Important
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_qedrntv_3.40.5.53-17vmw.702.0.20.18426014 VMW_bootbank_qedentv_3.40.5.53-20vmw.702.0.20.18426014
PRs Fixed	N/A
CVE numbers	N/A

Updates the qedrntv​ VIB.

Intel-i40en_1.8.1.137-1vmw.702.0.20.18426014

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014
PRs Fixed	2750390
CVE numbers	N/A

Updates the i40enu VIB to resolve the following issue:

• PR 2750390: After an upgrade to ESXi 7.0 Update 2a, VIBs of the async i40en network drivers for ESXi are skipped or reverted to the VMware inbox driver i40enu

  Starting with vSphere 7.0 Update 2, the inbox i40en driver was renamed to i40enu. As a result, if you attempt to install an i40en partner async driver, the i40en VIB is either skipped or reverted to the VMware i40enu inbox driver.

  This issue is resolved in this release.

Broadcom-ELX-lpfc_12.8.298.3-2vmw.702.0.20.18426014

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_lpfc_12.8.298.3-2vmw.702.0.20.18426014
PRs Fixed	2715138
CVE numbers	N/A

Updates the lpfc VIB to resolve the following issue:

• PR 2715138: ESXi hosts might fail with purple diagnostic screen during installation of Dell EMC PowerPath plug-in

  A rare Page Fault error might cause ESXi hosts to fail with purple diagnostic screen during installation of Dell EMC PowerPath plug-in.

  This issue is resolved in this release.

VMware-vmkusb_0.1-4vmw.702.0.20.18426014

Patch Category	Bugfix
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_vmkusb_0.1-4vmw.702.0.20.18426014
PRs Fixed	2753546
CVE numbers	N/A

Updates the vmkusb VIB to resolve the following issues:

• PR 2753546: If you use a USB boot device, connection to the /bootbank partition might break or the VMFS-L LOCKER partition gets corrupted

  If the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the device path cannot be released by the time the device reconnects. As a result, ESXi provides a new path to the USB device, which breaks connection to the /bootbank partition or corrupts the VMFS-L LOCKER partition.

  This issue is resolved in this release. The fix enhances the USB host controller driver to tolerate command timeout failures to allow ESXi sufficient time to release USB device resources.
  As a best practice, do not set dump partition on USB storage device. For more information, see VMware knowledge base articles 2077516 and 2149257.

ESXi_7.0.2-0.15.18295176

Patch Category	Security
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_vdfs_7.0.2-0.15.18295176 VMware_bootbank_esx-xserver_7.0.2-0.15.18295176 VMware_bootbank_esx-base_7.0.2-0.15.18295176 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.15.18295176 VMware_bootbank_crx_7.0.2-0.15.18295176 VMware_bootbank_native-misc-drivers_7.0.2-0.15.18295176 VMware_bootbank_cpu-microcode_7.0.2-0.15.18295176 VMware_bootbank_gc_7.0.2-0.15.18295176 VMware_bootbank_vsanhealth_7.0.2-0.15.18295176 VMware_bootbank_clusterstore_7.0.2-0.15.18295176 VMware_bootbank_vsan_7.0.2-0.15.18295176
PRs Fixed	2738816, 2738857, 2735594
CVE numbers	N/A

The ESXi and esx-update bulletins are dependent on each other. Always include both in a single ESXi host patch baseline or include the rollup bulletin in the baseline to avoid failure during host patching.

Updates the vdfs, esx-xserver, esx-base, esx-dvfilter-generic-fastpath, crx, native-misc-drivers, cpu-microcode, gc, vsanhealth, clusterstore, vsan, and esx-base VIBs to resolve the following issues:

• Update to OpenSSH

  OpenSSH is updated to version 8.5p1.

• Update to the OpenSSL library

  The OpenSSL library is updated to version openssl-1.0.2y.

• Update to the Libarchive library

  The Libarchive library is updated to version 3.5.1.

esx-update_7.0.2-0.15.18295176

Patch Category	Security
Patch Severity	Critical
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_bootbank_loadesx_7.0.2-0.15.18295176 VMware_bootbank_esx-update_7.0.2-0.15.18295176
PRs Fixed	N/A
CVE numbers	N/A

Updates the loadesx and esx-update VIBs.

VMware-VM-Tools_11.2.6.17901274-18295176

Patch Category	Security
Patch Severity	Critical
Host Reboot Required	No
Virtual Machine Migration or Shutdown Required	No
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMware_locker_tools-light_11.2.6.17901274-18295176
PRs Fixed	2761942
CVE numbers	N/A

Updates the tools-light​ VIB to resolve the following issue:

• The following VMware Tools 11.2.6 ISO images are bundled with ESXi:

  • windows.iso: VMware Tools image for Windows 7 SP1 or Windows Server 2008 R2 SP1 or later
  • linux.iso: VMware Tools 10.3.23 image for older versions of Linux OS with glibc 2.5 or later

  The following VMware Tools 10.3.10 ISO images are available for download:

  • solaris.iso: VMware Tools image for Solaris
  • darwin.iso: VMware Tools image for Mac OS X versions 10.11 and later.

  Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

  • VMware Tools 11.2.6 Release Notes
  • What Every vSphere Admin Must Know About VMware Tools
  • VMware Tools for hosts provisioned with Auto Deploy
  • Updating VMware Tools

VMware-vmkusb_0.1-1vmw.702.0.15.18295176

Patch Category	Security
Patch Severity	Moderate
Host Reboot Required	Yes
Virtual Machine Migration or Shutdown Required	Yes
Affected Hardware	N/A
Affected Software	N/A
VIBs Included	VMW_bootbank_vmkusb_0.1-1vmw.702.0.15.18295176
PRs Fixed	N/A
CVE numbers	N/A

Updates the vmkusb VIB.

ESXi-7.0U2c-18426014-standard

Profile Name	ESXi-7.0U2c-18426014-standard
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	August 24, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsanhealth_7.0.2-0.20.18426014 VMware_bootbank_vsan_7.0.2-0.20.18426014 VMware_bootbank_gc_7.0.2-0.20.18426014 VMware_bootbank_esx-xserver_7.0.2-0.20.18426014 VMware_bootbank_clusterstore_7.0.2-0.20.18426014 VMware_bootbank_vdfs_7.0.2-0.20.18426014 VMware_bootbank_native-misc-drivers_7.0.2-0.20.18426014 VMware_bootbank_cpu-microcode_7.0.2-0.20.18426014 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.20.18426014 VMware_bootbank_esx-base_7.0.2-0.20.18426014 VMware_bootbank_loadesx_7.0.2-0.20.18426014 VMware_bootbank_esx-update_7.0.2-0.20.18426014 VMW_bootbank_bnxtroce_216.0.58.0-20vmw.702.0.20.18426014 VMW_bootbank_bnxtnet_216.0.50.0-34vmw.702.0.20.18426014 VMW_bootbank_qedrntv_3.40.5.53-17vmw.702.0.20.18426014 VMW_bootbank_qedentv_3.40.5.53-20vmw.702.0.20.18426014 VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014 VMW_bootbank_lpfc_12.8.298.3-2vmw.702.0.20.18426014 VMW_bootbank_vmkusb_0.1-4vmw.702.0.20.18426014
PRs Fixed	2717698, 2731446, 2515171, 731306, 2751564, 2718934, 2755056, 2725886, 2725886, 2753231, 2728910, 2741111, 2759343, 2721379, 2751277, 2708326, 2755977, 2737934, 2777001, 2760932, 2731263, 2731142, 2760267, 2749688, 2763986, 2765534, 2766127, 2777003, 2778793, 2755807, 2760081, 2731644, 2749815, 2749962, 2738238, 2799408, 2722263, 2812906, 2750390, 2715138, 2753546
Related CVE numbers	N/A

This patch updates the following issues:

• • A rare race condition in the qedentv driver might cause an ESXi host to fail with a purple diagnostic screen. The issue occurs when an Rx complete interrupt arrives just after a General Services Interface (GSI) queue pair (QP) is destroyed, for example during a qedentv driver unload or a system shut down. In such a case, the qedentv driver might access an already freed QP address that leads to a PF exception. The issue might occur in ESXi hosts that are connected to a busy physical switch with heavy unsolicited GSI traffic. In the backtrace, you see messages such as:

    cpu4:2107287)0x45389609bcb0:[0x42001d3e6f72]qedrntv_ll2_rx_cb@(qedrntv)#<None>+0x1be stack: 0x45b8f00a7740, 0x1e146d040, 0x432d65738d40, 0x0, 0x
2021-02-11T03:31:53.882Z cpu4:2107287)0x45389609bd50:[0x42001d421d2a]ecore_ll2_rxq_completion@(qedrntv)#<None>+0x2ab stack: 0x432bc20020ed, 0x4c1e74ef0, 0x432bc2002000,
2021-02-11T03:31:53.967Z cpu4:2107287)0x45389609bdf0:[0x42001d1296d0]ecore_int_sp_dpc@(qedentv)#<None>+0x331 stack: 0x0, 0x42001c3bfb6b, 0x76f1e5c0, 0x2000097, 0x14c2002
2021-02-11T03:31:54.039Z cpu4:2107287)0x45389609be60:[0x42001c0db867]IntrCookieBH@vmkernel#nover+0x17c stack: 0x45389609be80, 0x40992f09ba, 0x43007a436690, 0x43007a43669
2021-02-11T03:31:54.116Z cpu4:2107287)0x45389609bef0:[0x42001c0be6b0]BH_Check@vmkernel#nover+0x121 stack: 0x98ba, 0x33e72f6f6e20, 0x0, 0x8000000000000000, 0x430000000001
2021-02-11T03:31:54.187Z cpu4:2107287)0x45389609bf70:[0x42001c28370c]NetPollWorldCallback@vmkernel#nover+0x129 stack: 0x61, 0x42001d0e0000, 0x42001c283770, 0x0, 0x0
2021-02-11T03:31:54.256Z cpu4:2107287)0x45389609bfe0:[0x42001c380bad]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001c0c2b44, 0x0, 0x0, 0x0
2021-02-11T03:31:54.319Z cpu4:2107287)0x45389609c000:[0x42001c0c2b43]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
2021-02-11T03:31:54.424Z cpu4:2107287)^[[45m^[[33;1mVMware ESXi 7.0.2 [Releasebuild-17435195 x86_64]^[[0m
#PF Exception 14 in world 2107287:vmnic7-pollW IP 0x42001d3e6f72 addr 0x1c

  • In rare cases, while the vSphere Replication appliance powers on, the hostd service might fail and cause temporary unresponsiveness of ESXi hosts to vCenter Server. The hostd service restarts automatically and connectivity restores.

  • When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  • ESXi hosts might become unresponsive to vCenter Server even when the hosts are accessible and running. The issue occurs in vendor images with customized drivers where ImageConfigManager consumes more RAM than allocated. As a result, repeated failures of ImageConfigManager cause ESXi hosts to disconnect from vCenter Server.
    In the vmkernel logs, you see repeating errors such as:
    Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.<pid1>:python.<pid2>:uw.<pid2>.
    In the hostd logs, you see messages such as: Task Created : haTask-ha-host-vim.host.ImageConfigManager.fetchSoftwarePackages-<sequence>, followed by: ForkExec(/usr/bin/sh) <pid1>, where <pid1> identifies an ImageConfigManager process.

  • If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  • In certain conditions, the power on or power off operations of virtual machines might take as long as 20 minutes. The issue occurs when the underlying storage of the VMs runs a background operation such as HBA rescan or APD events, and some locks are continuously held.

  • In rare cases, when the VPD page response size from the target ESXi host is different on different paths to the host, ESXi might write more bytes than the allocated length for a given path. As a result, the ESXi host fails with a purple diagnostic screen and a message such as:
    Panic Message: @BlueScreen: PANIC bora/vmkernel/main/dlmalloc.c:4878 - Corruption in dlmalloc

  • A rare lock contention might cause virtual machine operations such as failover or migration to fail.
    Locking failure messages look similar to:
    vol 'VM_SYS_11', lock at 480321536: [Req mode 1] Checking liveness:
type 10c00003 offset 480321536 v 1940464, hb offset 3801088
    or
    Res3: 2496: Rank violation threshold reached: cid 0xc1d00002, resType 2, cnum 5 vol VM_SYS_11

  • In complicated workflows that result in creating multiple virtual machines, when you navigate to VMs > Virtual Machines, you might see huge numbers in the Provisioned Space column for some VMs. For example, 2.95 TB instead of the actual provisioned space of 25 GB.

  • While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  • In very rare cases, VMFS resource clusters that are included in a journal transaction might not be locked. As a result, during power on of virtual machines, multiple ESXi hosts might fail with a purple diagnostic screen due to exception 14 in the VMFS layer. A typical message and stack trace seen the vmkernel log is as follows:
    @BlueScreen: #PF Exception 14 in world 2097684:VSCSI Emulat IP 0x418014d06fca addr 0x0
    PTEs:0x16a47a027;0x600faa8007;0x0;
    2020-06-24T17:35:57.073Z cpu29:2097684)Code start: 0x418013c00000 VMK uptime: 0:01:01:20.555
2020-06-24T17:35:57.073Z cpu29:2097684)0x451a50a1baa0:[0x418014d06fca]Res6MemUnlockTxnRCList@esx#nover+0x176 stack: 0x1 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb10:[0x418014c7cdb6]J3_DeleteTransaction@esx#nover+0x33f stack: 0xbad0003
2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb40:[0x418014c7db10]J3_AbortTransaction@esx#nover+0x105 stack: 0x0 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb80:[0x418014cbb752]Fil3_FinalizePunchFileHoleTxnVMFS6@esx#nover+0x16f stack: 0x430fe950e1f0
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bbd0:[0x418014c7252b]Fil3UpdateBlocks@esx#nover+0x348 stack: 0x451a50a1bc78 2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bce0:[0x418014c731dc]Fil3_PunchFileHoleWithRetry@esx#nover+0x89 stack: 0x451a50a1bec8
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bd90:[0x418014c739a5]Fil3_FileBlockUnmap@esx#nover+0x50e stack: 0x230eb5 2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be40:[0x418013c4c551]FSSVec_FileBlockUnmap@vmkernel#nover+0x6e stack: 0x230eb5
2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be90:[0x418013fb87b1]VSCSI_ExecFSSUnmap@vmkernel#nover+0x8e stack: 0x0 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf00:[0x418013fb71cb]VSCSIDoEmulHelperIO@vmkernel#nover+0x2c stack: 0x430145fbe070
2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf30:[0x418013ceadfa]HelperQueueFunc@vmkernel#nover+0x157 stack: 0x430aa05c9618 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bfe0:[0x418013f0eaa2]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0
2020-06-24T17:35:57.083Z cpu29:2097684)base fs=0x0 gs=0x418047400000 Kgs=0x0

  • In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault.

  • If you restart ESXi management agents from DCUI, the sfcbd service does not start. You must manually start the SFCB service.

  • The hostd service logs the message BuildPowerSupplySensorList found 0 power supply sensors every 2 minutes. The issue occurs when you upgrade to or fresh install ESXi 7.0 Update 1 on an ESXi host.

  • In rare cases, the kernel memory allocator might return a NULL pointer that might not be correctly dereferenced. As a result, the ESXi host might fail with a purple diagnostic screen with an error such as #GP Exception 13 in world 2422594:vmm:overCom @ 0x42003091ede4. In the backtrace, you see errors similar to:
    #1 Util_ZeroPageTable
#2 Util_ZeroPageTableMPN
#3 VmMemPfUnmapped
#4 VmMemPfInt
#5 VmMemPfGetMapping
#6 VmMemPf
#7 VmMemPfLockPageInt
#8 VMMVMKCall_Call
#9 VMKVMM_ArchEnterVMKernel

  • If an NVMe device is hot added and hot removed in a short interval, the NVMe driver might fail to initialize the NVMe controller due to a command timeout. As a result, the driver might access memory that is already freed in a cleanup process. In the backtrace, you see a message such as WARNING: NVMEDEV: NVMEInitializeController:4045: Failed to get controller identify data, status: Timeout.
    Eventually, the ESXi host might fail with a purple diagnostic screen with an error similar to #PF Exception ... in world ...:vmkdevmgr.

  • When an ESXi host does not have free large file blocks (LFBs) to allocate, the host fulfills thick VMDK provisioning or the creation of a swap file with small file blocks (SFBs). However, in rare cases, the host might fail to allocate SFBs as well. As a result, thick VMDK provisioning or the creation of a swap file fails. When virtual machines try to power on, you see an error in the vmkernel logs such as:
     vmkernel.1:2021-01-08T19:13:28.341Z cpu20:2965260)Fil3: 10269: Max no space retries (10) exceeded for caller Fil3_SetFileLength (status 'No space left on device')

  • When you use very large VMFS6 datastores, the process of allocating file blocks for thin VMDKs in a resource cluster might cause a CPU lockup. As a result, you see CPU lockup messages and in rare cases, ESXi hosts might fail. In the backtrace, you see messages such as:
    2021-04-07T02:18:08.730Z cpu31:2345212)WARNING: Heartbeat: 849: PCPU 10 didn't have a heartbeat for 7 seconds; *may* be locked up.
    The associated backtrace involving Res6AffMgr functions looks similar to:
    2021-01-06T02:16:16.073Z cpu0:2265741)ALERT: NMI: 694: NMI IPI: RIPOFF(base):RBP:CS [0x121d74d(0x42001a000000):0x50d0:0xf48] (Src 0x1, CPU0)
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b320:[0x42001b21d74c]Res6AffMgrComputeSortIndices@esx#nover+0x369 stack: 0x43119125a000
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b3b0:[0x42001b224e46]Res6AffMgrGetCluster@esx#nover+0xb1f stack: 0xd 2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b4b0:[0x42001b226491]Res6AffMgr_AllocResourcesInt@esx#nover+0x40a stack: 0x4311910f4890
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b680:[0x42001b2270e2]Res6AffMgr_AllocResources@esx#nover+0x1b stack: 0x0

  • During or after an upgrade to ESXi 7.0 Update 2, the ESX storage layer might not allocate sufficient memory resources for ESXi hosts with a large physical CPU count and many storage devices or paths connected to the hosts. As a result, such ESXi hosts might fail with a purple diagnostic screen.

  • Virtual machines with hardware version 18 and VMware Tools 11.2.0 or later might fail due to a virtual graphics device issue on the destination side during a vSphere vMotion or Storage vMotion.
    In the vmkernel.log you see a line such as:
    PF failed to handle a fault on mmInfo at va 0x114f5ae0: Out of memory. Terminating...

  • After updating to hardware version 18, some guest OS, such as CentOS 7, on AMD CPUs might fail with kernel panic when you boot the virtual machine. You see the kernel panic message when you open a web console for the virtual machine.

  • Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines, because of the IPv6 extension header.

  • When virtual media, such as a CD drive, is connected to your vSphere environment by using an Integrated Dell Remote Access Controller (iDRAC), and no media file is mapped to the drive, in the vmkernel log you might see multiple warning messages such as:
    vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L1 in STANDBY state for device mpx.vmhba32:C0:T0:L1.WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L1 for device mpx.vmhba32:C0:T0:L1 to activate. This may lead to path thrashing.

  • During 24-hour sync workflows of the vSphere Distributed Switch, vCenter Server might intermittently remove NSX-controlled distributed virtual ports such as the Service Plane Forwarding (SPF) port, from ESXi hosts. As a result, vSphere vMotion operations of virtual machines might fail. You see an error such as Could not connect SPF port : Not found in the logs.

  • ESXi 7.0 introduced the VMFS-L based ESX-OSData partition, which takes on the role of the legacy scratch partition, locker partition for VMware Tools, and core dump destination. If you configure the scratch partition on the root directory of a VMFS volume during an upgrade to ESXi 7.0.x, the system tries to convert the VMFS files into VMFS-L format to match the OS-Data partition requirements. As a result, the OS-Data partition might overwrite the VMFS volume and cause data loss.

  • Due to a log handling issue, after an upgrade to ESXi 7.0 Update 2a, when you check the Communication Channel health status you see all ESXi hosts with status down, because NSX Manager fails to connect to the Firewall Agent.

  • If an ESXi host of version 7.0 Update 2 is installed on a FCoE LUN and uses UEFI boot mode, when you try to upgrade the host by using vSphere QuickBoot, the physical server might fail with a purple diagnostic screen because of a memory error.

  • USB devices have a small queue depth and due to a race condition in the ESXi storage stack, some I/O operations might not get to the device. Such I/Os queue in the ESXi storage stack and ultimately time out. As a result, ESXi hosts become unresponsive.
    In the vSphere Client, you see alerts such as Alert: /bootbank not to be found at path '/bootbank' and Host not-responding.
    In vmkernel logs, you see errors such as:
    2021-04-12T04:47:44.940Z cpu0:2097441)ScsiPath: 8058: Cancelled Cmd(0x45b92ea3fd40) 0xa0, cmdId.initiator=0x4538c859b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-12T04:48:50.527Z cpu2:2097440)ScsiDeviceIO: 4315: Cmd(0x45b92ea76d40) 0x28, cmdId.initiator=0x4305f74cc780 CmdSN 0x1279 from world 2099370 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-12T04:48:50.527Z cpu2:2097440)Queued:4

  • You might see a slowdown of up to 60 min in the booting time of ESXi hosts after an upgrade to ESXi Update 2a.
    The issue does not affect updates to ESXi Update 2a from earlier versions of ESXi 7.0.x.
    The issue occurs only when you use a vSphere Lifecycle Manager image or baseline, or the esxcli software profile update command to perform the upgrade operation. If you use an ISO image or a scripted installation, you do not encounter the problem.
    The issue is most likely to affect iSCSI configurations, but is related to the ESXi algorithm for boot bank detection, not to slow external storage targets.

  • Due to a memory leak in the portset heap, vSphere vMotion operations on an ESXi host might fail with an out of memory error. In the vSphere Client, you see an error such as:
    Failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 1 39.
Failed with error status: Out of memory.

  • If vCenter Server cannot resolve vcsa.vmware.com to an IP address, the vSAN release catalog cannot be uploaded. The following vSAN heath check displays a warning: vSAN Build Recommendation Engine Health.
    You might see an error message similar to the following:
    2021-04-12T11:40:59.435Z ERROR vsan-mgmt[32221] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z WARNING vsan-mgmt[32221] [VsanVumCoreUtil::CheckVumServiceStatus opID=noOpId] Failed to connect VUM service: Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumIntegration::VumStatusCallback opID=noOpId] VUM is not installed or not responsive ​
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumSystemUtil::HasConfigIssue opID=noOpId] Config issue health.test.vum.vumDisabled exists on entity None

  • An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  • When a resync runs in parallel with guest writes on objects owned by an ESXi host, the host might fail with a purple diagnostic screen.

  • When the cmmdstimemachineDump.log file reaches a max limit of 10 MB, rotation happens and a new cmmdstimemachineDump file is created. However, during the rotation, in the vmsyslogd.err logs you might see an error such as:
    2021-04-08T03:15:53.719Z vmsyslog.loggers.file : ERROR ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: '/var/run/cmmdstimemachineDumpHeader.txt'

  • In some cases, the cluster monitoring, membership, and directory service (CMMDS) does not provide correct input to the Object Format Change task and the task takes long to respond.

  • In rare scenarios, if the witness host is replaced during the upgrade process and a Disk Format Conversion task runs shortly after the replacement, multiple ESX hosts on a stretched cluster might fail with a purple diagnostic screen.

  • If you create a custom 7.0 Update 2a image by using the ESXi Packaging Kit, override some of the Components in the base image, for example Intel-ixgben:1.8.9.0-1OEM.700.1.0.15525992, and then upgrade your ESXi 7.0.x hosts, you cannot remove the custom Component after the upgrade. Also, some VIBs from the custom Components might be missing.

  • When you clone an ESXi boot device, you create instances with identical UUIDs. Since UUIDs are used for VMFS Heartbeat and Journal operations, when you have duplicating UUIDS, after installation or upgrade operations multiple ESXi hosts attempt to access metadata regions on the same VMFS datastore. As a result, you might see a massive metadata corruption in VMFS datastores on multiple ESXi hosts.
    In the vmkernel logs, you see messages such as:
    vmkernel 608: VMFS volume DCS_HCFA_TEDS_Regulated_10032/5ba2905b-ac11166d-b145-0025b5920a02 on naa.60060e8012a34f005040a34f00000d64:1 has been detected corrupted.
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 319: FS3RCMeta 2245 200 21 105 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 326: 0 0 0 0 0 0 0 0 0 0 0 7 0 254 1 4 0 248 3 96 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 332: 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 338: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

  • Starting with vSphere 7.0 Update 2, the inbox i40en driver was renamed to i40enu. As a result, if you attempt to install an i40en partner async driver, the i40en VIB is either skipped or reverted to the VMware i40enu inbox driver.

  • A rare Page Fault error might cause ESXi hosts to fail with purple diagnostic screen during installation of Dell EMC PowerPath plug-in.

  • If the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the device path cannot be released by the time the device reconnects. As a result, ESXi provides a new path to the USB device, which breaks connection to the /bootbank partition or corrupts the VMFS-L LOCKER partition.

ESXi-7.0U2c-18426014-no-tools

Profile Name	ESXi-7.0U2c-18426014-no-tools
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	August 24, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vsanhealth_7.0.2-0.20.18426014 VMware_bootbank_vsan_7.0.2-0.20.18426014 VMware_bootbank_gc_7.0.2-0.20.18426014 VMware_bootbank_esx-xserver_7.0.2-0.20.18426014 VMware_bootbank_clusterstore_7.0.2-0.20.18426014 VMware_bootbank_vdfs_7.0.2-0.20.18426014 VMware_bootbank_native-misc-drivers_7.0.2-0.20.18426014 VMware_bootbank_cpu-microcode_7.0.2-0.20.18426014 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.20.18426014 VMware_bootbank_esx-base_7.0.2-0.20.18426014 VMware_bootbank_loadesx_7.0.2-0.20.18426014 VMware_bootbank_esx-update_7.0.2-0.20.18426014 VMW_bootbank_bnxtroce_216.0.58.0-20vmw.702.0.20.18426014 VMW_bootbank_bnxtnet_216.0.50.0-34vmw.702.0.20.18426014 VMW_bootbank_qedrntv_3.40.5.53-17vmw.702.0.20.18426014 VMW_bootbank_qedentv_3.40.5.53-20vmw.702.0.20.18426014 VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014 VMW_bootbank_lpfc_12.8.298.3-2vmw.702.0.20.18426014 VMW_bootbank_vmkusb_0.1-4vmw.702.0.20.18426014
PRs Fixed	2717698, 2731446, 2515171, 731306, 2751564, 2718934, 2755056, 2725886, 2725886, 2753231, 2728910, 2741111, 2759343, 2721379, 2751277, 2708326, 2755977, 2737934, 2777001, 2760932, 2731263, 2731142, 2760267, 2749688, 2763986, 2765534, 2766127, 2777003, 2778793, 2755807, 2760081, 2731644, 2749815, 2749962, 2738238, 2799408, 2722263, 2812906, 2750390, 2715138, 2753546
Related CVE numbers	N/A

This patch updates the following issues:

• • A rare race condition in the qedentv driver might cause an ESXi host to fail with a purple diagnostic screen. The issue occurs when an Rx complete interrupt arrives just after a General Services Interface (GSI) queue pair (QP) is destroyed, for example during a qedentv driver unload or a system shut down. In such a case, the qedentv driver might access an already freed QP address that leads to a PF exception. The issue might occur in ESXi hosts that are connected to a busy physical switch with heavy unsolicited GSI traffic. In the backtrace, you see messages such as:

    cpu4:2107287)0x45389609bcb0:[0x42001d3e6f72]qedrntv_ll2_rx_cb@(qedrntv)#<None>+0x1be stack: 0x45b8f00a7740, 0x1e146d040, 0x432d65738d40, 0x0, 0x
2021-02-11T03:31:53.882Z cpu4:2107287)0x45389609bd50:[0x42001d421d2a]ecore_ll2_rxq_completion@(qedrntv)#<None>+0x2ab stack: 0x432bc20020ed, 0x4c1e74ef0, 0x432bc2002000,
2021-02-11T03:31:53.967Z cpu4:2107287)0x45389609bdf0:[0x42001d1296d0]ecore_int_sp_dpc@(qedentv)#<None>+0x331 stack: 0x0, 0x42001c3bfb6b, 0x76f1e5c0, 0x2000097, 0x14c2002
2021-02-11T03:31:54.039Z cpu4:2107287)0x45389609be60:[0x42001c0db867]IntrCookieBH@vmkernel#nover+0x17c stack: 0x45389609be80, 0x40992f09ba, 0x43007a436690, 0x43007a43669
2021-02-11T03:31:54.116Z cpu4:2107287)0x45389609bef0:[0x42001c0be6b0]BH_Check@vmkernel#nover+0x121 stack: 0x98ba, 0x33e72f6f6e20, 0x0, 0x8000000000000000, 0x430000000001
2021-02-11T03:31:54.187Z cpu4:2107287)0x45389609bf70:[0x42001c28370c]NetPollWorldCallback@vmkernel#nover+0x129 stack: 0x61, 0x42001d0e0000, 0x42001c283770, 0x0, 0x0
2021-02-11T03:31:54.256Z cpu4:2107287)0x45389609bfe0:[0x42001c380bad]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x42001c0c2b44, 0x0, 0x0, 0x0
2021-02-11T03:31:54.319Z cpu4:2107287)0x45389609c000:[0x42001c0c2b43]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0
2021-02-11T03:31:54.424Z cpu4:2107287)^[[45m^[[33;1mVMware ESXi 7.0.2 [Releasebuild-17435195 x86_64]^[[0m
#PF Exception 14 in world 2107287:vmnic7-pollW IP 0x42001d3e6f72 addr 0x1c

  • In rare cases, while the vSphere Replication appliance powers on, the hostd service might fail and cause temporary unresponsiveness of ESXi hosts to vCenter Server. The hostd service restarts automatically and connectivity restores.

  • When attempting to set a value to false for an advanced option parameter in a host profile, the user interface creates a non-empty string value. Values that are not empty are interpreted as true and the advanced option parameter receives a true value in the host profile.

  • ESXi hosts might become unresponsive to vCenter Server even when the hosts are accessible and running. The issue occurs in vendor images with customized drivers where ImageConfigManager consumes more RAM than allocated. As a result, repeated failures of ImageConfigManager cause ESXi hosts to disconnect from vCenter Server.
    In the vmkernel logs, you see repeating errors such as:
    Admission failure in path: host/vim/vmvisor/hostd-tmp/sh.<pid1>:python.<pid2>:uw.<pid2>.
    In the hostd logs, you see messages such as: Task Created : haTask-ha-host-vim.host.ImageConfigManager.fetchSoftwarePackages-<sequence>, followed by: ForkExec(/usr/bin/sh) <pid1>, where <pid1> identifies an ImageConfigManager process.

  • If you change the DiskMaxIOSize advanced config option to a lower value, I/Os with large block sizes might get incorrectly split and queue at the PSA path. As a result, ESXi hosts I/O operations might time out and fail.

  • In certain conditions, the power on or power off operations of virtual machines might take as long as 20 minutes. The issue occurs when the underlying storage of the VMs runs a background operation such as HBA rescan or APD events, and some locks are continuously held.

  • In rare cases, when the VPD page response size from the target ESXi host is different on different paths to the host, ESXi might write more bytes than the allocated length for a given path. As a result, the ESXi host fails with a purple diagnostic screen and a message such as:
    Panic Message: @BlueScreen: PANIC bora/vmkernel/main/dlmalloc.c:4878 - Corruption in dlmalloc

  • A rare lock contention might cause virtual machine operations such as failover or migration to fail.
    Locking failure messages look similar to:
    vol 'VM_SYS_11', lock at 480321536: [Req mode 1] Checking liveness:
type 10c00003 offset 480321536 v 1940464, hb offset 3801088
    or
    Res3: 2496: Rank violation threshold reached: cid 0xc1d00002, resType 2, cnum 5 vol VM_SYS_11

  • In complicated workflows that result in creating multiple virtual machines, when you navigate to VMs > Virtual Machines, you might see huge numbers in the Provisioned Space column for some VMs. For example, 2.95 TB instead of the actual provisioned space of 25 GB.

  • While handling the SCSI command READ CAPACITY (10), ESXi might copy excess data from the response and corrupt the call stack. As a result, the ESXi host fails with a purple diagnostic screen.

  • In very rare cases, VMFS resource clusters that are included in a journal transaction might not be locked. As a result, during power on of virtual machines, multiple ESXi hosts might fail with a purple diagnostic screen due to exception 14 in the VMFS layer. A typical message and stack trace seen the vmkernel log is as follows:
    @BlueScreen: #PF Exception 14 in world 2097684:VSCSI Emulat IP 0x418014d06fca addr 0x0
    PTEs:0x16a47a027;0x600faa8007;0x0;
    2020-06-24T17:35:57.073Z cpu29:2097684)Code start: 0x418013c00000 VMK uptime: 0:01:01:20.555
2020-06-24T17:35:57.073Z cpu29:2097684)0x451a50a1baa0:[0x418014d06fca]Res6MemUnlockTxnRCList@esx#nover+0x176 stack: 0x1 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb10:[0x418014c7cdb6]J3_DeleteTransaction@esx#nover+0x33f stack: 0xbad0003
2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb40:[0x418014c7db10]J3_AbortTransaction@esx#nover+0x105 stack: 0x0 2020-06-24T17:35:57.074Z cpu29:2097684)0x451a50a1bb80:[0x418014cbb752]Fil3_FinalizePunchFileHoleTxnVMFS6@esx#nover+0x16f stack: 0x430fe950e1f0
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bbd0:[0x418014c7252b]Fil3UpdateBlocks@esx#nover+0x348 stack: 0x451a50a1bc78 2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bce0:[0x418014c731dc]Fil3_PunchFileHoleWithRetry@esx#nover+0x89 stack: 0x451a50a1bec8
2020-06-24T17:35:57.075Z cpu29:2097684)0x451a50a1bd90:[0x418014c739a5]Fil3_FileBlockUnmap@esx#nover+0x50e stack: 0x230eb5 2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be40:[0x418013c4c551]FSSVec_FileBlockUnmap@vmkernel#nover+0x6e stack: 0x230eb5
2020-06-24T17:35:57.076Z cpu29:2097684)0x451a50a1be90:[0x418013fb87b1]VSCSI_ExecFSSUnmap@vmkernel#nover+0x8e stack: 0x0 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf00:[0x418013fb71cb]VSCSIDoEmulHelperIO@vmkernel#nover+0x2c stack: 0x430145fbe070
2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bf30:[0x418013ceadfa]HelperQueueFunc@vmkernel#nover+0x157 stack: 0x430aa05c9618 2020-06-24T17:35:57.077Z cpu29:2097684)0x451a50a1bfe0:[0x418013f0eaa2]CpuSched_StartWorld@vmkernel#nover+0x77 stack: 0x0
2020-06-24T17:35:57.083Z cpu29:2097684)base fs=0x0 gs=0x418047400000 Kgs=0x0

  • In rare cases, some Intel CPUs might fail to forward #DB traps and if a timer interrupt happens during the Windows system call, virtual machine might triple fault.

  • If you restart ESXi management agents from DCUI, the sfcbd service does not start. You must manually start the SFCB service.

  • The hostd service logs the message BuildPowerSupplySensorList found 0 power supply sensors every 2 minutes. The issue occurs when you upgrade to or fresh install ESXi 7.0 Update 1 on an ESXi host.

  • In rare cases, the kernel memory allocator might return a NULL pointer that might not be correctly dereferenced. As a result, the ESXi host might fail with a purple diagnostic screen with an error such as #GP Exception 13 in world 2422594:vmm:overCom @ 0x42003091ede4. In the backtrace, you see errors similar to:
    #1 Util_ZeroPageTable
#2 Util_ZeroPageTableMPN
#3 VmMemPfUnmapped
#4 VmMemPfInt
#5 VmMemPfGetMapping
#6 VmMemPf
#7 VmMemPfLockPageInt
#8 VMMVMKCall_Call
#9 VMKVMM_ArchEnterVMKernel

  • If an NVMe device is hot added and hot removed in a short interval, the NVMe driver might fail to initialize the NVMe controller due to a command timeout. As a result, the driver might access memory that is already freed in a cleanup process. In the backtrace, you see a message such as WARNING: NVMEDEV: NVMEInitializeController:4045: Failed to get controller identify data, status: Timeout.
    Eventually, the ESXi host might fail with a purple diagnostic screen with an error similar to #PF Exception ... in world ...:vmkdevmgr.

  • When an ESXi host does not have free large file blocks (LFBs) to allocate, the host fulfills thick VMDK provisioning or the creation of a swap file with small file blocks (SFBs). However, in rare cases, the host might fail to allocate SFBs as well. As a result, thick VMDK provisioning or the creation of a swap file fails. When virtual machines try to power on, you see an error in the vmkernel logs such as:
     vmkernel.1:2021-01-08T19:13:28.341Z cpu20:2965260)Fil3: 10269: Max no space retries (10) exceeded for caller Fil3_SetFileLength (status 'No space left on device')

  • When you use very large VMFS6 datastores, the process of allocating file blocks for thin VMDKs in a resource cluster might cause a CPU lockup. As a result, you see CPU lockup messages and in rare cases, ESXi hosts might fail. In the backtrace, you see messages such as:
    2021-04-07T02:18:08.730Z cpu31:2345212)WARNING: Heartbeat: 849: PCPU 10 didn't have a heartbeat for 7 seconds; *may* be locked up.
    The associated backtrace involving Res6AffMgr functions looks similar to:
    2021-01-06T02:16:16.073Z cpu0:2265741)ALERT: NMI: 694: NMI IPI: RIPOFF(base):RBP:CS [0x121d74d(0x42001a000000):0x50d0:0xf48] (Src 0x1, CPU0)
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b320:[0x42001b21d74c]Res6AffMgrComputeSortIndices@esx#nover+0x369 stack: 0x43119125a000
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b3b0:[0x42001b224e46]Res6AffMgrGetCluster@esx#nover+0xb1f stack: 0xd 2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b4b0:[0x42001b226491]Res6AffMgr_AllocResourcesInt@esx#nover+0x40a stack: 0x4311910f4890
2021-01-06T02:16:16.073Z cpu0:2265741)0x4538d261b680:[0x42001b2270e2]Res6AffMgr_AllocResources@esx#nover+0x1b stack: 0x0

  • During or after an upgrade to ESXi 7.0 Update 2, the ESX storage layer might not allocate sufficient memory resources for ESXi hosts with a large physical CPU count and many storage devices or paths connected to the hosts. As a result, such ESXi hosts might fail with a purple diagnostic screen.

  • Virtual machines with hardware version 18 and VMware Tools 11.2.0 or later might fail due to a virtual graphics device issue on the destination side during a vSphere vMotion or Storage vMotion.
    In the vmkernel.log you see a line such as:
    PF failed to handle a fault on mmInfo at va 0x114f5ae0: Out of memory. Terminating...

  • After updating to hardware version 18, some guest OS, such as CentOS 7, on AMD CPUs might fail with kernel panic when you boot the virtual machine. You see the kernel panic message when you open a web console for the virtual machine.

  • Some Linux kernels add the IPv6 Tunnel Encapsulation Limit option to IPv6 tunnel packets as described in the RFC 2473, par. 5.1. As a result, IPv6 tunnel packets are dropped in traffic between virtual machines, because of the IPv6 extension header.

  • When virtual media, such as a CD drive, is connected to your vSphere environment by using an Integrated Dell Remote Access Controller (iDRAC), and no media file is mapped to the drive, in the vmkernel log you might see multiple warning messages such as:
    vmw_psp_fixed: psp_fixedSelectPathToActivateInt:439: Switching to preferred path vmhba32:C0:T0:L1 in STANDBY state for device mpx.vmhba32:C0:T0:L1.WARNING: vmw_psp_fixed: psp_fixedSelectPathToActivateInt:464: Selected current STANDBY path vmhba32:C0:T0:L1 for device mpx.vmhba32:C0:T0:L1 to activate. This may lead to path thrashing.

  • During 24-hour sync workflows of the vSphere Distributed Switch, vCenter Server might intermittently remove NSX-controlled distributed virtual ports such as the Service Plane Forwarding (SPF) port, from ESXi hosts. As a result, vSphere vMotion operations of virtual machines might fail. You see an error such as Could not connect SPF port : Not found in the logs.

  • ESXi 7.0 introduced the VMFS-L based ESX-OSData partition, which takes on the role of the legacy scratch partition, locker partition for VMware Tools, and core dump destination. If you configure the scratch partition on the root directory of a VMFS volume during an upgrade to ESXi 7.0.x, the system tries to convert the VMFS files into VMFS-L format to match the OS-Data partition requirements. As a result, the OS-Data partition might overwrite the VMFS volume and cause data loss.

  • Due to a log handling issue, after an upgrade to ESXi 7.0 Update 2a, when you check the Communication Channel health status you see all ESXi hosts with status down, because NSX Manager fails to connect to the Firewall Agent.

  • If an ESXi host of version 7.0 Update 2 is installed on a FCoE LUN and uses UEFI boot mode, when you try to upgrade the host by using vSphere QuickBoot, the physical server might fail with a purple diagnostic screen because of a memory error.

  • USB devices have a small queue depth and due to a race condition in the ESXi storage stack, some I/O operations might not get to the device. Such I/Os queue in the ESXi storage stack and ultimately time out. As a result, ESXi hosts become unresponsive.
    In the vSphere Client, you see alerts such as Alert: /bootbank not to be found at path '/bootbank' and Host not-responding.
    In vmkernel logs, you see errors such as:
    2021-04-12T04:47:44.940Z cpu0:2097441)ScsiPath: 8058: Cancelled Cmd(0x45b92ea3fd40) 0xa0, cmdId.initiator=0x4538c859b8f8 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:1.
2021-04-12T04:48:50.527Z cpu2:2097440)ScsiDeviceIO: 4315: Cmd(0x45b92ea76d40) 0x28, cmdId.initiator=0x4305f74cc780 CmdSN 0x1279 from world 2099370 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-04-12T04:48:50.527Z cpu2:2097440)Queued:4

  • You might see a slowdown of up to 60 min in the booting time of ESXi hosts after an upgrade to ESXi Update 2a.
    The issue does not affect updates to ESXi Update 2a from earlier versions of ESXi 7.0.x.
    The issue occurs only when you use a vSphere Lifecycle Manager image or baseline, or the esxcli software profile update command to perform the upgrade operation. If you use an ISO image or a scripted installation, you do not encounter the problem.
    The issue is most likely to affect iSCSI configurations, but is related to the ESXi algorithm for boot bank detection, not to slow external storage targets.

  • Due to a memory leak in the portset heap, vSphere vMotion operations on an ESXi host might fail with an out of memory error. In the vSphere Client, you see an error such as:
    Failed to reserve port: DVSwitch 50 18 34 d1 76 81 ec 9e-03 62 e2 d4 6f 23 30 ba port 1 39.
Failed with error status: Out of memory.

  • If vCenter Server cannot resolve vcsa.vmware.com to an IP address, the vSAN release catalog cannot be uploaded. The following vSAN heath check displays a warning: vSAN Build Recommendation Engine Health.
    You might see an error message similar to the following:
    2021-04-12T11:40:59.435Z ERROR vsan-mgmt[32221] [VsanHttpRequestWrapper::urlopen opID=noOpId] Exception while sending request : Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z WARNING vsan-mgmt[32221] [VsanVumCoreUtil::CheckVumServiceStatus opID=noOpId] Failed to connect VUM service: Cannot resolve localhost or Internet websites.
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumIntegration::VumStatusCallback opID=noOpId] VUM is not installed or not responsive ​
2021-04-12T11:40:59.435Z INFO vsan-mgmt[32221] [VsanVumSystemUtil::HasConfigIssue opID=noOpId] Config issue health.test.vum.vumDisabled exists on entity None

  • An integer overflow bug might cause vSAN DOM to issue scrubs more frequently than the configured setting.

  • When a resync runs in parallel with guest writes on objects owned by an ESXi host, the host might fail with a purple diagnostic screen.

  • When the cmmdstimemachineDump.log file reaches a max limit of 10 MB, rotation happens and a new cmmdstimemachineDump file is created. However, during the rotation, in the vmsyslogd.err logs you might see an error such as:
    2021-04-08T03:15:53.719Z vmsyslog.loggers.file : ERROR ] Failed to write header on rotate. Exception: [Errno 2] No such file or directory: '/var/run/cmmdstimemachineDumpHeader.txt'

  • In some cases, the cluster monitoring, membership, and directory service (CMMDS) does not provide correct input to the Object Format Change task and the task takes long to respond.

  • In rare scenarios, if the witness host is replaced during the upgrade process and a Disk Format Conversion task runs shortly after the replacement, multiple ESX hosts on a stretched cluster might fail with a purple diagnostic screen.

  • If you create a custom 7.0 Update 2a image by using the ESXi Packaging Kit, override some of the Components in the base image, for example Intel-ixgben:1.8.9.0-1OEM.700.1.0.15525992, and then upgrade your ESXi 7.0.x hosts, you cannot remove the custom Component after the upgrade. Also, some VIBs from the custom Components might be missing.

  • When you clone an ESXi boot device, you create instances with identical UUIDs. Since UUIDs are used for VMFS Heartbeat and Journal operations, when you have duplicating UUIDS, after installation or upgrade operations multiple ESXi hosts attempt to access metadata regions on the same VMFS datastore. As a result, you might see a massive metadata corruption in VMFS datastores on multiple ESXi hosts.
    In the vmkernel logs, you see messages such as:
    vmkernel 608: VMFS volume DCS_HCFA_TEDS_Regulated_10032/5ba2905b-ac11166d-b145-0025b5920a02 on naa.60060e8012a34f005040a34f00000d64:1 has been detected corrupted.
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 319: FS3RCMeta 2245 200 21 105 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 326: 0 0 0 0 0 0 0 0 0 0 0 7 0 254 1 4 0 248 3 96 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 332: 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 338: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2021-06-08T15:58:22.126Z cpu1:2097600)FS3: 346: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

  • Starting with vSphere 7.0 Update 2, the inbox i40en driver was renamed to i40enu. As a result, if you attempt to install an i40en partner async driver, the i40en VIB is either skipped or reverted to the VMware i40enu inbox driver.

  • A rare Page Fault error might cause ESXi hosts to fail with purple diagnostic screen during installation of Dell EMC PowerPath plug-in.

  • If the USB host controller disconnects from your vSphere system and any resource on the device, such as dump files, is still in use by ESXi, the device path cannot be released by the time the device reconnects. As a result, ESXi provides a new path to the USB device, which breaks connection to the /bootbank partition or corrupts the VMFS-L LOCKER partition.

ESXi-7.0U2sc-18295176-standard

Profile Name	ESXi-7.0U2sc-18295176-standard
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	August 24, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vdfs_7.0.2-0.15.18295176 VMware_bootbank_esx-xserver_7.0.2-0.15.18295176 VMware_bootbank_esx-base_7.0.2-0.15.18295176 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.15.18295176 VMware_bootbank_crx_7.0.2-0.15.18295176 VMware_bootbank_native-misc-drivers_7.0.2-0.15.18295176 VMware_bootbank_cpu-microcode_7.0.2-0.15.18295176 VMware_bootbank_gc_7.0.2-0.15.18295176 VMware_bootbank_vsanhealth_7.0.2-0.15.18295176 VMware_bootbank_clusterstore_7.0.2-0.15.18295176 VMware_bootbank_vsan_7.0.2-0.15.18295176 VMware_bootbank_loadesx_7.0.2-0.15.18295176 VMware_bootbank_esx-update_7.0.2-0.15.18295176 VMware_locker_tools-light_11.2.6.17901274-18295176 VMW_bootbank_vmkusb_0.1-1vmw.702.0.15.18295176
PRs Fixed	2738816, 2738857, 2735594, 2761942
Related CVE numbers	N/A

This patch updates the following issues:

• • OpenSSH is updated to version 8.5p1.

  • The OpenSSL library is updated to version openssl-1.0.2y.

  • The Libarchive library is updated to version 3.5.1.

  • The following VMware Tools 11.2.6 ISO images are bundled with ESXi:

    • windows.iso: VMware Tools image for Windows 7 SP1 or Windows Server 2008 R2 SP1 or later
    • linux.iso: VMware Tools 10.3.23 image for older versions of Linux OS with glibc 2.5 or later

    The following VMware Tools 10.3.10 ISO images are available for download:

    • solaris.iso: VMware Tools image for Solaris
    • darwin.iso: VMware Tools image for Mac OS X versions 10.11 and later.

    Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

    • VMware Tools 11.2.6 Release Notes
    • What Every vSphere Admin Must Know About VMware Tools
    • VMware Tools for hosts provisioned with Auto Deploy
    • Updating VMware Tools

ESXi-7.0U2sc-18295176-no-tools

Profile Name	ESXi-7.0U2sc-18295176-no-tools
Build	For build information, see Patches Contained in this Release.
Vendor	VMware, Inc.
Release Date	August 24, 2021
Acceptance Level	PartnerSupported
Affected Hardware	N/A
Affected Software	N/A
Affected VIBs	VMware_bootbank_vdfs_7.0.2-0.15.18295176 VMware_bootbank_esx-xserver_7.0.2-0.15.18295176 VMware_bootbank_esx-base_7.0.2-0.15.18295176 VMware_bootbank_esx-dvfilter-generic-fastpath_7.0.2-0.15.18295176 VMware_bootbank_crx_7.0.2-0.15.18295176 VMware_bootbank_native-misc-drivers_7.0.2-0.15.18295176 VMware_bootbank_cpu-microcode_7.0.2-0.15.18295176 VMware_bootbank_gc_7.0.2-0.15.18295176 VMware_bootbank_vsanhealth_7.0.2-0.15.18295176 VMware_bootbank_clusterstore_7.0.2-0.15.18295176 VMware_bootbank_vsan_7.0.2-0.15.18295176 VMware_bootbank_loadesx_7.0.2-0.15.18295176 VMware_bootbank_esx-update_7.0.2-0.15.18295176 VMware_locker_tools-light_11.2.6.17901274-18295176 VMW_bootbank_vmkusb_0.1-1vmw.702.0.15.18295176
PRs Fixed	2738816, 2738857, 2735594, 2761942
Related CVE numbers	N/A

This patch updates the following issues:

• • OpenSSH is updated to version 8.5p1.

  • The OpenSSL library is updated to version openssl-1.0.2y.

  • The Libarchive library is updated to version 3.5.1.

  • The following VMware Tools 11.2.6 ISO images are bundled with ESXi:

    • windows.iso: VMware Tools image for Windows 7 SP1 or Windows Server 2008 R2 SP1 or later
    • linux.iso: VMware Tools 10.3.23 image for older versions of Linux OS with glibc 2.5 or later

    The following VMware Tools 10.3.10 ISO images are available for download:

    • solaris.iso: VMware Tools image for Solaris
    • darwin.iso: VMware Tools image for Mac OS X versions 10.11 and later.

    Follow the procedures listed in the following documents to download VMware Tools for platforms not bundled with ESXi:

    • VMware Tools 11.2.6 Release Notes
    • What Every vSphere Admin Must Know About VMware Tools
    • VMware Tools for hosts provisioned with Auto Deploy
    • Updating VMware Tools

ESXi Image - ESXi70U2c-18426014

Name	ESXi
Version	ESXi70U2c-18426014
Release Date	August 24, 2020
Category	Bugfix
Affected Components​	ESXi Component - core ESXi VIBs Emulex FC Driver Broadcom NetXtreme-E Network and ROCE/RDMA Drivers for VMware ESXi QLogic FastLinQ 10/25/40/50/100 GbE Ethernet and RoCE/RDMA Drivers for VMware ESXi ESXi Install/Upgrade Component Network driver for Intel(R) X710/XL710/XXV710/X722 Adapters USB Driver
PRs Fixed	2717698, 2731446, 2515171, 731306, 2751564, 2718934, 2755056, 2725886, 2725886, 2753231, 2728910, 2741111, 2759343, 2721379, 2751277, 2708326, 2755977, 2737934, 2777001, 2760932, 2731263, 2731142, 2760267, 2749688, 2763986, 2765534, 2766127, 2777003, 2778793, 2755807, 2760081, 2731644, 2749815, 2749962, 2738238, 2799408, 2722263, 2812906, 2750390, 2715138, 2753546
Related CVE numbers	N/A

ESXi image - ESXi70U2sc-18295176

Name	ESXi
Version	ESXi70U2sc-18295176
Release Date	August 24, 2020
Category	Security
Affected Components​	ESXi Component - core ESXi VIBs USB Driver ESXi Tools Component ESXi Install/Upgrade Component
PRs Fixed	2738816, 2738857, 2735594, 2761942
Related CVE numbers	N/A

Known Issues

The known issues are grouped as follows.

• Miscellaneous Issues
• Installation, Upgrade and Migration Issues
• Known Issues from Earlier Releases

Miscellaneous Issues

• The sensord daemon fails to report ESXi host hardware status

  A logic error in the IPMI SDR validation might cause sensord to fail to identify a source for power supply information. As a result, when you run the command vsish -e get /power/hostStats, you might not see any output.

  Workaround: None

Installation, Upgrade and Migration Issues

• ESXi hosts might lose connectivity after brcmfcoe driver upgrade on Hitachi storage arrays

  After an upgrade of the brcmfcoe driver on Hitachi storage arrays, ESXi hosts might fail to boot and lose connectivity.

  Workaround: None.

Known Issues from Earlier Releases

To view a list of previous known issues, click here.

The earlier known issues are grouped as follows.

• vSAN Issues
• Installation, Upgrade, and Migration Issues
• Security Features Issues
• Networking Issues
• Storage Issues
• vCenter Server and vSphere Client Issues
• Virtual Machine Management Issues
• vSphere HA and Fault Tolerance Issues
• vSphere Lifecycle Manager Issues
• Miscellaneous Issues

vSAN Issues

• If you change the preferred site in a VMware vSAN Stretched Cluster, some objects might incorrectly appear as compliant

  If you change the preferred site in a stretched cluster, some objects might incorrectly appear as compliant, because their policy settings might not automatically change. For example, if you configure a virtual machine to keep data at the preferred site, when you change the preferred site, data might remain on the nonpreferred site.

  Workaround: Before you change a preferred site, in Advanced Settings, lower the ClomMaxCrawlCycleMinutes setting to 15 min to make sure objects policies are updated. After the change, revert the ClomMaxCrawlCycleMinutes option to the earlier value.

Installation, Upgrade, and Migration Issues

• The vCenter Upgrade/Migration pre-checks fail with "Unexpected error 87"

  The vCenter Server Upgrade/Migration pre-checks fail when the Security Token Service (STS) certificate does not contain a Subject Alternative Name (SAN) field. This situation occurs when you have replaced the vCenter 5.5 Single Sign-On certificate with a custom certificate that has no SAN field, and you attempt to upgrade to vCenter Server 7.0. The upgrade considers the STS certificate invalid and the pre-checks prevent the upgrade process from continuing.

  Workaround: Replace the STS certificate with a valid certificate that contains a SAN field then proceed with the vCenter Server 7.0 Upgrade/Migration.

• Problems upgrading to vSphere 7.0 with pre-existing CIM providers

  After upgrade, previously installed 32-bit CIM providers stop working because ESXi requires 64-bit CIM providers. Customers may lose management API functions related to CIMPDK, NDDK (native DDK), HEXDK, VAIODK (IO filters), and see errors related to uwglibc dependency.
  The syslog reports module missing, "32 bit shared libraries not loaded."

  Workaround: There is no workaround. The fix is to download new 64-bit CIM providers from your vendor.

• Installation of 7.0 Update 1 drivers on ESXi 7.0 hosts might fail

  You cannot install drivers applicable to ESXi 7.0 Update 1 on hosts that run ESXi 7.0 or 7.0b.
  The operation fails with an error, such as:
  VMW_bootbank_qedrntv_3.40.4.0-12vmw.701.0.0.xxxxxxx requires vmkapi_2_7_0_0, but the requirement cannot be satisfied within the ImageProfile. ​
Please refer to the log file for more details.

  Workaround: Update the ESXi host to 7.0 Update 1. Retry the driver installation.

• UEFI booting of ESXi hosts might stop with an error during an update to ESXi 7.0 Update 2 from an earlier version of ESXi 7.0

  If you attempt to update your environment to 7.0 Update 2 from an earlier version of ESXi 7.0 by using vSphere Lifecycle Manager patch baselines, UEFI booting of ESXi hosts might stop with an error such as:
  Loading /boot.cfg
Failed to load crypto64.efi
Fatal error: 15 (Not found)

  Workaround: For more information, see VMware knowledge base articles 83063 and 83107 .

• If legacy VIBs are in use on an ESXi host, vSphere Lifecycle Manager cannot extract a desired software specification to seed to a new cluster

  With vCenter Server 7.0 Update 2, you can create a new cluster by importing the desired software specification from a single reference host.  However, if legacy VIBs are in use on an ESXi host, vSphere Lifecycle Manager cannot extract in the vCenter Server instance where you create the cluster a reference software specification from such a host. In the /var/log/lifecycle.log, you see messages such as:
  020-11-11T06:54:03Z lifecycle: 1000082644: HostSeeding:499 ERROR Extract depot failed: Checksum doesn't match. Calculated 5b404e28e83b1387841bb417da93c8c796ef2497c8af0f79583fd54e789d8826, expected: 0947542e30b794c721e21fb595f1851b247711d0619c55489a6a8cae6675e796 2020-11-11T06:54:04Z lifecycle: 1000082644: imagemanagerctl:366 ERROR Extract depot failed. 2020-11-11T06:54:04Z lifecycle: 1000082644: imagemanagerctl:145 ERROR [VibChecksumError]

  Workaround: Follow the steps described in VMware knowledge base article 83042.

• You see a short burst of log messages in the syslog.log after every ESXi boot

  After updating to ESXi 7.0 Update 2, you might see a short burst of log messages after every ESXi boot.
  Such logs do not indicate any issue with ESXi and you can ignore these messages. For example:
  ​2021-01-19T22:44:22Z watchdog-vaai-nasd: '/usr/lib/vmware/nfs/bin/vaai-nasd -f' exited after 0 seconds (quick failure 127) 1
2021-01-19T22:44:22Z watchdog-vaai-nasd: Executing '/usr/lib/vmware/nfs/bin/vaai-nasd -f'
2021-01-19T22:44:22.990Z aainasd[1000051135]: Log for VAAI-NAS Daemon for NFS version=1.0 build=build-00000 option=DEBUG
2021-01-19T22:44:22.990Z vaainasd[1000051135]: DictionaryLoadFile: No entries loaded by dictionary.
2021-01-19T22:44:22.990Z vaainasd[1000051135]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2021-01-19T22:44:22.990Z vaainasd[1000051135]: DictionaryLoad: Cannot open file "//.vmware/config": No such file or directory.
2021-01-19T22:44:22.990Z vaainasd[1000051135]: DictionaryLoad: Cannot open file "//.vmware/preferences": No such file or directory.
2021-01-19T22:44:22.990Z vaainasd[1000051135]: Switching to VMware syslog extensions
2021-01-19T22:44:22.992Z vaainasd[1000051135]: Loading VAAI-NAS plugin(s).
2021-01-19T22:44:22.992Z vaainasd[1000051135]: DISKLIB-PLUGIN : Not loading plugin /usr/lib/vmware/nas_plugins/lib64: Not a shared library.

  Workaround: None

• You see warning messages for missing VIBs in vSphere Quick Boot compatibility check reports

  After you upgrade to ESXi 7.0 Update 2, if you check vSphere Quick Boot compatibility of your environment by using the /usr/lib/vmware/loadesx/bin/loadESXCheckCompat.py command, you might see some warning messages for missing VIBs in the shell. For example:
  Cannot find VIB(s) ... in the given VIB collection.
Ignoring missing reserved VIB(s) ..., they are removed from reserved VIB IDs.
  Such warnings do not indicate a compatibility issue.

  Workaround: The missing VIB messages can be safely ignored and do not affect the reporting of vSphere Quick Boot compatibility. The final output line of the loadESXCheckCompat command unambiguously indicates if the host is compatible.

• Auto bootstrapping a cluster that you manage with a vSphere Lifecycle Manager image fails with an error

  If you attempt auto bootstrapping a cluster that you manage with a vSphere Lifecycle Manager image to perform a stateful install and overwrite the VMFS partitions, the operation fails with an error. In the support bundle, you see messages such as:
  2021-02-11T19:37:43Z Host Profiles[265671 opID=MainThread]: ERROR: EngineModule::ApplyHostConfig. Exception: [Errno 30] Read-only file system

  Workaround: Follow vendor guidance to clean the VMFS partition in the target host and retry the operation. Alternatively, use an empty disk. For more information on the disk-partitioning utility on ESXi, see VMware knowledge base article 1036609.

• Upgrades to ESXi 7.x from 6.5.x and 6.7.0 by using ESXCLI might fail due to a space limitation

  Upgrades to ESXi 7.x from 6.5.x and 6.7.0 by using the esxcli software profile update or esxcli software profile install ESXCLI commands might fail, because the ESXi bootbank might be less than the size of the image profile. In the ESXi Shell or the PowerCLI shell, you see an error such as:

  [InstallationError]
   The pending transaction requires 244 MB free space, however the maximum supported size is 239 MB.
   Please refer to the log file for more details.

  The issue also occurs when you attempt an ESXi host upgrade by using the ESXCLI commands esxcli software vib update or esxcli software vib install.

  Workaround: You can perform the upgrade in two steps, by using the esxcli software profile update command to update ESXi hosts to ESXi 6.7 Update 1 or later, and then update to 7.0 Update 1c. Alternatively, you can run an upgrade by using an ISO image and the vSphere Lifecycle Manager.

• Smart Card and RSA SecurID authentication might stop working after upgrading to vCenter Server 7.0

  If you have configured vCenter Server for either Smart Card or RSA SecurID authentication, see the VMware knowledge base article at https://kb.vmware.com/s/article/78057 before starting the vSphere 7.0 upgrade process. If you do not perform the workaround as described in the KB, you might see the following error messages and Smart Card or RSA SecurID authentication does not work.

  "Smart card authentication may stop working. Smart card settings may not be preserved, and smart card authentication may stop working."

  or

  "RSA SecurID authentication may stop working. RSA SecurID settings may not be preserved, and RSA SecurID authentication may stop working."

  Workaround: Before upgrading to vSphere 7.0, see the VMware knowledge base article at https://kb.vmware.com/s/article/78057.

• Upgrading a vCenter Server with an external Platform Services Controller from 6.7u3 to 7.0 fails with VMAFD error

  When you upgrade a vCenter Server deployment using an external Platform Services Controller, you converge the Platform Services Controller into a vCenter Server appliance. If the upgrade fails with the error install.vmafd.vmdir_vdcpromo_error_21, the VMAFD firstboot process has failed. The VMAFD firstboot process copies the VMware Directory Service Database (data.mdb) from the source Platform Services Controller and replication partner vCenter Server appliance.

  Workaround: Disable TCP Segmentation Offload (TSO) and Generic Segmentation Offload (GSO) on the Ethernet adapter of the source Platform Services Controller or replication partner vCenter Server appliance before upgrading a vCenter Server with an external Platform Services Controller. See Knowledge Base article: https://kb.vmware.com/s/article/74678

• Smart card and RSA SecurID settings may not be preserved during vCenter Server upgrade

  Authentication using RSA SecurID will not work after upgrading to vCenter Server 7.0. An error message will alert you to this issue when attempting to login using your RSA SecurID login.

  Workaround: Reconfigure the smart card or RSA SecureID.

• Migration of vCenter Server for Windows to vCenter Server appliance 7.0 fails with network error message

  Migration of vCenter Server for Windows to vCenter Server appliance 7.0 fails with the error message IP already exists in the network. This prevents the migration process from configuring the network parameters on the new vCenter Server appliance. For more information, examine the log file: /var/log/vmware/upgrade/UpgradeRunner.log

  Workaround:

  1. Verify that all Windows Updates have been completed on the source vCenter Server for Windows instance, or disable automatic Windows Updates until after the migration finishes.
  2. Retry the migration of vCenter Server for Windows to vCenter Server appliance 7.0.

   

• When you configure the number of virtual functions for an SR-IOV device by using the max_vfs module parameter, the changes might not take effect

  In vSphere 7.0, you can configure the number of virtual functions for an SR-IOV device by using the Virtual Infrastructure Management (VIM) API, for example, through the vSphere Client. The task does not require reboot of the ESXi host. After you use the VIM API configuration, if you try to configure the number of SR-IOV virtual functions by using the max_vfs module parameter, the changes might not take effect because they are overridden by the VIM API configuration.

  Workaround: None. To configure the number of virtual functions for an SR-IOV device, use the same method every time. Use the VIM API or use the max_vfs module parameter and reboot the ESXi host.

• Upgraded vCenter Server appliance instance does not retain all the secondary networks (NICs) from the source instance

  During a major upgrade, if the source instance of the vCenter Server appliance is configured with multiple secondary networks other than the VCHA NIC, the target vCenter Server instance will not retain secondary networks other than the VCHA NIC. If the source instance is configured with multiple NICs that are part of DVS port groups, the NIC configuration will not be preserved during the upgrade. Configurations for vCenter Server appliance instances that are part of the standard port group will be preserved.

  Workaround: None. Manually configure the secondary network in the target vCenter Server appliance instance.

• After upgrading or migrating a vCenter Server with an external Platform Services Controller, users authenticating using Active Directory lose access to the newly upgraded vCenter Server instance

  After upgrading or migrating a vCenter Server with an external Platform Services Controller, if the newly upgraded vCenter Server is not joined to an Active Directory domain, users authenticating using Active Directory will lose access to the vCenter Server instance.

  Workaround: Verify that the new vCenter Server instance has been joined to an Active Directory domain. See Knowledge Base article: https://kb.vmware.com/s/article/2118543

• Migrating a vCenter Server for Windows with an external Platform Services Controller using an Oracle database fails

  If there are non-ASCII strings in the Oracle events and tasks table the migration can fail when exporting events and tasks data. The following error message is provided: UnicodeDecodeError

  Workaround: None.

• After an ESXi host upgrade, a Host Profile compliance check shows non-compliant status while host remediation tasks fail

  The non-compliant status indicates an inconsistency between the profile and the host.

  This inconsistency might occur because ESXi 7.0 does not allow duplicate claim rules, but the profile you use contains duplicate rules. For example, if you attempt to use the Host Profile that you extracted from the host before upgrading ESXi 6.5 or ESXi 6.7 to version 7.0 and the Host Profile contains any duplicate claim rules of system default rules, you might experience the problems.

  Workaround:

  1. Remove any duplicate claim rules of the system default rules from the Host Profile document.
  2. Check the compliance status.
  3. Remediate the host.
  4. If the previous steps do not help, reboot the host.

   

• Error message displays in the vCenter Server Management Interface

  After installing or upgrading to vCenter Server 7.0, when you navigate to the Update panel within the vCenter Server Management Interface, the error message "Check the URL and try again" displays. The error message does not prevent you from using the functions within the Update panel, and you can view, stage, and install any available updates.

  Workaround: None.

Security Features Issues

• Turn off the Service Location Protocol service in ESXi, slpd, to prevent potential security vulnerabilities

  Some services in ESXi that run on top of the host operating system, including slpd, the CIM object broker, sfcbd, and the related openwsmand service, have proven security vulnerabilities. VMware has addressed all known vulnerabilities in VMSA-2019-0022 and VMSA-2020-0023, and the fixes are part of the vSphere 7.0 Update 2 release. While sfcbd and openwsmand are disabled by default in ESXi, slpd is enabled by default and you must turn it off, if not necessary, to prevent exposure to a future vulnerability after an upgrade. 

  Workaround: To turn off the slpd service, run the following PowerCLI commands:
  $ Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq “slpd”} | Set-VMHostService -policy “off”
$ Get-VMHost | Get-VmHostService | Where-Object {$_.key -eq “slpd”} | Stop-VMHostService -Confirm:$false
  
  Alternatively, you can use the command chkconfig slpd off && /etc/init.d/slpd stop.
  
  The openwsmand service is not on the ESXi services list and you can check the service state by using the following PowerCLI commands:
  $esx=(Get-EsxCli -vmhost xx.xx.xx.xx -v2)
$esx.system.process.list.invoke() | where CommandLine -like '*openwsman*' | select commandline

  In the ESXi services list, the sfcbd service appears as sfcbd-watchdog.
  
  For more information, see VMware knowledge base articles 76372 and 1025757.

• Encrypted virtual machine fails to power on when HA-enabled Trusted Cluster contains an unattested host

  In VMware® vSphere Trust Authority™, if you have enabled HA on the Trusted Cluster and one or more hosts in the cluster fails attestation, an encrypted virtual machine cannot power on.

  Workaround: Either remove or remediate all hosts that failed attestation from the Trusted Cluster.

• Encrypted virtual machine fails to power on when DRS-enabled Trusted Cluster contains an unattested host

  In VMware® vSphere Trust Authority™, if you have enabled DRS on the Trusted Cluster and one or more hosts in the cluster fails attestation, DRS might try to power on an encrypted virtual machine on an unattested host in the cluster. This operation puts the virtual machine in a locked state.

  Workaround: Either remove or remediate all hosts that failed attestation from the Trusted Cluster.

• Migrating or cloning encrypted virtual machines across vCenter Server instances fails when attempting to do so using the vSphere Client

  If you try to migrate or clone an encrypted virtual machine across vCenter Server instances using the vSphere Client, the operation fails with the following error message: "The operation is not allowed in the current state."

  Workaround: You must use the vSphere APIs to migrate or clone encrypted virtual machines across vCenter Server instances.

Networking Issues

• Reduced throughput in networking performance on Intel 82599/X540/X550 NICs

  The new queue-pair feature added to ixgben driver to improve networking performance on Intel 82599EB/X540/X550 series NICs might reduce throughput under some workloads in vSphere 7.0 as compared to vSphere 6.7.

  Workaround: To achieve the same networking performance as vSphere 6.7, you can disable the queue-pair with a module parameter. To disable the queue-pair, run the command:

  # esxcli system module parameters set -p "QPair=0,0,0,0..." -m ixgben

  After running the command, reboot.

• One or more I/O devices do not generate interrupts when the AMD IOMMU is in use

  If the I/O devices on your ESXi host provide more than a total of 512 distinct interrupt sources, some sources are erroneously assigned an interrupt-remapping table entry (IRTE) index in the AMD IOMMU that is greater than the maximum value. Interrupts from such a source are lost, so the corresponding I/O device behaves as if interrupts are disabled.

  Workaround: Use the ESXCLI command esxcli system settings kernel set -s iovDisableIR -v true to disable the AMD IOMMU interrupt remapper. Reboot the ESXi host so that the command takes effect.

• When you set auto-negotiation on a network adapter, the device might fail

  In some environments, if you set link speed to auto-negotiation for network adapters by using the command esxcli network nic set -a -n vmmicx, the devices might fail and reboot does not recover connectivity. The issue is specific to a combination of some Intel X710/X722 network adapters, a SFP+ module and a physical switch, where auto-negotiate speed/duplex scenario is not supported.

  Workaround: Make sure you use an Intel-branded SFP+ module. Alternatively, use a Direct Attach Copper (DAC) cable.

• Paravirtual RDMA (PVRDMA) network adapters do not support NSX networking policies

  If you configure an NSX distributed virtual port for use in PVRDMA traffic, the RDMA protocol traffic over the PVRDMA network adapters does not comply with the NSX network policies.

  Workaround: Do not configure NSX distributed virtual ports for use in PVRDMA traffic.

• Solarflare x2542 and x2541 network adapters configured in 1x100G port mode achieve throughput of up to 70Gbps in a vSphere environment

  vSphere 7.0 Update 2 supports Solarflare x2542 and x2541 network adapters configured in 1x100G port mode. However, you might see a hardware limitation in the devices that causes the actual throughput to be up to some 70Gbps in a vSphere environment.

  Workaround: None

• VLAN traffic might fail after a NIC reset

  A NIC with PCI device ID 8086:1537 might stop to send and receive VLAN tagged packets after a reset, for example, with a command vsish -e set /net/pNics/vmnic0/reset 1.

  Workaround: Avoid resetting the NIC. If you already face the issue, use the following commands to restore the VLAN capability, for example at vmnic0:
  # esxcli network nic software set --tagging=1 -n vmnic0
# esxcli network nic software set --tagging=0 -n vmnic0

• Any change in the NetQueue balancer settings causes NetQueue to be disabled after an ESXi host reboot

  Any change in the NetQueue balancer settings by using the command esxcli/localcli network nic queue loadbalancer set -n <nicname> --<lb_setting> causes NetQueue, which is enabled by default, to be disabled after an ESXi host reboot.

  Workaround: After a change in the NetQueue balancer settings and host reboot, use the command configstorecli config current get -c esx -g network -k nics to retrieve ConfigStore data to verify whether the /esx/network/nics/net_queue/load_balancer/enable is working as expected.

  After you run the command, you see output similar to:
  {
"mac": "02:00:0e:6d:14:3e",
"name": "vmnic1",
"net_queue": {
  "load_balancer": {
    "dynamic_pool": true,
      "enable": true
  }
 },
 "virtual_mac": "00:50:56:5a:21:11"
}

  If the output is not as expected, for example "load_balancer": "enable": false", run the following command:
  esxcli/localcli network nic queue loadbalancer state set -n <nicname> -e true

• High throughput virtual machines may experience degradation in network performance when Network I/O Control (NetIOC) is enabled

  Virtual machines requiring high network throughput can experience throughput degradation when upgrading from vSphere 6.7 to vSphere 7.0 with NetIOC enabled.

  Workaround: Adjust the ethernetx.ctxPerDev setting to enable multiple worlds.

• IPv6 traffic fails to pass through VMkernel ports using IPsec

  When you migrate VMkernel ports from one port group to another, IPv6 traffic does not pass through VMkernel ports using IPsec.

  Workaround: Remove the IPsec security association (SA) from the affected server, and then reapply the SA. To learn how to set and remove an IPsec SA, see the vSphere Security documentation.

• Higher ESX network performance with a portion of CPU usage increase

  ESX network performance may increase with a portion of CPU usage.

  Workaround: Remove and add the network interface with only 1 rx dispatch queue. For example:

  esxcli network ip interface remove --interface-name=vmk1

  esxcli network ip interface add --interface-name=vmk1 --num-rxqueue=1

• VM might lose Ethernet traffic after hot-add, hot-remove or storage vMotion

  A VM might stop receiving Ethernet traffic after a hot-add, hot-remove or storage vMotion. This issue affects VMs where the uplink of the VNIC has SR-IOV enabled. PVRDMA virtual NIC exhibits this issue when the uplink of the virtual network is a Mellanox RDMA capable NIC and RDMA namespaces are configured.

  Workaround: You can hot-remove and hot-add the affected Ethernet NICs of the VM to restore traffic. On Linux guest operating systems, restarting the network might also resolve the issue. If these workarounds have no effect, you can reboot the VM to restore network connectivity.

• Change of IP address for a VCSA deployed with static IP address requires that you create the DNS records in advance

  With the introduction of the DDNS, the DNS record update only works for VCSA deployed with DHCP configured networking. While changing the IP address of the vCenter server via VAMI, the following error is displayed:

  The specified IP address does not resolve to the specified hostname.

  Workaround: There are two possible workarounds.

  1. Create an additional DNS entry with the same FQDN and desired IP address. Log in to the VAMI and follow the steps to change the IP address.
  2. Log in to the VCSA using ssh. Execute the following script:

     ./opt/vmware/share/vami/vami_config_net

     Use option 6 to change the IP adddress of eth0. Once changed, execute the following script:

     ./opt/likewise/bin/lw-update-dns

     Restart all the services on the VCSA to update the IP information on the DNS server.

• It may take several seconds for the NSX Distributed Virtual Port Group (NSX DVPG) to be removed after deleting the corresponding logical switch in NSX Manager.

  As the number of logical switches increases, it may take more time for the NSX DVPG in vCenter Server to be removed after deleting the corresponding logical switch in NSX Manager. In an environment with 12000 logical switches, it takes approximately 10 seconds for an NSX DVPG to be deleted from vCenter Server.

  Workaround: None.

• Hostd runs out of memory and fails if a large number of NSX Distributed Virtual port groups are created.

  In vSphere 7.0, NSX Distributed Virtual port groups consume significantly larger amounts of memory than opaque networks. For this reason, NSX Distributed Virtual port groups cannot support the same scale as an opaque network given the same amount of memory.

  Workaround:To support the use of NSX Distributed Virtual port groups, increase the amount of memory in your ESXi hosts. If you verify that your system has adequate memory to support your VMs, you can directly increase the memory of hostd using the following command.

  localcli --plugin-dir /usr/lib/vmware/esxcli/int/ sched group setmemconfig --group-path host/vim/vmvisor/hostd --units mb --min 2048 --max 2048

  Note that this will cause hostd to use memory normally reserved for your environment's VMs. This may have the affect of reducing the number of VMs your ESXi host can support.

• DRS may incorrectly launch vMotion if the network reservation is configured on a VM

  If the network reservation is configured on a VM, it is expected that DRS only migrates the VM to a host that meets the specified requirements. In a cluster with NSX transport nodes, if some of the transport nodes join the transport zone by NSX-T Virtual Distributed Switch (N-VDS), and others by vSphere Distributed Switch (VDS) 7.0, DRS may incorrectly launch vMotion. You might encounter this issue when:

  • The VM connects to an NSX logical switch configured with a network reservation.
  • Some transport nodes join transport zone using N-VDS, and others by VDS 7.0, or, transport nodes join the transport zone through different VDS 7.0 instances.

  Workaround: Make all transport nodes join the transport zone by N-VDS or the same VDS 7.0 instance.

• When adding a VMkernel NIC (vmknic) to an NSX portgroup, vCenter Server reports the error "Connecting VMKernel adapter to a NSX Portgroup on a Stateless host is not a supported operation. Please use Distributed Port Group instead."
  • For stateless ESXi on Distributed Virtual Switch (DVS), the vmknic on a NSX port group is blocked. You must instead use a Distributed Port Group.
  • For stateful ESXi on DVS, vmknic on NSX port group is supported, but vSAN may have an issue if it is using vmknic on a NSX port group.

  Workaround: Use a Distributed Port Group on the same DVS.

• Enabling SRIOV from vCenter for QLogic 4x10GE QL41164HFCU CNA might fail

  If you navigate to the Edit Settings dialog for physical network adapters and attempt to enable SR-IOV, the operation might fail when using QLogic 4x10GE QL41164HFCU CNA. Attempting to enable SR-IOV might lead to a network outage of the ESXi host.

  Workaround: Use the following command on the ESXi host to enable SRIOV:

  esxcfg-module

• vCenter Server fails if the hosts in a cluster using Distributed Resource Scheduler (DRS) join NSX-T networking by a different Virtual Distributed Switch (VDS) or combination of NSX-T Virtual Distributed Switch (NVDS) and VDS

  In vSphere 7.0, when using NSX-T networking on vSphere VDS with a DRS cluster, if the hosts do not join the NSX transport zone by the same VDS or NVDS, it can cause vCenter Server to fail.

  Workaround: Have hosts in a DRS cluster join the NSX transport zone using the same VDS or NVDS. 

Storage Issues

• VMFS datastores are not mounted automatically after disk hot remove and hot insert on HPE Gen10 servers with SmartPQI controllers

  When SATA disks on HPE Gen10 servers with SmartPQI controllers without expanders are hot removed and hot inserted back to a different disk bay of the same machine, or when multiple disks are hot removed and hot inserted back in a different order, sometimes a new local name is assigned to the disk. The VMFS datastore on that disk appears as a snapshot and will not be mounted back automatically because the device name has changed.

  Workaround: None. SmartPQI controller does not support unordered hot remove and hot insert operations.

• ESXi might terminate I/O to NVMeOF devices due to errors on all active paths

  Occasionally, all active paths to NVMeOF device register I/O errors due to link issues or controller state. If the status of one of the paths changes to Dead, the High Performance Plug-in (HPP) might not select another path if it shows high volume of errors. As a result, the I/O fails.

  Workaround: Disable the configuration option /Misc/HppManageDegradedPaths to unblock the I/O.

• VOMA check on NVMe based VMFS datastores fails with error

  VOMA check is not supported for NVMe based VMFS datastores and will fail with the error:

  ERROR: Failed to reserve device. Function not implemented 

  Example:

  # voma -m vmfs -f check -d /vmfs/devices/disks/: <partition#>
  Running VMFS Checker version 2.1 in check mode
  Initializing LVM metadata, Basic Checks will be done
  
  Checking for filesystem activity
  Performing filesystem liveness check..|Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).
  ERROR: Failed to reserve device. Function not implemented
  Aborting VMFS volume check
  VOMA failed to check device : General Error

  Workaround: None. If you need to analyse VMFS metadata, collect it using the -l option, and pass to VMware customer support. The command for collecting the dump is:

  voma -l -f dump -d /vmfs/devices/disks/:<partition#> 

• Using the VM reconfigure API to attach an encrypted First Class Disk to an encrypted virtual machine might fail with error

  If an FCD and a VM are encrypted with different crypto keys, your attempts to attach the encrypted FCD to the encrypted VM using the VM reconfigure API might fail with the error message:

  Cannot decrypt disk because key or password is incorrect.

  Workaround: Use the attachDisk API rather than the VM reconfigure API to attach an encrypted FCD to an encrypted VM.

• ESXi host might get in non responding state if a non-head extent of its spanned VMFS datastore enters the Permanent Device Loss (PDL) state

  This problem does not occur when a non-head extent of the spanned VMFS datastore fails along with the head extent. In this case, the entire datastore becomes inaccessible and no longer allows I/Os.

  In contrast, when only a non-head extent fails, but the head extent remains accessible, the datastore heartbeat appears to be normal. And the I/Os between the host and the datastore continue. However, any I/Os that depend on the failed non-head extent start failing as well. Other I/O transactions might accumulate while waiting for the failing I/Os to resolve, and cause the host to enter the non responding state.

  Workaround: Fix the PDL condition of the non-head extent to resolve this issue.

• Virtual NVMe Controller is the default disk controller for Windows 10 guest operating systems

  The Virtual NVMe Controller is the default disk controller for the following guest operating systems when using Hardware Version 15 or later:

  Windows 10
  Windows Server 2016
  Windows Server 2019

  Some features might not be available when using a Virtual NVMe Controller. For more information, see https://kb.vmware.com/s/article/2147714

  Note: Some clients use the previous default of LSI Logic SAS. This includes ESXi host client and PowerCLI.

  Workaround: If you need features not available on Virtual NVMe, switch to VMware Paravirtual SCSI (PVSCSI) or LSI Logic SAS. For information on using VMware Paravirtual SCSI (PVSCSI), see https://kb.vmware.com/s/article/1010398

• After an ESXi host upgrade to vSphere 7.0, presence of duplicate core claim rules might cause unexpected behavior

  Claim rules determine which multipathing plugin, such as NMP, HPP, and so on, owns paths to a particular storage device. ESXi 7.0 does not support duplicate claim rules. However, the ESXi 7.0 host does not alert you if you add duplicate rules to the existing claim rules inherited through an upgrade from a legacy release. As a result of using duplicate rules, storage devices might be claimed by unintended plugins, which can cause unexpected outcome.

  Workaround: Do not use duplicate core claim rules. Before adding a new claim rule, delete any existing matching claim rule.

• A CNS query with the compliance status filter set might take unusually long time to complete

  The CNS QueryVolume API enables you to obtain information about the CNS volumes, such as volume health and compliance status. When you check the compliance status of individual volumes, the results are obtained quickly. However, when you invoke the CNS QueryVolume API to check the compliance status of multiple volumes, several tens or hundreds, the query might perform slowly.

  Workaround: Avoid using bulk queries. When you need to get compliance status, query one volume at a time or limit the number of volumes in the query API to 20 or fewer. While using the query, avoid running other CNS operations to get the best performance.

• A VMFS datastore backed by an NVMe over Fabrics namespace or device might become permanently inaccessible after recovering from an APD or PDL failure

  If a VMFS datastore on an ESXi host is backed by an NVMe over Fabrics namespace or device, in case of an all paths down (APD) or permanent device loss (PDL) failure, the datastore might be inaccessible even after recovery. You cannot access the datastore from either the ESXi host or the vCenter Server system.

  Workaround: To recover from this state, perform a rescan on a host or cluster level. For more information, see Perform Storage Rescan.

• Deleted CNS volumes might temporarily appear as existing in the CNS UI

  After you delete an FCD disk that backs a CNS volume, the volume might still show up as existing in the CNS UI. However, your attempts to delete the volume fail. You might see an error message similar to the following:
  The object or item referred to could not be found.

  Workaround: The next full synchronization will resolve the inconsistency and correctly update the CNS UI.

• Attempts to attach multiple CNS volumes to the same pod might occasionally fail with an error

  When you attach multiple volumes to the same pod simultaneously, the attach operation might occasionally choose the same controller slot. As a result, only one of the operations succeeds, while other volume mounts fail.

  Workaround: After Kubernetes retries the failed operation, the operation succeeds if a controller slot is available on the node VM.

• Under certain circumstances, while a CNS operation fails, the task status appears as successful in the vSphere Client

  This might occur when, for example, you use an incompliant storage policy to create a CNS volume. The operation fails, while the vSphere Client shows the task status as successful.

  Workaround: The successful task status in the vSphere Client does not guarantee that the CNS operation succeeded. To make sure the operation succeeded, verify its results.

• Unsuccessful delete operation for a CNS persistent volume might leave the volume undeleted on the vSphere datastore

  This issue might occur when the CNS Delete API attempts to delete a persistent volume that is still attached to a pod. For example, when you delete the Kubernetes namespace where the pod runs. As a result, the volume gets cleared from CNS and  the CNS query operation does not return the volume. However, the volume continues to reside on the datastore and cannot be deleted through the repeated CNS Delete API operations.

  Workaround: None.

vCenter Server and vSphere Client Issues

• Vendor providers go offline after a PNID change​

  When you change the vCenter IP address (PNID change), the registered vendor providers go offline.

  Workaround: Re-register the vendor providers.

• Cross vCenter migration of a virtual machine fails with an error

  When you use cross vCenter vMotion to move a VM's storage and host to a different vCenter server instance, you might receive the error The operation is not allowed in the current state. 

  This error appears in the UI wizard after the Host Selection step and before the Datastore Selection step, in cases where the VM has an assigned storage policy containing host-based rules such as encryption or any other IO filter rule.

  Workaround: Assign the VM and its disks to a storage policy without host-based rules. You might need to decrypt the VM if the source VM is encrypted. Then retry the cross vCenter vMotion action.

• Storage Sensors information in Hardware Health tab shows incorrect values on vCenter UI, host UI, and MOB

  When you navigate to Host > Monitor > Hardware Health > Storage Sensors on vCenter UI, the storage information displays either incorrect or unknown values. The same issue is observed on the host UI and the MOB path “runtime.hardwareStatusInfo.storageStatusInfo” as well.

  Workaround: None.

• vSphere UI host advanced settings shows the current product locker location as empty with an empty default

  vSphere UI host advanced settings shows the current product locker location as empty with an empty default. This is inconsistent as the actual product location symlink is created and valid. This causes confusion to the user. The default cannot be corrected from UI.

  Workaround: User can use the esxcli command on the host to correct the current product locker location default as below.

  1. Remove the existing Product Locker Location setting with: "esxcli system settings advanced remove -o ProductLockerLocation"

  2. Re-add the Product Locker Location setting with the appropriate default:

        2.a.  If the ESXi is a full installation, the default value is "/locker/packages/vmtoolsRepo" export PRODUCT_LOCKER_DEFAULT="/locker/packages/vmtoolsRepo"

        2.b. If the ESXi is a PXEboot configuration such as autodeploy, the default value is: "/vmtoolsRepo" export PRODUCT_LOCKER_DEFAULT="/vmtoolsRepo"

        Run the following command to automatically figure out the location: export PRODUCT_LOCKER_DEFAULT=`readlink /productLocker`

        Add the setting: esxcli system settings advanced add -d "Path to VMware Tools repository" -o ProductLockerLocation -t string -s $PRODUCT_LOCKER_DEFAULT

  You can combine all the above steps in step 2 by issuing the single command:

  esxcli system settings advanced add -d "Path to VMware Tools repository" -o ProductLockerLocation -t string -s `readlink /productLocker`

• Linked Software-Defined Data Center (SDDC) vCenter Server instances appear in the on-premises vSphere Client if a vCenter Cloud Gateway is linked to the SDDC.

  When a vCenter Cloud Gateway is deployed in the same environment as an on-premises vCenter Server, and linked to an SDDC, the SDDC vCenter Server will appear in the on-premises vSphere Client. This is unexpected behavior and the linked SDDC vCenter Server should be ignored. All operations involving the linked SDDC vCenter Server should be performed on the vSphere Client running within the vCenter Cloud Gateway.

  Workaround: None.

Virtual Machine Management Issues

• Virtual machines with enabled AMD Secure Encrypted Virtualization-Encrypted State (SEV-ES) cannot create Virtual Machine Communication Interface (VMCI) sockets

  Performance and functionality of features that require VMCI might be affected on virtual machines with enabled AMD SEV-ES, because such virtual machines cannot create VMCI sockets.

  Workaround: None.

• UEFI HTTP booting of virtual machines on ESXi hosts of version earlier than 7.0 Update 2 fails

  UEFI HTTP booting of virtual machines is supported only on hosts of version ESXi 7.0 Update 2 and later and VMs with HW version 19 or later.

  Workaround: Use UEFI HTTP booting only in virtual machines with HW version 19 or later. Using HW version 19 ensures the virtual machines are placed only on hosts with ESXi version 7.0 Update 2 or later.

• The postcustomization section of the customization script runs before the guest customization

  When you run the guest customization script for a Linux guest operating system, the precustomization section of the customization script that is defined in the customization specification runs before the guest customization and the postcustomization section runs after that. If you enable Cloud-Init in the guest operating system of a virtual machine, the postcustomization section runs before the customization due to a known issue in Cloud-Init.

  Workaround: Disable Cloud-Init and use the standard guest customization. 

• Group migration operations in vSphere vMotion, Storage vMotion, and vMotion without shared storage fail with error

  When you perform group migration operations on VMs with multiple disks and multi-level snapshots, the operations might fail with the error com.vmware.vc.GenericVmConfigFault Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout.

  Workaround: Retry the migration operation on the failed VMs one at a time.

• Deploying an OVF or OVA template from a URL fails with a 403 Forbidden error

  The URLs that contain an HTTP query parameter are not supported. For example, http://webaddress.com?file=abc.ovf or the Amazon pre-signed S3 URLs.

  Workaround: Download the files and deploy them from your local file system.

• The third level of nested objects in a virtual machine folder is not visible

  Perform the following steps:

  1. Navigate to a data center and create a virtual machine folder.
  2. In the virtual machine folder, create a nested virtual machine folder.
  3. In the second folder, create another nested virtual machine, virtual machine folder, vApp, or VM Template.

  As a result, from the VMs and Templates inventory tree you cannot see the objects in the third nested folder.

  Workaround: To see the objects in the third nested folder, navigate to the second nested folder and select the VMs tab.

vSphere HA and Fault Tolerance Issues

• VMs in a cluster might be orphaned after recovering from storage inaccessibility such as a cluster wide APD

  Some VMs might be in orphaned state after cluster wide APD recovers, even if HA and VMCP are enabled on the cluster.

  This issue might be encountered when the following conditions occur simultaneously:

  • All hosts in the cluster experience APD and do not recover until VMCP timeout is reached.
  • HA primary initiates failover due to APD on a host.
  • Power on API during HA failover fails due to one of the following:
    • APD across the same host
    • Cascading APD across the entire cluster
    • Storage issues
    • Resource unavailability
  • FDM unregistration and VCs steal VM logic might initiate during a window where FDM has not unregistered the failed VM and VC's host synchronization responds that multiple hosts are reporting the same VM. Both FDM and VC unregister the different registered copies of the same VM from different hosts, causing the VM to be orphaned.

   

  Workaround: You must unregister and reregister the orphaned VMs manually within the cluster after the APD recovers.

  If you do not manually reregister the orphaned VMs, HA attempts failover of the orphaned VMs, but it might take between 5 to 10 hours depending on when APD recovers.

  The overall functionality of the cluster is not affected in these cases and HA will continue to protect the VMs. This is an anomaly in what gets displayed on VC for the duration of the problem.

vSphere Lifecycle Manager Issues

• You cannot enable NSX-T on a cluster that is already enabled for managing image setup and updates on all hosts collectively

  NSX-T is not compatible with the vSphere Lifecycle Manager functionality for image management. When you enable a cluster for image setup and updates on all hosts in the cluster collectively, you cannot enable NSX-T on that cluster. However, you can deploy NSX Edges to this cluster.

  Workaround: Move the hosts to a new cluster that you can manage with baselines and enable NSX-T on that new cluster.

• vSphere Lifecycle Manager and vSAN File Services cannot be simultaneously enabled on a vSAN cluster in vSphere 7.0 release

  If vSphere Lifecycle Manager is enabled on a cluster, vSAN File Services cannot be enabled on the same cluster and vice versa. In order to enable vSphere Lifecycle Manager on a cluster, which has VSAN File Services enabled already, first disable vSAN File Services and retry the operation. Please note that if you transition to a cluster that is managed by a single image, vSphere Lifecycle Manager cannot be disabled on that cluster.

  Workaround: None.

• When a hardware support manager is unavailable, vSphere High Availability (HA) functionality is impacted

  If hardware support manager is unavailable for a cluster that you manage with a single image, where a firmware and drivers addon is selected and vSphere HA is enabled, the vSphere HA functionality is impacted. You may experience the following errors.

  • Configuring vSphere HA on a cluster fails.
  • Cannot complete the configuration of the vSphere HA agent on a host: Applying HA VIBs on the cluster encountered a failure.
  • Remediating vSphere HA fails: A general system error occurred: Failed to get Effective Component map.
  • Disabling vSphere HA fails: Delete Solution task failed. A general system error occurred: Cannot find hardware support package from depot or hardware support manager.

   

  Workaround:

  • If the hardware support manager is temporarily unavailable, perform the following steps.
  1. Reconnect the hardware support manager to vCenter Server.
  2. Select a cluster from the Hosts and Cluster menu.
  3. Select the Configure tab.
  4. Under Services, click vSphere Availability.
  5. Re-enable vSphere HA.
  • If the hardware support manager is permanently unavailable, perform the following steps.
  1. Remove the hardware support manager and the hardware support package from the image specification
  2. Re-enable vSphere HA.
  3. Select a cluster from the Hosts and Cluster menu.
  4. Select the Updates tab.
  5. Click Edit .
  6. Remove the firmware and drivers addon and click Save.
  7. Select the Configure tab.
  8. Under Services, click vSphere Availability.
  9. Re-enable vSphere HA.

   

• I/OFilter is not removed from a cluster after a remediation process in vSphere Lifecycle Manager

  Removing I/OFilter from a cluster by remediating the cluster in vSphere Lifecycle Manager, fails with the following error message: iofilter XXX already exists. Тhe iofilter remains listed as installed.

  Workaround:

  1. Call IOFilter API UninstallIoFilter_Task from the vCenter Server managed object (IoFilterManager).
  2. Remediate the cluster in vSphere Lifecycle Manager.
  3. Call IOFilter API ResolveInstallationErrorsOnCluster_Task from the vCenter Server managed object (IoFilterManager) to update the database.

   

• While remediating a vSphere HA enabled cluster in vSphere Lifecycle Manager, adding hosts causes a vSphere HA error state

  Adding one or multiple ESXi hosts during a remediation process of a vSphere HA enabled cluster, results in the following error message: Applying HA VIBs on the cluster encountered a failure.

  Workaround: Аfter the cluster remediation operation has finished, perform one of the following tasks.

  • Right-click the failed ESXi host and select Reconfigure for vSphere HA.
  • Disable and re-enable vSphere HA for the cluster.

   

• While remediating a vSphere HA enabled cluster in vSphere Lifecycle Manager, disabling and re-enabling vSphere HA causes a vSphere HA error state

  Disabling and re-enabling vSphere HA during remediation process of a cluster, may fail the remediation process due to vSphere HA health checks reporting that hosts don't have vSphere HA VIBs installed. You may see the following error message: Setting desired image spec for cluster failed.

  Workaround: Аfter the cluster remediation operation has finished, disable and re-enable vSphere HA for the cluster.

• Checking for recommended images in vSphere Lifecycle Manager has slow performance in large clusters

  In large clusters with more than 16 hosts, the recommendation generation task could take more than an hour to finish or may appear to hang. The completion time for the recommendation task depends on the number of devices configured on each host and the number of image candidates from the depot that vSphere Lifecycle Manager needs to process before obtaining a valid image to recommend.

  Workaround: None.

• Checking for hardware compatibility in vSphere Lifecycle Manager has slow performance in large clusters

  In large clusters with more than 16 hosts, the validation report generation task could take up to 30 minutes to finish or may appear to hang. The completion time depends on the number of devices configured on each host and the number of hosts configured in the cluster.

  Workaround: None

• Incomplete error messages in non-English languages are displayed, when remediating a cluster in vSphere Lifecycle Manager

  You can encounter incomplete error messages for localized languages in the vCenter Server user interface. The messages are displayed, after a cluster remediation process in vSphere Lifecycle Manager fails. For example, your can observe the following error message.
  
  The error message in English language: Virtual machine 'VMC on DELL EMC -FileServer' that runs on cluster 'Cluster-1' reported an issue which prevents entering maintenance mode: Unable to access the virtual machine configuration: Unable to access file[local-0] VMC on Dell EMC - FileServer/VMC on Dell EMC - FileServer.vmx
  
  The error message in French language: La VM « VMC on DELL EMC -FileServer », située sur le cluster « {Cluster-1} », a signalé un problème empêchant le passage en mode de maintenance : Unable to access the virtual machine configuration: Unable to access file[local-0] VMC on Dell EMC - FileServer/VMC on Dell EMC - FileServer.vmx

  Workaround: None.

• Importing an image with no vendor addon, components, or firmware and drivers addon to a cluster which image contains such elements, does not remove the image elements of the existing image

  Only the ESXi base image is replaced with the one from the imported image.

  Workaround: After the import process finishes, edit the image, and if needed, remove the vendor addon, components, and firmware and drivers addon.

• When you convert a cluster that uses baselines to a cluster that uses a single image, a warning is displayed that vSphere HA VIBs will be removed

  Converting a vSphere HA enabled cluster that uses baselines to a cluster that uses a single image, may result a warning message displaying that vmware-fdm component will be removed.

  Workaround: This message can be ignored. The conversion process installs the vmware-fdm component.

• If vSphere Update Manager is configured to download patch updates from the Internet through a proxy server, after upgrade to vSphere 7.0 that converts Update Manager to vSphere Lifecycle Manager, downloading patches from VMware patch repository might fail

  In earlier releases of vCenter Server you could configure independent proxy settings for vCenter Server and vSphere Update Manager. After an upgrade to vSphere 7.0, vSphere Update Manager service becomes part of the vSphere Lifecycle Manager service. For the vSphere Lifecycle Manager service, the proxy settings are configured from the vCenter Server appliance settings. If you had configured Update Manager to download patch updates from the Internet through a proxy server but the vCenter Server appliance had no proxy setting configuration, after a vCenter Server upgrade to version 7.0, the vSphere Lifecycle Manager fails to connect to the VMware depot and is unable to download patches or updates.

  Workaround: Log in to the vCenter Server Appliance Management Interface, https://vcenter-server-appliance-FQDN-or-IP-address:5480, to configure proxy settings for the vCenter Server appliance and enable vSphere Lifecycle Manager to use proxy.

Miscellaneous Issues

• When applying a host profile with version 6.5 to a ESXi host with version 7.0, the compliance check fails

  Applying a host profile with version 6.5 to a ESXi host with version 7.0, results in Coredump file profile reported as not compliant with the host.

  Workaround: There are two possible workarounds.

  1. When you create a host profile with version 6.5, set an advanced configuration option VMkernel.Boot.autoCreateDumpFile to false on the ESXi host.
  2. When you apply an existing host profile with version 6.5, add an advanced configuration option VMkernel.Boot.autoCreateDumpFile in the host profile, configure the option to a fixed policy, and set value to false.

   

• Mellanox ConnectX-4 or ConnectX-5 native ESXi drivers might exhibit minor throughput degradation when Dynamic Receive Side Scaling (DYN_RSS) or Generic RSS (GEN_RSS) feature is turned on

  Mellanox ConnectX-4 or ConnectX-5 native ESXi drivers might exhibit less than 5 percent throughput degradation when DYN_RSS and GEN_RSS feature is turned on, which is unlikely to impact normal workloads.

  Workaround: You can disable DYN_RSS and GEN_RSS feature with the following commands:

  # esxcli system module parameters set -m nmlx5_core -p "DYN_RSS=0 GEN_RSS=0"

  # reboot

• RDMA traffic between two VMs on the same host might fail in PVRDMA environment

  In a vSphere 7.0 implementation of a PVRDMA environment, VMs pass traffic through the HCA for local communication if an HCA is present. However, loopback of RDMA traffic does not work on qedrntv driver.  For instance, RDMA Queue Pairs running on VMs that are configured under same uplink port cannot communicate with each other.

  In vSphere 6.7 and earlier, HCA was used for local RDMA traffic if SRQ was enabled. vSphere 7.0 uses HCA loopback with VMs using versions of PVRDMA that have SRQ enabled with a minimum of HW v14 using RoCE v2.

  The current version of Marvell FastLinQ adapter firmware does not support loopback traffic between QPs of the same PF or port.

  Workaround: Required support is being added in the out-of-box driver certified for vSphere 7.0. If you are using the inbox qedrntv driver, you must use a 3-host configuration and migrate VMs to the third host.

• Unreliable Datagram traffic QP limitations in qedrntv driver

  There are limitations with the Marvell FastLinQ qedrntv RoCE driver and Unreliable Datagram (UD) traffic. UD applications involving bulk traffic might fail with qedrntv driver. Additionally, UD QPs can only work with DMA Memory Regions (MR). Physical MRs or FRMR are not supported. Applications attempting to use physical MR or FRMR along with UD QP fail to pass traffic when used with qedrntv driver. Known examples of such test applications are ibv_ud_pingpong and ib_send_bw.

  Standard RoCE and RoCEv2 use cases in a VMware ESXi environment such as iSER, NVMe-oF (RoCE) and PVRDMA are not impacted by this issue. Use cases for UD traffic are limited and this issue impacts a small set of applications requiring bulk UD traffic.

  Marvell FastLinQ hardware does not support RDMA UD traffic offload. In order to meet the VMware PVRDMA requirement to support GSI QP, a restricted software only implementation of UD QP support was added to the qedrntv driver. The goal of the implementation is to provide support for control path GSI communication and is not a complete implementation of UD QP supporting bulk traffic and advanced features.

  Since UD support is implemented in software, the implementation might not keep up with heavy traffic and packets might be dropped. This can result in failures with bulk UD traffic.

  Workaround: Bulk UD QP traffic is not supported with qedrntv driver and there is no workaround at this time. VMware ESXi RDMA (RoCE) use cases like iSER, NVMe, RDMA and PVRDMA are unaffected by this issue.

• Servers equipped with QLogic 578xx NIC might fail when frequently connecting or disconnecting iSCSI LUNs

  If you trigger QLogic 578xx NIC iSCSI connection or disconnection frequently in a short time, the server might fail due to an issue with the qfle3 driver. This is caused by a known defect in the device's firmware.

  Workaround: None.

• ESXi might fail during driver unload or controller disconnect operation in Broadcom NVMe over FC environment

  In Broadcom NVMe over FC environment, ESXi might fail during driver unload or controller disconnect operation and display an error message such as: @BlueScreen: #PF Exception 14 in world 2098707:vmknvmeGener IP 0x4200225021cc addr 0x19

  Workaround: None.

• ESXi does not display OEM firmware version number of i350/X550 NICs on some Dell servers

  The inbox ixgben driver only recognizes firmware data version or signature for i350/X550 NICs. On some Dell servers the OEM firmware version number is programmed into the OEM package version region, and the inbox ixgben driver does not read this information. Only the 8-digit firmware signature is displayed.

  Workaround: To display the OEM firmware version number, install async ixgben driver version 1.7.15 or later.

• X710 or XL710 NICs might fail in ESXi

  When you initiate certain destructive operations to X710 or XL710 NICs, such as resetting the NIC or manipulating VMKernel's internal device tree, the NIC hardware might read data from non-packet memory.

  Workaround: Do not reset the NIC or manipulate vmkernel internal device state.

• NVMe-oF does not guarantee persistent VMHBA name after system reboot

  NVMe-oF is a new feature in vSphere 7.0. If your server has a USB storage installation that uses vmhba30+ and also has NVMe over RDMA configuration, the VMHBA name might change after a system reboot. This is because the VMHBA name assignment for NVMe over RDMA is different from PCIe devices. ESXi does not guarantee persistence.

  Workaround: None.

• Backup fails for vCenter database size of 300 GB or greater

  If the vCenter database size is 300 GB or greater, the file-based backup will fail with a timeout. The following error message is displayed: Timeout! Failed to complete in 72000 seconds

  Workaround: None.

• A restore of vCenter Server 7.0 which is upgraded from vCenter Server 6.x with External Platform Services Controller to vCenter Server 7.0 might fail

  When you restore a vCenter Server 7.0 which is upgraded from 6.x with External Platform Services Controller to vCenter Server 7.0, the restore might fail and display the following error: Failed to retrieve appliance storage list

  Workaround: During the first stage of the restore process, increase the storage level of the vCenter Server 7.0. For example if the vCenter Server 6.7 External Platform Services Controller setup storage type is small, select storage type large for the restore process.

• Enabled SSL protocols configuration parameter is not configured during a host profile remediation process

  Enabled SSL protocols configuration parameter is not configured during a host profile remediation and only the system default protocol tlsv1.2 is enabled. This behavior is observed for a host profile with version 7.0 and earlier in a vCenter Server 7.0 environment.

  Workaround: To enable TLSV 1.0 or TLSV 1.1 SSL protocols for SFCB, log in to an ESXi host by using SSH, and run the following ESXCLI command: esxcli system wbem -P <protocol_name>

• Unable to configure Lockdown Mode settings by using Host Profiles

  Lockdown Мode cannot be configured by using a security host profile and cannot be applied to multiple ESXi hosts at once. You must manually configure each host.  

  Workaround: In vCenter Server 7.0, you can configure Lockdown Mode and manage Lockdown Mode exception user list by using a security host profile.

• When a host profile is applied to a cluster, Enhanced vMotion Compatibility (EVC) settings are missing from the ESXi hosts

  Some settings in the VMware config file /etc/vmware/config are not managed by Host Profiles and are blocked, when the config file is modified. As a result, when the host profile is applied to a cluster, the EVC settings are lost, which causes loss of EVC functionalities. For example, unmasked CPUs can be exposed to workloads.

  Workaround: Reconfigure the relevant EVC baseline on cluster to recover the EVC settings.

• Using a host profile that defines a core dump partition in vCenter Server 7.0 results in an error

  In vCenter Server 7.0, configuring and managing a core dump partition in a host profile is not available. Attempting to apply a host profile that defines a core dump partition, results in the following error: No valid coredump partition found.

  Workaround: None. In vCenter Server 7.0., Host Profiles supports only file-based core dumps.

• If you run the ESXCLI command to unload the firewall module, the hostd service fails and ESXi hosts lose connectivity

  If you automate the firewall configuration in an environment that includes multiple ESXi hosts, and run the ESXCLI command esxcli network firewall unload that destroys filters and unloads the firewall module, the hostd service fails and ESXi hosts lose connectivity.

  Workaround: Unloading the firewall module is not recommended at any time. If you must unload the firewall module, use the following steps:

  1. Stop the hostd service by using the command:
     /etc/init.d/hostd stop.
  2. Unload the firewall module by using the command:
     esxcli network firewall unload.
  3. Perform the required operations.
  4. Load the firewall module by using the command:
     esxcli network firewall load.
  5. Start the hostd service by using the command:
     /etc/init.d/hostd start.
• vSphere Storage vMotion operations might fail in a vSAN environment due to an unauthenticated session of the Network File Copy (NFC) manager

  Migrations to a vSAN datastore by using vSphere Storage vMotion of virtual machines that have at least one snapshot and more than one virtual disk with different storage policy might fail. The issue occurs due to an unauthenticated session of the NFC manager because the Simple Object Access Protocol (SOAP) body exceeds the allowed size.

  Workaround: First migrate the VM home namespace and just one of the virtual disks. After the operation completes, perform a disk only migration of the remaining 2 disks. 

• Changes in the properties and attributes of the devices and storage on an ESXi host might not persist after a reboot

  If the device discovery routine during a reboot of an ESXi host times out, the jumpstart plug-in might not receive all configuration changes of the devices and storage from all the registered devices on the host. As a result, the process might restore the properties of some devices or storage to the default values after the reboot.

  Workaround: Manually restore the changes in the properties of the affected device or storage.

• If you use a beta build of ESXi 7.0, ESXi hosts might fail with a purple diagnostic screen during some lifecycle operations

  If you use a beta build of ESXi 7.0, ESXi hosts might fail with a purple diagnostic screen during some lifecycle operations such as unloading a driver or switching between ENS mode and native driver mode. For example, if you try to change the ENS mode, in the backtrace you see an error message similar to: case ENS::INTERRUPT::NoVM_DeviceStateWithGracefulRemove hit BlueScreen: ASSERT bora/vmkernel/main/dlmalloc.c:2733 This issue is specific for beta builds and does not affect release builds such as ESXi 7.0.

  Workaround: Update to ESXi 7.0 GA.

• You cannot create snapshots of virtual machines due to an error that a digest operation has failed

  A rare race condition when an All-Paths-Down (APD) state occurs during the update of the Content Based Read Cache (CBRC) digest file might cause inconsistencies in the digest file. As a result, you cannot create virtual machine snapshots. You see an error such as An error occurred while saving the snapshot: A digest operation has failed in the backtrace.

  Workaround: Power cycle the virtual machines to trigger a recompute of the CBRC hashes and clear the inconsistencies in the digest file.

• HTTP requests from certain libraries to vSphere might be rejected

  The HTTP reverse proxy in vSphere 7.0 enforces stricter standard compliance than in previous releases. This might expose pre-existing problems in some third-party libraries used by applications for SOAP calls to vSphere.

  If you develop vSphere applications that use such libraries or include applications that rely on such libraries in your vSphere stack, you might experience connection issues when these libraries send HTTP requests to VMOMI. For example, HTTP requests issued from vijava libraries can take the following form:

  POST /sdk HTTP/1.1
  SOAPAction
  Content-Type: text/xml; charset=utf-8
  User-Agent: Java/1.8.0_221

  The syntax in this example violates an HTTP protocol header field requirement that mandates a colon after SOAPAction. Hence, the request is rejected in flight.

  Workaround: Developers leveraging noncompliant libraries in their applications can consider using a library that follows HTTP standards instead. For example, developers who use the vijava library can consider using the latest version of the yavijava library instead.

• You might see a dump file when using Broadcom driver lsi_msgpt3, lsi_msgpt35 and lsi_mr3

  When using the lsi_msgpt3, lsi_msgpt35 and lsi_mr3 controllers, there is a potential risk to see dump file lsuv2-lsi-drivers-plugin-util-zdump. There is an issue when exiting the storelib used in this plugin utility. There is no impact on ESXi operations, you can ignore the dump file.

  Workaround: You can safely ignore this message. You can remove the lsuv2-lsi-drivers-plugin with the following command:
  
  esxcli software vib remove -n lsuv2-lsiv2-drivers-plugin

• You might see reboot is not required after configuring SR-IOV of a PCI device in vCenter, but device configurations made by third party extensions might be lost and require reboot to be re-applied.

  In ESXi 7.0, SR-IOV configuration is applied without a reboot and the device driver is reloaded. ESXi hosts might have third party extensions perform device configurations that need to run after the device driver is loaded during boot. A reboot is required for those third party extensions to re-apply the device configuration.

  Workaround: You must reboot after configuring SR-IOV to apply third party device configurations.

To collapse the list of previous known issues, click here.