| vCenter Server 6.7 Update 3 | AUG 20 2019 | ISO Build 14367737 vCenter Server Appliance 6.7 Update 3 | AUG 20 2019 | ISO Build 14367737 Check for additions and updates to these release notes. |
What's in the Release Notes
The release notes cover the following topics:
What's New
- vCenter Server 6.7 Update 3 supports a secure dynamic relationship between the IP address settings of a vCenter Server Appliance and a DNS server. DNS resource records update as soon as any IP address of the vCenter Server Appliance changes. The Dynamic Domain Name Service (DDNS) client on the vCenter Server Appliance also sends automatic secure updates to DNS servers on scheduled intervals. However, such dynamic DNS updates are supported only when the vCenter Server Appliance is joined to an Active Directory domain and you have Administrator privileges.
- With vCenter Server 6.7 Update 3, you can configure virtual machines and templates with up to four NVIDIA virtual GPU (vGPU) devices to cover use cases requiring multiple GPU accelerators attached to a virtual machine. To use the vMotion vGPU feature, you must set the
vgpu.hotmigrate.enabled advanced setting to true and make sure that both your vCenter Server and ESXi hosts are running vSphere 6.7 Update 3.
vMotion of multi GPU-accelerated virtual machines might fail gracefully under heavy GPU workload due to the maximum switchover time of 100 secs. To avoid this failure, either increase the maximum allowable switchover time or wait until the virtual machine is performing a less intensive GPU workload.
- With vCenter Server 6.7 Update 3, you can change the Primary Network Identifier (PNID) of your vCenter Server Appliance. You can change the vCenter Server Appliance FQDN or host name, and also modify the IP address configuration of the virtual machine Management Network (NIC 0). For more information, see this VMware blog post.
- With vCenter Server 6.7 Update 3, if the overall health status of a vSAN cluster is Red, APIs to configure or extend HCI clusters throw
InvalidState exception to prevent further configuration or extension. This fix aims to resolve situations when mixed versions of ESXi host in a HCI cluster might cause vSAN network partition.
- vCenter Server 6.7 adds new SandyBridge microcode to the cpu-microcode VIB to bring SandyBridge security up to par with other CPUs and fix per-VM Enhanced vMotion Compatibility (EVC) support. For more information, see VMware knowledge base article 1003212.
Earlier Releases of vCenter Server 6.7
Features and known issues of vCenter Server are described in the release notes for each release. Release notes for earlier releases of vCenter Server 6.7 are:
For internationalization, compatibility, installation and upgrade, open source components and product support notices see the VMware vCenter Sever 6.7 Update 1 Release Notes.
Upgrade Notes for This Release
Upgrade from vCenter Server 6.7 Update 1a to 6.7 Update 3 is not supported. You must first upgrade to vCenter Server 6.7 Update 1b or 6.7 Update 2, and then patch your system to 6.7 Update 3.
Patches Contained in This Release
This release of vCenter Server 6.7 Update 3 delivers the following patches. See the VMware Patch Download Center for more information on downloading patches.
Security Patch for VMware vCenter Server 6.7 Update 3
Third-party product fixes (for example: JRE, tcServer). This patch is applicable for vCenter Server for Windows, Platform Services Controller for Windows, and vSphere Update Manager.
NOTE: This patch updates only the JRE version 8U212 b31.
For vCenter Server and Platform Services Controller for Windows
| Download Filename |
VMware-VIMPatch-T-6.7.0-14367737.iso |
| Build |
14367737 |
| Download Size |
40.7 MB |
| md5sum |
a50e2ef2b1da9eac74bf2cc12e3a7b6d |
| sha1checksum |
8d216b132b8bc1a4dc306bd4a41007b6857ed88e |
These vCenter Server components depend on JRE and have to be patched:
- vCenter Server
- Platform Services Controller
- vSphere Update Manager
Download and Installation
You can download this patch by going to the VMware Patch Download Center and choosing VC from the Select a Product drop-down menu.
- Mount the
VMware-VIMPatch-T-6.7.0-14367737.iso file to the system where the vCenter Server component is installed.
- Double-click
ISO_mount_directory/autorun.exe.
- In the vCenter Server Java Components Update wizard, click Patch All.
Full Patch for VMware vCenter Server Appliance 6.7 Update 3
Product Patch for vCenter Server Appliance containing VMware software fixes, security fixes, and Third Party Product fixes (for example: JRE and tcServer).
This patch is applicable to the vCenter Server Appliance and Platform Services Controller Appliance.
For vCenter Server and Platform Services Controller Appliances
| Download Filename |
VMware-vCenter-Server-Appliance-6.7.0.40000-14367737-patch-FP.iso |
| Build |
14367737 |
| Download Size |
1980.5 MB |
| md5sum |
57c943bd8bfd6580a49ca1e58ecdd382 |
| sha1checksum |
7ec1ba261f7bd236803155bad1ec727fa40a63ae |
Download and Installation
You can download this patch by going to the VMware Patch Download Center and choosing VC from the Select a Product drop-down menu.
- Attach the
VMware-vCenter-Server-Appliance-6.7.0.40000-14367737-patch-FP.iso file to the vCenter Server Appliance CD or DVD drive.
- Log in to the appliance shell with your root credentials and run the commands given below:
- To stage the ISO:
software-packages stage --iso
- To see the staged content:
software-packages list --staged
- To install the staged rpms:
software-packages install --staged
For more information on using the vCenter Server Appliance shells, see VMware knowledge base article 2100508.
For more information on patching the vCenter Server Appliance, see Patching the vCenter Server Appliance.
For more information on staging patches, see Stage Patches to vCenter Server Appliance.
For more information on installing patches, see Install vCenter Server Appliance Patches.
For issues resolved in this patch see Resolved Issues.
For Photon OS updates, see VMware vCenter Server Appliance Photon OS Security Patches.
For more information on patching using the Appliance Management Interface, see Patching the vCenter Server Appliance by Using the Appliance Management Interface.
Resolved Issues
The resolved issues are grouped as follows.
Security Issues
- Update to the Spring Framework
The Spring Framework is updated to version 4.3.22.
- Update to the Apache Tomcat server
The Apache Tomcat server is updated to version 8.5.38.
- Upgrade of Eclipse Jetty
Eclipse Jetty is upgraded to version 9.4.15.v20190215.
- Update to PostgreSQL
PostgreSQL is updated to version 9.6.12.
- Update of the JSON processor package
The JSON processor package is updated to version 2.9.8.
- Security vulnerabilities with jQuery version 2.1.4
The jQuery version 2.1.4 might have security vulnerabilities while performing AJAX calls to Domino agents.
This issue is resolved in this release. This fix upgrades the versions of jQuery and the jQuery UI.
- Update to the Apache HTTP Server (httpd)
Httpd is updated to version 2.4.39 to resolve security issues with identifiers CVE-2019-0211, CVE-2019-2017, and CVE-2019-0215.
- Update to Apache Struts
Apache Struts is updated to version 2.5.20.
- Connecting VMware OVF Tool to a target vCenter Server through a secure web proxy fails
The OVF Tool supported connections to a vCenter Server target through a regular HTTP proxy but not through a Secure Proxy.
This issue is resolved in this release. You can connect OVF Tool to a target vCenter Server through a secure web proxy.
- Update to the Spring Framework
The Spring Framework is updated to version 4.3.22.
- After the Active Directory server is temporarily inaccessible, the user and group permissions for the Active Directory domain are removed
vCenter Server regularly validates users and groups against the Windows Active Directory domain. If an identity is removed, the vCenter Server system removes the permissions associated with it. If the Active Directory server is unreachable during the validation process, the vCenter Server system might incorrectly interpret this as removal of all domain users and groups, and might remove the permissions associated with them.
This issue is resolved in this release. The fix differentiates between a missing identity and no connection.
- Update to the Network Time Protocol (NTP) daemon
The NTP daemon is updated to version 4.2.8p13.
- Update to OpenSSL
The OpenSSL package is updated to version openssl-1.0.2r.
- Update to zlib library
The zlib library is updated to version 1.2.11.
- Update to cURL
cURL is updated to version 7.64.1.
- Update to Photon OS kernel
Photon OS kernel is updated to version 4.4.182 to resolve security issues with identifiers CVE-2019-11477, CVE-2019-11478, and CVE-2019-11479.
- Update to JRE
Oracle (Sun) JRE is updated to version 8U212 b31.
Networking Issues
- After an API call to an ESXi host, if a transaction element is not removed, the vpxd service might run out of memory
In some cases, after an API call to an ESXi host, if a transaction element is not removed, the vpxd service might run out of memory. This issue occurs in large deployments with complex traffic filters configured on distributed virtual port groups.
This issue is resolved in this release.
- After you revert to a snapshot, the virtual machine NIC might be incorrectly connected to the vSphere Distributed Switch (VDS)
If you take a snapshot of a virtual machine when the VM NIC is disconnected from a VDS, reverting to that same snapshot, might result in the VM NIC getting connected to the VDS automatically.
This issue is resolved in this release.
- In complex Active Directory environments, you might fail to log in to the vCenter Server system with a NoPermission error
In environments with complex or slow Active Directory deployments, the Active Directory users might fail to log in to the vCenter Server system with a NoPermission error.
This issue is resolved in this release.
- vSphere Distributed Resource Scheduler (DRS) might incorrectly determine that a cluster is imbalanced
vSphere DRS might incorrectly determine that a cluster is severely imbalanced when the number of failover and non-failover hosts is nearly equal. As a result, DRS frequently triggers vMotion migrations.
This issue is resolved in this release. For the fix to take effect, you must enable the advanced option ExcludeFailoverHostFromSD to 1, so that you exclude any failover hosts from the calculations performed by DRS to detect if a cluster is balanced or not.
vCenter Server, vSphere Web Client, and vSphere Client Issues
- The Other hard disks section might disappear from the VM Hardware pane in the vSphere Web Client
In the vSphere Web Client, when you select a virtual machine from the vCenter Server inventory, you might not see the Other hard disks section in the VM Hardware pane.
This issue is resolved in this release.
- Disk Bay sensors are categorized as Other devices
In the vSphere Client, vSphere Web Client, and the VMware Host Client, Disk Bay sensors are categorized as Other devices.
This issue is resolved in this release. The fix categorizes the Disk Bay sensors under the Storage category.
- Numbering of firewall rules might unexpectedly change if you reorder the rules
If you create more than 9 firewall rules in a vCenter Server Appliance and change the order, setting a rule with a double-digit numbering among rules with one-digit numbering, the numbering might change. For instance, if you move a rule with number 10, such as 10 RETURN all -- X.X.X.10 anywhere, to position 2, the numbering might change to 2 RETURN all -- X.X.X.10 anywhere.
This issue is resolved in this release.
Upgrade Issues
- Upgrades to vCenter Server to 6.7 Update 2 might fail if transaction recovery is enabled
Upgrades to vCenter Server to 6.7 Update 2 might intermittently fail if transaction recovery is enabled.
This issue is resolved in this release.
- Your SNMP configuration might not retain after an upgrade of the vCenter Server system
Your SNMP configuration might not retain after an upgrade of the vCenter Server system.
This issue is resolved in this release. The fix adds support for exports of the SNMP configuration during an upgrade in the upgradeRunner process.
- vCenter Server upgrades might fail due to an unresolved component dependency
vCenter Server upgrades from version 6.0.x to 6.7.x might intermittently fail due to an unresolved component dependency. You might see an error similar to eam:Export failed due to missing vpxd_connector.ext file.
This issue is resolved in this release.
Virtual Machine Management Issues
- When you turn on the pair-wise imbalance check, DRS might migrate virtual machines even if the cluster seems balanced
In vCenter Server 6.0 Update 3 and later, if the advanced DRS option CheckPairwiseImbalance is turned on, when DRS detects pair-wise imbalance between two hosts, it continues to do load balancing by migrating virtual machines between the two hosts, even if the cluster seems balanced. A pair-wise imbalance occurs when any two hosts in the cluster have CPU or memory usage difference bigger than the set threshold, which by default is 20%.
This issue is resolved in this release.
- In the vSphere Web Client, you cannot set the CPU limit for virtual machines to Unlimited
When you deploy virtual machines or edit the settings of virtual machines in the vSphere Web Client, the option Maximum: Unlimited that sets the CPU limit for virtual machines to unlimited is not available.
This issue is resolved in this release. To use the option Maximum: Unlimited in the vSphere Web Client, right-click on a virtual machine and click Edit Settings. In the Virtual Hardware tab, navigate to CPU > Limit and select Maximum: Unlimited from the drop-down menu.
Storage Issues
- Standalone First Class Disks are not queried and considered for the recomputation of compliance during datastore tag association changes
In case of a change in the datastore tag associations, the periodic datastore tag change poller task queries for virtual machine and VMDK entities on a particular datastore. Standalone First Class Disks that are not attached to any virtual machine are not queried and considered. As a result, after tag association changes for a datastore, compliance re-computation is not triggered for standalone First Class Disks.
This issue is resolved in this release. After a change in the tag associations, the First Class Disks on a datastore are identified and compliance is recomputed for them.
- You might fail to upload any type of file to a datastore
When you try to upload a file to a datastore, the operation might fail. You might see a 0% progress for the task in the Recent Tasks pane.
This issue is resolved in this release.
- vSphere Storage Distributed Resource Scheduler (vSphere Storage DRS) might not work properly in environments with linked clone virtual machines
In environments with linked clone virtual machines or virtual machines that have snapshots, vSphere Storage DRS might not work properly. vSphere Storage DRS might make incorrect recommendations, no recommendations to migrate the virtual machines, or migrate them between two datastores in an infinite loop. When vSphere Storage DRS migrates a linked clone virtual machine, an incorrect calculation of the disk size might also cause a full copy of the base disk to be created.
This issue is resolved in this release.
- vSphere Storage DRS fails to generate enough migrations with vMotion during ExpandDisk operations and the reconfiguration for the virtual machine might fail
During ExpandDisk operations, vSphere Storage DRS recommends the migration of some of the existing virtual machines or disks to another datastore to free resources for the new requested capacity. However, vSphere Storage DRS does not include some of the Storage vMotion actions in the final recommendation. As a result, only a part of the required resources are freed up on the current datastore on which the disk is expanded and the ExpandDisk operation fails with a NoDiskSpace error.
This issue is resolved in this release.
- Reports for storage usage of virtual machines stop refreshing within a few hours after an ESXi host boots
Reports for storage usage of virtual machines might stop refreshing within a few hours after an ESXi host boots. This issue is due to an underflow error in calculating the storage refresh duration, resulting in unusually long wait times instead of the intended 105-135 minute window.
This issue is resolved in this release. Upgrading to vCenter Server 6.7 Update 3 fixes the timing and re-enables periodic refreshes of storage usage reports.
Miscellaneous Issues
- In an environment with a heavy workload, the VMware Security Token Service (vmware-stsd) might stop responding and cause login failures
The vmware-stsd service has a fixed amount of allocated memory in the setenv.sh file but the memory size might be insufficient in an environment with a heavy workload. As a result, the vmware-stsd service might stop responding and cause login failures.
This issue is resolved in this release. The fix dynamically sets the memory allocation for the vmware-stsd service at the start time, based on the size of the deployment.
Guest OS Issues
- The application to generate computer names and IP addresses in a guest OS customization process might not work properly
When you implement an application to generate computer names and IP addresses in a guest OS customization process, the operation might fail with an error: Operation timed out.
This issue is resolved in this release.
- Static routes of Red Hat virtual machines might be lost after guest OS customization
If you configure a Red Hat virtual machine with static routes and do guest OS customization, the routes might be deleted from the customized virtual machine.
This issue is resolved in this release.
Server Configuration Issues
- You might be unable to add a self-signed certificate to the ESXi trust store and fail to add an ESXi host to the vCenter Server system
The ESXi trust store contains a list of Certificate Authority (CA) certificates that are used to build the chain of trust when an ESXi host is the client in a TLS channel communication. The certificates in the trust store must be with a CA bit set: X509v3 Basic Constraints: CA: TRUE. If a certificate without this bit set is passed to the trust store, for example, a self-signed certificate, the certificate is rejected. As a result, you might fail to add an ESXi host to the vCenter Server system.
This issue is resolved in this release. The fix adds the advanced option Config.HostAgent.ssl.keyStore.allowSelfSigned. If you already face the issue, set this option to TRUE to add a self-signed server certificate to the ESXi trust store.
CLI Issues
- If during the convergence to a vCenter Server with an embedded Platform Services Controller any certificates or keys in the VMware Endpoint Certificate Store (VECS) contain a forward slash (/), the converge operation fails with an error
If during the convergence to a vCenter Server with an embedded Platform Services Controller any certificates or keys in the VECS contains a forward slash (/), the converge operation fails with an error in the converge log: Operation failed with error ERROR_FILE_NOT_FOUND.
This issue is resolved in this release.
CIM and API Issues
- If you remove an ESXi host that is part of a cluster from the vCenter Server system by using a dcli.com.vmware.vcenter.host.delete REST API, both the host and the parent cluster are removed
The dcli.com.vmware.vcenter.host.delete REST API removes a standalone ESXi host from the vCenter Server system. However, if an ESXi host is part of a cluster, and you remove the ESXi host from the vCenter Server system by using the dcli.com.vmware.vcenter.host.delete REST API, the parent cluster is also removed from the vCenter Server system.
This issue is resolved in this release. If you try to remove an ESXi host that is part of a cluster by using the dcli.com.vmware.vcenter.host.delete REST API, an error such as ResourceInUse displays.
Tools Issues
- The OVF Tool fails with an error during the upload of OVF or ISO files to vCloud Director
During the upload of OVF or ISO files to vCloud Director, the OVF Tool sends requests with Transfer-Encoding and Content-Length headers together. RFC 7230 is responsible for the rules for the Transfer-Encoding and Content-Length headers. According to RFC 7230, sending both headers is an error condition and the server sends a 400 response: Bad Content-Length. As a result, the OVF Tool fails with an error: Transfer Failed - Error: Failed to send http data.
This issue is resolved in this release. The fix adds a flag to the OVF Tool, so that the OVF Tool does not send a Content-Length header. When the OVF Tool communicates with vCloud Director, the flag --X:skipContentLength is set to True to make the OVF Tool consistent with RFC 7230.
- When the flag --noImageFiles is used during an export of a virtual machine or a virtual machine template, the OVF Tool might fail with a segmentation fault
When you try to export a virtual machine or a virtual machine template with an NVRAM file by using the OVF Tool, and if you use the flag --noImageFile, the OVF Tool might fail with a segmentation fault. The OVF Tool finds that the ESXi host returns the NVRAM file for the virtual machine or the virtual machine template in the network file copy lease which has the list of files. As a result, the OVF Tool might stop responding due to a discrepancy between the ESXi host and the command line. In the OVF Tool logs, you might see an error message similar to: Could not find any sha digest for ../disk-1.nvram while reading.
This issue is resolved in this release. The segmentation fault does not occur when you use the flag --noImageFiles. A new --noNvramFile flag is introduced to skip the download of NVRAM files during an import or an export. The existing--noImageFiles flag is used to skip the download of image files such as CD-ROM and floppy.
Known Issues
The known issues are grouped as follows.
High Availability Issues
- Duplicate DNS records after configuring a vCenter Server High Availability environment might interrupt access to the vCenter Server system
After configuring or patching a vCenter Server High Availability environment followed by a failover, access to the vCenter Server system might be blocked due to duplicate DNS records for the vCenter Sever Appliance.
Workaround: Before patching vCenter Server High Availability environments, clean up duplicate DNS records by following the steps described in VMware knowledge base article 76406.
Convergence Issues
- Convergence, domain repointing and fresh installation of a vCenter Server Appliance with an embedded Platform Services Controller connected in Embedded Linked Mode might fail with error install.vmafd.vmdir_vdcpromo_error_21
The following operations might fail with an error similar to: id: install.vmafd.vmdir_vdcpromo_error_21 in the /var/log/firstboot/vmafd-firstboot.py_<PID>_stderr.log file:
- Converging instances of vCenter Server Appliance with an external Platform Services Controller instances into vCenter Server Appliance with an embedded Platform Services Controller connected in Embedded Linked Mode.
- Repointing a vCenter Server with an Embedded Platform Services Controller to a new domain.
- Adding a new vCenter Server Appliance with an embedded Platform Services Controller connected in Embedded Linked Mode.
Workaround: For more information, see VMware knowledge base article 74678.
Upgrade Issues
- After an upgrade of a vCenter Server system to version 6.7 Update 2, pre-upgrade First Class Disks (FCD) might not be listed in the Global Catalog
During an upgrade of your vCenter Server system to version 6.7 Update 2, the FCD Global Catalog might pick an ESXi host that is not yet updated to invoke a sync and the sync fails. As a result, the listVStorageObjectForSpec API might not return all FCDs created prior to the upgrade.
Workaround: After you upgrade all ESXi hosts in the inventory, start syncDatastore API with fullSync set to true.
Storage Issues
- Incorrect behavior of the backingObjectId and SnapshotInfo fields of VStorageObjectResult
In non-vSAN datastores, the backingObjectId and SnapshotInfo fields of VStorageObjectResult for a First Class Disk are always set to null.
In vSAN datastores, when you create a snapshot of a First Class Disk, backingObjectId and SnapshotInfo fields of VStorageObjectResult for the First Class Disk are populated. If the First Class Disk has multiple snapshots, deleting the latest snapshot updates the backingObjectId and SnapshotInfo fields, but deleting older snapshots does not update the fields.
Workaround: None.
Miscellaneous Issues
- After updating to vCenter Server 6.7 Update 3, the rsyslog service stops forwarding logs after some time
After updating a vCenter Server system to 6.7 Update 3, the rsyslog service might stop forwarding logs to the system after a short undefined period of time.
Workaround: To resume forwarding logs to the configured remote server, restart the syslog service by using the command systemctl restart syslog. For more information, see VMware knowledge base article 75088.
Known Issues from Prior Releases
To view a list of previous known issues, click here.
The earlier known issues are grouped as follows.
CLI Issues
- Views might switch from the appliance shell to the Direct Console User Interface during an upgrade of the vCenter Server Appliance by using the CLI installer
During an upgrade of the vCenter Server Appliance by using the CLI installer, views might switch intermittently from the appliance shell to the Direct Console User Interface. Restarts of the applmgmt service during updates causes the issue.
Workaround: Switch to appliance shell tty to monitor the progress.
Internationalization Issues
- A VMkernel network using an NSX logical switch might fail for stateless hosts if you register vCenter Server with non-ASCII characters on VMware NSX Manager
If you register vCenter Server to an NSX Manager with a password containing characters from the extended ASCII codes between 128 and 255, or non-ASCII characters, a VMkernel network using an NSX logical switch might be lost after deploying a stateless host.
Workaround: Register vCenter Server to an NSX Manager with a password containing only ASCII characters.
- An ESXi host might stop responding if you add a vSphere Distributed Switch named with a string containing tens of non-ASCII characters to a physical adapter in a hyper-converged infrastructure (HCI) cluster
If you name a VDS with a string containing more than 40 characters from the extended ASCII codes between 128 and 255, or more than 26 non-ASCII characters, ESXi hosts might stop responding when you attempt to add the VDS to a physical adapter during the configuration of a hyper-converged infrastructure (HCI) cluster.
Workaround: Use strings with less than 40 characters from the extended ASCII codes and 26 non-ASCII characters when naming a VDS.
Tools Issues
- The OVF Tool might fail to verify an SSL thumbprint if you use CLI
If you set the SSL thumbprint value by using CLI, the OVF Tool might fail to verify the thumbprint. The issue is not monitored if you use the Direct Console User Interface (DCUI).
Workaround: Use any of the following alternatives:
- In the DCUI, specify the thumbprint in the section of
ssl_certificate_verification.
- In the DCUI, specify to ignore certificate thumbprint for ESXi by putting
ssl_certificate_verification verification_mode to False.
- Ignore all certificate thumbprints globally by using the command-line parameter:
--no-ssl-certificate-verification.
- Wait for the CLI prompt to accept the thumbprint that it receives from the source.
Installation, Upgrade, and Migration Issues
- ESXi installation or upgrade fail due to memory corruption on HPE ProLiant - DL380/360 Gen 9 Servers
The issue occurs on HPE ProLiant - DL380/360 Gen 9 Servers that have a Smart Array P440ar storage controller.
Workaround: Set the server BIOS mode to UEFI before you install or upgrade ESXi.
- After an ESXi upgrade to version 6.7 and a subsequent rollback to version 6.5 or earlier, you might experience failures with error messages
You might see failures and error messages when you perform one of the following on your ESXi host after reverting to 6.5 or earlier versions:
- Install patches and VIBs on the host
Error message: [DependencyError] VIB VMware_locker_tools-light requires esx-version >= 6.6.0
- Install or upgrade VMware Tools on VMs
Error message: Unable to install VMware Tools.
After the ESXi rollback from version 6.7, the new tools-light VIB does not revert to the earlier version. As a result, the VIB becomes incompatible with the rolled back ESXi host causing these issues.
Workaround: Perform the following to fix this problem.
SSH to the host and run one of these commands:
esxcli software vib install -v /path/to/tools-light.vib
or
esxcli software vib install -d /path/to/depot/zip -n tools-light
Where the vib and zip are of the currently running ESXi version.
Note: For VMs that already have new VMware Tools installed, you do not have to revert VMware Tools back when ESXi host is rolled back.
- Special characters backslash (\) or double-quote (") used in passwords causes installation pre-check to fail
If the special characters backslash (\) or double quote (") are used in ESXi, vCenter Single Sign-On, or operating system password fields during the vCenter Server Appliance Installation templates, the installation pre-check fails with the following error:
Error message: com.vmware.vcsa.installer.template.cli_argument_validation: Invalid \escape: line ## column ## (char ###)
Workaround: If you include special characters backslash (\) or double quote (") in the passwords for ESXi, operating systems, or Single-Sign-On, the special characters need to be escaped. For example, the password pass\word should be escaped as pass\\word.
- Windows vCenter Server 6.7 installer fails when non-ASCII characters are present in password
The Windows vCenter Server 6.7 installer fails when the Single Sign-on password contains non-ASCII characters for Chinese, Japanese, Korean, and Taiwanese locales.
Workaround: Ensure that the Single Sign-on password contains ASCII characters only for Chinese, Japanese, Korean, and Taiwanese locales.
- Cannot log in to vSphere Appliance Management Interface if the colon character (:) is part of vCenter Server root password
During the vCenter Server Appliance UI installation (Set up appliance VM page of Stage 1), if you include the colon character (:) as part of the vCenter Server root password, logging into the vSphere Appliance Management Interface (https://vc_ip:5480) fails and you are unable to login. The password might be accepted by the password rule check during the setup, but login fails.
Workaround: Do not use the colon character (:) to set the vCenter Server root password in the vCenter Server Appliance UI (Set up appliance VM of Stage 1).
- vCenter Server Appliance installation fails when the backslash character (\) is included in the vCenter Single Sign-On password
During the vCenter Server Appliance UI installation (SSO setup page of Stage 2), if you include the backslash character (\) as part of the vCenter Single Sign-On password, the installation fails with the error Analytics Service registration with Component Manager failed. The password might be accepted by the password rule check, but installation fails.
Workaround: Do not use the backslash character (\) to set the vCenter Single Sign-On password in the vCenter Server Appliance UI installer (SSO setup page of Stage 2)
- Scripted ESXi installation fails on HP ProLiant Gen 9 Servers with an error
When you perform a scripted ESXi installation on an HP ProLiant Gen 9 Server under the following conditions:
- The Embedded User Partition option is enabled in the BIOS.
- You use multiple USB drives during installation: one USB drive contains the ks.cfg file, and the others USB drive is not formatted and usable.
The installation fails with the error message Partitions not initialized.
Workaround:
- Disable the Embedded User Partition option in the server BIOS.
- Format the unformatted USB drive with a file system or unplug it from the server.
- Upgrading vCenter Server 6.5 for Windows to vCenter Server 6.7 might fail if the vSphere Authentication Proxy service is active
If the vSphere Authentication Proxy service is active while you perform an upgrade from vCenter Server 6.5 for Windows to vCenter Server 6.7, the operation might fail during the pre-check. You might see an error similar to:
The following non-configurable port(s) are already in use:
2016, 7475, 7476
Stop the process(es) that use these port(s).
Workaround: Stop the vSphere Authentication Proxy service. You can restart the service after the successful upgrade to vCenter Server 6.7.
- Patching to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7 might fail when vCenter Server High Availability is active
Patching to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7 might fail when vCenter Server High Availability is active due to a DB schema change. For more information, see VMware knowledge base article 55938.
Workaround: To patch your system to vCenter Server 6.7 Update 1 from earlier versions of vCenter Server 6.7, you must remove vCenter Server High Availability and delete passive and witness nodes. After the upgrade, you must re-create your vCenter Server High Availability clusters.
- Windows vCenter Server 6.0.x or 6.5.x upgrade to vCenter Server 6.7 fails if vCenter Server contains non-ASCII or high-ASCII named 5.5 host profiles
When a source Windows vCenter Server 6.0.x or 6.5.x contains vCenter Server 5.5.x host profiles named with non-ASCII or high-ASCII characters, UpgradeRunner fails to start during the upgrade pre-check process.
Workaround: Before upgrading Windows vCenter Server 6.0.x or 6.5.x to vCenter Server 6.7, upgrade the ESXi 5.5.x with the non-ASCII or high-ASCII named host profiles to ESXi 6.0.x or 6.5.x, then update the host profile from the upgraded host by clicking Copy setting from the hosts.
- Upgrade to vCenter Server Appliance 6.7 Update 1 from vCenter Server Appliance 6.5 Update 2 and later, using custom HTTP and HTTPS ports, might fail
Upgrades from vCenter Server Appliance 6.5 Update 2 and later, using custom HTTP and HTTPS ports, to vCenter Server Appliance 6.7 Update 1 might fail. You might see the issue regardless if you use the GUI or CLI installer.
Workaround: None
- Converging an external Platform Services Controller to a vCenter Server might fail if the Platform Services Controller uses a custom HTTPS port
You might fail to converge an external Platform Services Controller to a vCenter Server system, if the vCenter Server system is configured with the default HTTPS port, 443, and the Platform Services Controller node is configured with a custom value for the HTTPS port. The operation fails in the firstboot stage due to convergence issues.
Workaround: Change the HTTPS port value to the default value, 443, for Platform Services Controller nodes before running vCenter External to Embedded Convergence tool. You can run the following commands to do the same:
/usr/lib/vmware-vmafd/bin/vmafd-cli set-dc-port --server-name localhost --dc-port 443/usr/lib/vmware-vmafd/bin/vmafd-cli set-rhttpproxy-port --server-name localhost --rhttpproxy-port 443
- You cannot run the camregister command with the -x option if the vCenter Single Sign-On password contains non-ASCII characters
When you run the camregister command with the -x file option, for example, to register the vSphere Authentication Proxy, the process fails with an access denied error when the vCenter Single Sign-On password contains non-ASCII characters.
Workaround: Either set up the vCenter Single Sign-On password with ASCII characters, or use the –p password option when you run the camregister command to enter the vCenter Single Sign-On password that contains non-ASCII characters.
- The Bash shell and SSH login are disabled after upgrading to vCenter Server 6.7
After upgrading to vCenter Server 6.7, you are not able to access the vCenter Server Appliance using either the Bash shell or SSH login.
Workaround:
- After successfully upgrading to vCenter Server 6.7, log in to the vCenter Server Appliance Management Interface. In a Web browser, go to: https://appliance_ip_address_or_fqdn:5480
- Log in as root.
The default root password is the password you set while deploying the vCenter Server Appliance.
-
Click Access, and click Edit.
-
Edit the access settings for the Bash shell and SSH login.
When enabling Bash shell access to the vCenter Server Appliance, enter the number of minutes to keep access enabled.
-
Click OK to save the settings.
- Management node migration is blocked if vCenter Server for Windows 6.0 is installed on Windows Server 2008 R2 without previously enabling Transport Layer Security 1.2
This issue occurs if you are migrating vCenter Server for Windows 6.0 using an external Platform Services Controller (an MxN topology) on Windows Server 2008 R2. After migrating the external Platform Services Controller, when you run Migration Assistant on the Management node it fails, reporting that it cannot retrieve the Platform Services Controller version. This error occurs because Windows Server 2008 R2 does not support Transport Layer Security (TLS) 1.2 by default, which is the default TLS protocol for Platform Services Controller 6.7.
Workaround: Enable TLS 1.2 for Windows Server 2008 R2.1.
- Navigate to the registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols
- Create a new folder and label it
TLS 1.2.
- Create two new keys with the
TLS 1.2 folder, and name the keys Client and Server.
- Under the Client key, create two DWORD (32-bit) values, and name them DisabledByDefault and Enabled.
- Under the Server key, create two DWORD (32-bit) values, and name them DisabledByDefault and Enabled.
- Ensure that the Value field is set to 0 and that the Base is Hexadecimal for DisabledByDefault.
- Ensure that the Value field is set to 1 and that the Base is Hexadecimal for Enabled.
- Reboot the Windows Server 2008 R2 computer.
For more information on using TLS 1.2 with Windows Server 2008 R2, refer to the operating system vendor's documentation.
- vCenter Server containing host profiles with version less than 6.0 fails during upgrade to version 6.7
vCenter Server 6.7 does not support host profiles with version less than 6.0. To upgrade to vCenter Server 6.7, you must first upgrade the host profiles to version 6.0 or later, if you have any of the following components:
- ESXi host(s) version - 5.1 or 5.5
- vCenter server version - 6.0 or 6.5
- Host profiles version - 5.1 or 5.5
Workaround: See KB 52932
- After upgrading to vCenter Server 6.7, any edits to the ESXi host's /etc/ssh/sshd_config file are discarded, and the file is restored to the vCenter Server 6.7 default configuration
Due to changes in the default values in the /etc/ssh/sshd_config file, the vCenter Server 6.7 upgrade replaces any manual edits to this configuration file with the default configuration. This change was necessary as some prior settings (for example, permitted ciphers) are no longer compatible with current ESXi behavior, and prevented SSHD (SSH daemon) from starting correctly.
CAUTION: Editing /etc/ssh/sshd_config is not recommended. SSHD is disabled by default, and the preferred method for editing the system configuration is through the VIM API (including the ESXi Host Client interface) or ESXCLI.
Workaround: If edits to /etc/ssh/sshd_config are needed, you can apply them after successfully completing the vCenter Server 6.7 upgrade. The default configuration file now contains a version number. Preserve the version number to avoid overwriting the file.
For further information on editing the /etc/ssh/sshd_config file, see the following Knowledge Base articles:
- For information on enabling public/private key authentication, see Knowledge Base article KB 1002866
- For information on changing the default SSHD configuration, see Knowledge Base article KB 1020530
Security Features Issues
- Virtualization Based Security (VBS) on vSphere in Windows Guest OSs RS1, RS2 and RS3 require HyperV to be enabled in the Guest OS.
Virtualization Based Security (VBS) on vSphere in Windows Guest OSs RS1, RS2 and RS3 require HyperV to be enabled in the Guest OS.
Workaround: Enable Hyper-V Platform on Windows Server 2016. In the Server Manager, under Local Server select Manage -> Add Roles and Features Wizard and under Role-based or feature-based installation select Hyper-V from the server pool and specify the server roles. Choose defaults for Server Roles, Features, Hyper-V, Virtual Switches, Migration and Default Stores. Reboot the host.
Enable Hyper-V on Windows 10: Browse to Control Panel -> Programs -> Turn Windows features on or off. Check the Hyper-V Platform which includes the Hyper-V Hypervisor and Hyper-V Services. Uncheck Hyper-V Management Tools. Click OK. Reboot the host.
Networking Issues
- Hostprofile PeerDNS flags do not work in some scenarios
If PeerDNS for IPv4 is enabled for a vmknic on a stateless host that has an associated host profile, the iPv6PeerDNS might appear with a different state in the extracted host profile after the host reboots.
Workaround: None.
- When you upgrade vSphere Distributed Switches to version 6.6, you might encounter a few known issues
During upgrade, the connected virtual machines might experience packet loss for a few seconds.
Workaround: If you have multiple vSphere Distributed Switches that need to be upgraded to version 6.6, upgrade the switches sequentially.
Schedule the upgrade of vSphere Distributed Switches during a maintenance window, set DRS mode to manual, and do not apply DRS recommendations for the duration of the upgrade.
For more details about known issues and solutions, see KB 52621
- VM fails to power on when Network I/O Control is enabled and all active uplinks are down
A VM fails to power on when Network I/O Control is enabled and the following conditions are met:
- The VM is connected to a distributed port group on a vSphere distributed switch
- The VM is configured with bandwidth allocation reservation and the VM's network adapter (vNIC) has a reservation configured
- The distributed port group teaming policy is set to Failover
- All active uplinks on the distributed switch are down. In this case, vSphere DRS cannot use the standby uplinks and the VM fails to power on.
Workaround: Move the available standby adapters to the active adapters list in the teaming policy of the distributed port group.
- Network flapping on a NIC that uses qfle3f driver might cause ESXi host to crash
The qfle3f driver might cause the ESXi host to crash (PSOD) when the physical NIC that uses the qfle3f driver experiences frequent link status flapping every 1-2 seconds.
Workaround: Make sure that network flapping does not occur. If the link status flapping interval is more than 10 seconds, the qfle3f driver does not cause ESXi to crash. For more information, see KB 2008093.
- Port Mirror traffic packets of ERSPAN Type III fail to be recognized by packet analyzers
A wrong bit that is incorrectly introduced in ERSPAN Type III packet header causes all ERSPAN Type III packets to appear corrupt in packet analyzers.
Workaround: Use GRE or ERSPAN Type II packets, if your traffic analyzer supports these types.
- DNS configuration esxcli commands are not supported on non-default TCP/IP stacks
DNS configuration of non-default TCP/IP stacks is not supported. Commands such as esxcli network ip dns server add -N vmotion -s 10.11.12.13 do not work.
Workaround: Do not use DNS configuration esxcli commands on non-default TCP/IP stacks.
- Compliance check fails with an error when applying a host profile with enabled default IPv4 gateway for vmknic interface
When applying a host profile with enabled default IPv4 gateway for vmknic interface, the setting is populated with "0.0.0.0" and does not match the host info, resulting with the following error:
IPv4 vmknic gateway configuration doesn't match the specification
Workaround:
- Edit the host profile settings.
- Navigate to Networking configuration > Host virtual nic or Host portgroup > (name of the vSphere Distributed Switch or name of portgroup) > IP address settings.
- From the Default gateway Vmkernal Network Adapter (IPv4) drop-down menu, select Choose a default IPv4 gateway for the vmknic and enter the Vmknic Default IPv4 gateway.
- Intel Fortville series NICs cannot receive Geneve encapsulation packets with option length bigger than 255 bytes
If you configure Geneve encapsulation with option length bigger than 255 bytes, the packets are not received correctly on Intel Fortville NICs X710, XL710, and XXV710.
Workaround: Disable hardware VLAN stripping on these NICs by running the following command:
esxcli network nic software set --untagging=1 -n vmnicX.
- RSPAN_SRC mirror session fails after migration
When a VM connected to a port assigned for RSPAN_SRC mirror session is migrated to another host, and there is no required pNic on the destination network of the destination host, then the RSPAN_SRC mirror session fails to configure on the port. This causes the port connection to fail failure but the vMotion migration process succeeds.
Workaround: To restore port connection failure, complete either one of the following:
- Remove the failed port and add a new port.
- Disable the port and enable it.
The mirror session fails to configure, but the port connection is restored.
Storage Issues
- NFS datastores intermittently become read-only
A host's NFS datastores may become read-only when the NFS vmknic temporarily loses its IP address or after a stateless hosts reboot.
Workaround: You can unmount and remount the datastores to regain connectivity through the NFS vmknic. You can also set the NFS datastore write permission to both the IP address of the NFS vmknic and the IP address of the Management vmknic.
- When editing a VM's storage policies, selecting Host-local PMem Storage Policy fails with an error
In the Edit VM Storage Policies dialog, if you select Host-local PMem Storage Policy from the dropdown menu and click OK, the task fails with one of these errors:
The operation is not supported on the object.
or
Incompatible device backing specified for device '0'"Detailed
Workaround: You cannot apply the Host-local PMem Storage Policy to VM home. For a virtual disk, you can use the migration wizard to migrate the virtual disk and apply the Host-local PMem Storage Policy.
- Datastores might appear as inaccessible after ESXi hosts in a cluster recover from a permanent device loss state
This issue might occur in the environment where the hosts in the cluster share a large number of datastore, for example, 512 to 1000 datastores.
After the hosts in the cluster recover from the permanent device loss condition, the datastores are mounted successfully at the host level. However, in vCenter Server, several datastores might continue to appear as inaccessible for a number of hosts.
Workaround: On the hosts that show inaccessible datastores in the vCenter Server view, perform the Rescan Storage operation from vCenter Server.
- Migration of a virtual machine from a VMFS3 datastore to VMFS5 fails in a mixed ESXi 6.5 and 6.7 host environment
If you have a mixed host environment, you cannot migrate a virtual machine from a VMFS3 datastore connected to an ESXi 6.5 host to a VMFS5 datastore on an ESXi 6.7 host.
Workaround: Upgrade the VMFS3 datastore to VMFS5 to be able to migrate the VM to the ESXi 6.7 host.
- Warning message about a VMFS3 datastore remains unchanged after you upgrade the VMFS3 datastore using the CLI
Typically, you use the CLI to upgrade the VMFS3 datastore that failed to upgrade during an ESXi upgrade. The VMFS3 datastore might fail to upgrade due to several reasons including the following:
- No space is available on the VMFS3 datastore.
- One of the extents on the spanned datastore is offline.
After you fix the reason of the failure and upgrade the VMFS3 datastore to VMFS5 using the CLI, the host continues to detect the VMFS3 datastore and reports the following error:
Deprecated VMFS (ver 3) volumes found. Upgrading such volumes to VMFS (ver5) is mandatory for continued availability on vSphere 6.7 host.
Workaround: To remove the error message, restart hostd using the /etc/init.d/hostd restart command or reboot the host.
- The Mellanox ConnectX-4/ConnectX-5 native ESXi driver might exhibit performance degradation when its Default Queue Receive Side Scaling (DRSS) feature is turned on
Receive Side Scaling (RSS) technology distributes incoming network traffic across several hardware-based receive queues, allowing inbound traffic to be processed by multiple CPUs. In Default Queue Receive Side Scaling (DRSS) mode, the entire device is in RSS mode. The driver presents a single logical queue to OS and is backed by several hardware queues.
The native nmlx5_core driver for the Mellanox ConnectX-4 and ConnectX-5 adapter cards enables the DRSS functionality by default. While DRSS helps to improve performance for many workloads, it could lead to possible performance degradation with certain multi-VM and multi-vCPU workloads.
Workaround: If significant performance degradation is observed, you can disable the DRSS functionality.
- Run the esxcli system module parameters set -m nmlx5_core -p DRSS=0 RSS=0 command.
- Reboot the host.
- Datastore name does not extract to the Coredump File setting in the host profile
When you extract a host profile, the Datastore name field is empty in the Coredump File setting of the host profile. Issue appears when using esxcli command to set coredump.
Workaround:
- Extract a host profile from an ESXi host.
- Edit the host profile settings and navigate to General System Settings > Core Dump Configuration > Coredump File.
- Select Create the Coredump file with an explicit datastore and size option and enter the Datastore name, where you want the Coredump File to reside.
- Native software FCoE adapters configured on an ESXi host might disappear when the host is rebooted
After you successfully enable the native software FCoE adapter (vmhba) supported by the vmkfcoe driver and then reboot the host, the adapter might disappear from the list of adapters. This might occur when you use Cavium QLogic 57810 or QLogic 57840 CNAs supported by the qfle3 driver.
Workaround: To recover the vmkfcoe adapter, perform these steps:
- Run the esxcli storage core adapter list command to make sure that the adapter is missing from the list.
- Verify the vSwitch configuration on vmnic associated with the missing FCoE adapter.
- Run the following command to discover the FCoE vmhba:
- On a fabric setup:
#esxcli fcoe nic discover -n vmnic_number
- On a VN2VN setup:
#esxcli fcoe nic discover -n vmnic_number
- Attempts to create a VMFS datastore on an ESXi 6.7 host might fail in certain software FCoE environments
Your attempts to create the VMFS datastore fail if you use the following configuration:
- Native software FCoE adapters configured on an ESXi 6.7 host.
- Cavium QLogic 57810 or 57840 CNAs.
- Cisco FCoE switch connected directly to an FCoE port on a storage array from the Dell EMC VNX5300 or VNX5700 series.
Workaround: None.
As an alternative, you can switch to the following end-to-end configuration:
ESXi host > Cisco FCoE switch > FC switch > storage array from the DELL EMC VNX5300 and VNX5700 series.
Backup and Restore Issues
- Windows Explorer displays some backups with unicode differently from how browsers and file system paths show them
Some backups containing unicode display differently in the Windows Explorer file system folder than they do in browsers and file system paths.
Workaround: Using http, https, or ftp, you can browse backups with your web browser instead of going to the storage folder locations through Windows Explorer.
vCenter Server Appliance, vCenter Server, vSphere Web Client, and vSphere Client Issues
- The time synchronization mode setting is not retained when upgrading vCenter Server Appliance
If NTP time synchronization is disabled on a source vCenter Server Appliance, and you perform an upgrade to vCenter Server Appliance 6.7, after the upgrade has successfully completed NTP time synchronization will be enabled on the newly upgraded appliance.
Workaround:
- After successfully upgrading to vCenter Server Appliance 6.7, log into the vCenter Server Appliance Management Interface as root.
The default root password is the password you set while deploying the vCenter Server Appliance.
https://IP_or_FQDN_of_appliance:5480
- In the vCenter Server Appliance Management Interface, click Time.
- In the Time Synchronization pane, click Edit.
- From the Mode drop-down menu, select Disabled.
The newly upgraded vCenter Server Appliance 6.7 will no longer use NTP time synchronization, and will instead use the system time zone settings.
- Login to vSphere Web Client with Windows session authentication fails on Firefox browsers of version 54 or later
If you use Firefox of version 54 or later to log in to the vSphere Web Client, and you use your Windows session for authentication, the VMware Enhanced Authentication Plugin might fail to populate your user name and to log you in.
Workaround: If you are using Windows session authentication to log in to the vSphere Web Client, use one of the following browsers: Internet Explorer, Chrome, or Firefox of version 53 and earlier.
- vCenter hardware health alarm notifications are not triggered in some instances
When multiple sensors in the same category on an ESXi host are tripped within a time span of less than five minutes, traps are not received and email notifications are not sent.
Workaround: None. You can check the hardware sensors section for any alerts.
- The vSphere Client and vSphere Web Client might not reflect update from vCenter Server 6.7 to vCenter Server 6.7 Update 1 for vCenter Server for Windows
If you update vCenter Server for Windows from vCenter Server 6.7 to vCenter Server 6.7 Update 1, the build number details for vpxd in the Summary tab of both the vSphere Client and vSphere Web Client might not reflect the update and show version 6.7.0.
Workaround: None.
- When using the VCSA Installer Time Sync option, you must connect the target ESX to the NTP server in the Time & Date Setting from the ESX Management
If you want to select Time Sync with NTP server from the VCSA Installer->Stage2->Appliance configuration->Time Sync option (ESX/NTP server), you also need to have the target ESX already connected to NTP server in the Time&Date Setting from the ESX Management, otherwise it'll fail in installation.
Workaround:
- Set the Time Sync option in stage2->Appliance configuration to sync with ESX
- Set the Time Sync option in stage2->Appliance configuration to sync with NTP Servers, make sure both the ESX and VC are set to connect to NTP servers.
- When you monitor Windows vCenter Server health, an error message appears
Health service is not available for Windows vCenter Server. If you select the vCenter Server, and click Monitor > Health, an error message appears:
Unable to query vSAN health information. Check vSphere Client logs for details.
This problem can occur after you upgrade the Windows vCenter Server from release 6.0 Update 1 or 6.0 Update 2 to release 6.7. You can ignore this message.
Workaround: None. Users can access vSAN health information through the vCenter Server Appliance.
- vCenter hardware health alarms do not function with earlier ESXi versions
If ESXi version 6.5 Update 1 or earlier is added to vCenter 6.7, hardware health related alarms will not be generated when hardware events occur such as high CPU temperatures, FAN failures, and voltage fluctuations.
Workaround: None.
- vCenter Server stops working in some cases when using vmodl to edit or expand a disk
When you configure a VM disk in a Storage DRS-enabled cluster using the latest vmodl, vCenter Server stops working. A previous workaround using an earlier vmodl no longer works and will also cause vCenter Server to stop working.
Workaround: None
- vCenter Server for Windows migration to vCenter Server Appliance fails with error
When you migrate vCenter Server for Windows 6.0.x or 6.5.x to vCenter Server Appliance 6.7, the migration might fail during the data export stage with the error: The compressed zip folder is invalid or corrupted.
Workaround: You must zip the data export folder manually and follow these steps:
- In the source system, create an environment variable MA_INTERACTIVE_MODE.
- Go to Computer > Properties > Advanced system settings > Environment Variables > System Variables > New.
- Enter "MA_INTERACTIVE_MODE" as variable name with value 0 or 1.
- Start the VMware Migration Assistant and provide your password.
- Start the Migration from the client machine. The migration will pause, and the Migration Assistant console will display the message
To continue the migration, create the export.zip file manually from the export data (include export folder).
- NOTE: Do not press any keys or tabs on the Migration Assistant console.
- Go to the
%appdata%\vmware\migration-assistant folder.
- Delete the export.zip created by the Migration Assistant.
- To continue the migration, manually create the export.zip file from the export folder.
- Return to the Migration Assistant console. Type
Y and press Enter.
- Discrepancy between the build number in VAMI and the build number in the vSphere Client
In vSphere 6.7, the VAMI summary tab displays the ISO build for the vCenter Server and vCenter Server Appliance products. The vSphere Client summary tab displays the build for the vCenter product, which is a component within the vCenter Server product.
Workaround: None
- vCenter Server Appliance 6.7 displays an error message in the Available Update section of the vCenter Server Appliance Management Interface (VAMI)
The Available Update section of the vCenter Server Appliance Management Interface (VAMI) displays the following error message:
Check the URL and try again.
This message is generated when the vCenter Server Appliance searches for and fails to find a patch or update. No functionality is impacted by this issue. This issue will be resolved with the release of the first patch for vSphere 6.7.
Workaround: None. No functionality is impacted by this issue.
Virtual Machine Management Issues
- Name of the virtual machine in the inventory changes to its path name
This issue might occur when a datastore where the VM resides enters the All Paths Down state and becomes inaccessible. When hostd is loading or reloading VM state, it is unable to read the VM's name and returns the VM path instead. For example, /vmfs/volumes/123456xxxxxxcc/cs-00.111.222.333.
Workaround: After you resolve the storage issue, the virtual machine reloads, and its name is displayed again.
- You must select the "Secure boot" Platform Security Level when enabling VBS in a Guest OS on AMD systems
On AMD systems, vSphere virtual machines do not provide a vIOMMU. Since vIOMMU is required for DMA protection, AMD users cannot select "Secure Boot and DMA protection" in the Windows Group Policy Editor when they "Turn on Virtualization Based Security". Instead select "Secure boot." If you select the wrong option it will cause VBS services to be silently disabled by Windows.
Workaround: Select "Secure boot" Platform Security Level in a Guest OS on AMD systems.
- You cannot hot add memory and CPU for Windows VMs when Virtualization Based Security (VBS) is enabled within Windows
Virtualization Based Security (VBS) is a new feature introduced in Windows 10 and Windows Server 2016. vSphere supports running Windows with VBS enabled starting in the vSphere 6.7 release. However, Hot add of memory and CPU will not operate for Windows VMs when Virtualization Based Security (VBS) is enabled.
Workaround: Power-off the VM, change memory or CPU settings and power-on the VM.
- Snapshot tree of a linked-clone VM might be incomplete after a vSAN network recovery from a failure
A vSAN network failure might impact accessibility of vSAN objects and VMs. After a network recovery, the vSAN objects regain accessibility. The hostd service reloads the VM state from storage to recover VMs. However, for a linked-clone VM, hostd might not detect that the parent VM namespace has recovered its accessibility. This results in the VM remaining in inaccessible state and VM snapshot information not being displayed in vCenter Server.
Workaround: Unregister the VM, then re-register it to force the hostd to reload the VM state. Snapshot information will be loaded from storage.
- The Virtual Appliance Management Interface might display a 0- message or a blank page during patching from vCenter Server 6.7 to later versions
The Virtual Appliance Management Interface might display a 0- message or a blank page during patching from vCenter Server 6.7 to later versions, if calls from the interface fail to reach the back end applmgmt service. You might also see the message Unable to get historical data import status. Check Server Status.
Workaround: These are not failure messages. Refresh the browser and log in to the Virtual Appliance Management Interface again once the reboot of appliance in the back end is complete.
- The Ready to Complete page of the Register Virtual Machine wizard displays only one horizontal line
The Ready to Complete page of the Register Virtual Machine wizard might display content similar to one horizontal line due to a rendering issue. This issue does not affect the workflow of the wizard.
Workaround: None
- An OVF Virtual Appliance fails to start in the vSphere Client
The vSphere Client does not support selecting vService extensions in the Deploy OVF Template wizard. As a result, if an OVF virtual appliance uses vService extensions and you use the vSphere Client to deploy the OVF file, the deployment succeeds, but the virtual appliance fails to start.
Workaround: Use the vSphere Web Client to deploy OVF virtual appliances that use vService extensions.
vSphere HA and Fault Tolerance Issues
- When you configure Proactive HA in Manual/MixedMode in vSphere 6.7 RC build you are prompted twice to apply DRS recommendations
When you configure Proactive HA in Manual/MixedMode in vSphere 6.7 RC build and a red health update is sent from the Proactive HA provider plug-in, you are prompted twice to apply the recommendations under Cluster -> Monitor -> vSphere DRS -> Recommendations. The first prompt is to enter the host into maintenance mode. The second prompt is to migrate all VMs on a host entering maintenance mode. In vSphere 6.5, these two steps are presented as a single recommendation for entering maintenance mode, which lists all VMs to be migrated.
Workaround: There is no impact to work flow or results. You must apply the recommendations twice. If you are using automated scripts, you must modify the scripts to include the additional step.
- Lazy import upgrade interaction when VCHA is not configured
The VCHA feature is available as part of 6.5 release. As of 6.5, a VCHA cluster cannot be upgraded while preserving the VCHA configuration. The recommended approach for upgrade is to first remove the VCHA configuration either through vSphere Client or by calling a destroy VCHA API. So for lazy import upgrade workflow without VCHA configuration, there is no interaction with VCHA.
Do not configure a fresh VCHA setup while lazy import is in progress. The VCHA setup requires cloning the Active VM as Passive/Witness VM. As a result of an ongoing lazy import, the amount of data that needs to be cloned is large and may lead to performance issues.
Workaround: None.
- You cannot add ESXi hosts running vSphere Fault Tolerance workloads to a vCenter Server system by using the vSphere Client
Attempts to add ESXi hosts running vSphere Fault Tolerance workloads to a vCenter Server system by using the vSphere Client might fail with the error Cannot add a host with virtual machines that have Fault Tolerance turned on as a stand-alone host.
Workaround: As alternatives, you can:
- Schedule a task to add the host and execute it immediately.
- In the vSphere Client, navigate to Configure > Scheduled tasks for a selected cluster.
- Select New scheduled task > Add Host.
- Schedule a time to run the task.
- Add a host and run the task.
- Delete the task after the host is added.
- Use the vSphere Web Client to add the host. Login to the vSphere Web Client and execute the standard add host workflow.
- Turn off the fault tolerance virtual machines temporarily, add the host to the new vCenter Server system, and then turn it on back again.
- vCenter Server High Availability cluster configuration by using an NSX-T logical switch might fail
Configuration of a vCenter Server High Availability cluster by using an NSX-T logical switch might fail with the error Failed to connect peer node.
Workaround: Configure vCenter Server High Availability clusters by using a vSphere Distributed Switch.
Auto Deploy and Image Builder Issues
- Reboot of an ESXi stateless host resets the numRxQueue value of the host
When an ESXi host provisioned with vSphere Auto Deploy reboots, it loses the previously set numRxQueue value. The Host Profiles feature does not support saving the numRxQueue value after the host reboots.
Workaround: After the ESXi stateless host reboots:
- Remove the vmknic from the host.
- Create a vmknic on the host with the expected numRxQueue value.
- After caching on a drive, if the server is in the UEFI mode, a boot from cache does not succeed unless you explicitly select the device to boot from the UEFI boot manager
In case of Stateless Caching, after the ESXi image is cached on a 512n, 512e, USB, or 4Kn target disk, the ESXi stateless boot from autodeploy might fail on a system reboot. This occurs if autodeploy service is down.
The system attempts to search for the cached ESXi image on the disk, next in the boot order. If the ESXi cached image is found, the host is booted from it. In legacy BIOS, this feature works without problems. However, in the UEFI mode of the BIOS, the next device with the cached image might not be found. As a result, the host cannot boot from the image even if the image is present on the disk.
Workaround: If autodeploy service is down, on the system reboot, manually select the disk with the cached image from the UEFI Boot Manager.
- A stateless ESXi host boot time might take 20 minutes or more
The booting of a stateless ESXi host with 1,000 configured datastores might require 20 minutes or more.
Workaround: None.
Miscellaneous Issues
- ESXi might fail during reboot with VMs running on the iSCSI LUNs claimed by the qfle3i driver
ESXi might fail during reboot with VMs running on the iSCSI LUNs claimed by the qfle3i driver if you attempt to reboot the server with VMs in the running I/O state.
Workaround: First power off VMs and then reboot the ESXi host.
- VXLAN stateless hardware offloads are not supported with Guest OS TCP traffic over IPv6 on UCS VIC 13xx adapters
You may experience issues with VXLAN encapsulated TCP traffic over IPv6 on Cisco UCS VIC 13xx adapters configured to use the VXLAN stateless hardware offload feature. For VXLAN deployments involving Guest OS TCP traffic over IPV6, TCP packets subject to TSO are not processed correctly by the Cisco UCS VIC 13xx adapters, which causes traffic disruption. The stateless offloads are not performed correctly. From a TCP protocol standpoint this may cause incorrect packet checksums being reported to the ESXi software stack, which may lead to incorrect TCP protocol processing in the Guest OS.
Workaround: To resolve this issue, disable the VXLAN stateless offload feature on the Cisco UCS VIC 13xx adapters for VXLAN encapsulated TCP traffic over IPV6. To disable the VXLAN stateless offload feature in UCS Manager, disable the Virtual Extensible LAN field in the Ethernet Adapter Policy. To disable the VXLAN stateless offload feature in the CIMC of a Cisco C-Series UCS server, uncheck Enable VXLAN field in the Ethernet Interfaces vNIC properties section.
- Significant time might be required to list a large number of unresolved VMFS volumes using the batch QueryUnresolvedVmfsVolume API
ESXi provides the batch QueryUnresolvedVmfsVolume API, so that you can query and list unresolved VMFS volumes or LUN snapshots. You can then use other batch APIs to perform operations, such as resignaturing specific unresolved VMFS volumes. By default, when the API QueryUnresolvedVmfsVolume is invoked on a host, the system performs an additional filesystem liveness check for all unresolved volumes that are found. The liveness check detects whether the specified LUN is mounted on other hosts, whether an active VMFS heartbeat is in progress, or if there is any filesystem activity. This operation is time consuming and requires at least 16 seconds per LUN. As a result, when your environment has a large number of snapshot LUNs, the query and listing operation might take significant time.
Workaround: To decrease the time of the query operation, you can disable the filesystem liveness check.
- Log in to your host as root.
- Open the configuration file for hostd using a text editor. The configuration file is located in /etc/vmware/hostd/config.xml under plugins/hostsvc/storage node.
- Add the checkLiveFSUnresolvedVolume parameter and set its value to FALSE. Use the following syntax:
<checkLiveFSUnresolvedVolume>FALSE</checkLiveFSUnresolvedVolume>
As an alternative, you can set the ESXi Advanced option VMFS.UnresolvedVolumeLiveCheck to FALSE in the vSphere Client.
- Importing a .csv file overwrites user input during host customization.
User input in the Customize hosts pane is overwritten by the import process and the values from the .csv file.
Workaround: Import the .csv file before adding manual changes in the Customize hosts pane.
- The vCenter Server Convergence Tool might fail to convert an external Platform Services Controller to an embedded Platform Services Controller due to conflicting IP and FQDN
If you have configured an external Platform Services Controller with an IP address as an optional FQDN field during the deployment, the vCenter Server Convergence Tool might fail to convert the external Platform Services Controller to an embedded Platform Services Controller because of a name conflict.
Workaround: Do not use the vCenter Server Convergence Tool for Platform Services Controllers installed with an IP address as an alternative or addition to the FQDN address.
- If you repoint and reconfigure the setup of a Platform Services Controller, the restore process might fail
If you repoint and reconfigure the setup of a Platform Services Controller, the restore process might fail due to a stale service ID entry.
Workaround: Follow the steps in VMware knowledge base article 2131327 to clean up the stale service ID before proceeding with restore.
- Attempts to log in to a vCenter Server system after an upgrade to vCenter Server 6.7 might fail with a credentials validation error
After an upgrade of your system to vCenter Server 6.7, if you try to log in to the system by using either the vSphere Web Client or vSphere Client, and a security token or smartcard, the login might fail with an error Unable to validate the submitted credential.
Workaround: Remove and re-add the identity source. For more information, see Add or Edit a vCenter Single Sign-On Identity Source.
- An Enhanced vMotion Compatibility (EVC) cluster might show new CPU IDs such as IBPB even if you revert an ESXi host to an older version
If you revert an ESXi host to an older version of ESXi, an EVC cluster might expose new CPU IDs, such as IBRS, STIBP and IBPB, even though the host does not have any of the features.
Workaround: This issue is resolved in this release. However, a host that does not meet the requirements of an EVC cluster does not automatically reconnect and you must remove it from the cluster.
- Some vCenter Server plug-ins might not correctly render the dark theme mode in the vSphere Client
If you change color schemes in the vSphere Client to display the interface in a dark theme, some vCenter Server plug-ins might not correctly render the mode.
Workaround: None
- If you enable per-VM EVC, virtual machines might fail to power on
If you install or use for upgrade only VMware vCenter Server 6.7 Update 1, but do not apply ESXi 6.7 Update 1, and if you configure or reconfigure per-VM EVC, virtual machines on unpatched hosts might fail to power on. You might also see the issue if you enable cluster-level EVC and even one of the hosts in the cluster is not patched with the latest update. The new CPU IDs of that cluster might not be available on the cluster. In such a cluster, if you configure or reconfigure per-VM EVC, virtual machines might fail to power on.
Workaround: Before you configure or reconfigure per-VM EVC, upgrade all the standalone ESXi hosts, as well as hosts inside a cluster, to the latest update for hypervisor-assisted guest mitigation for guest operating systems.
- Edits to the DNS settings might cause deletion of the IPv6 loopback address from the /etc/resolv.conf and /etc/systemd/resolved.conf files
Edits to the DNS settings by using either the Appliance Management Interface, appliance shell, or the vSphere Web Client, might cause deletion of the IPv6 loopback address from the/etc/resolv.conf and /etc/systemd/resolved.conf files.
Workaround: To avoid deletion of the IPv6 loopback address, edit the resolv.conf files by using the Bash shell:
- In the
/etc/resolv.conf file, set the following parameters: nameserver: ::1
nameserver: <dnsserver 1>
nameserver: <dnsserver 2>
- In the
/etc/systemd/resolved.conf file, set the following parameters: [Resolve]
LLMNR=false
DNS=::1 <dnsserver 1> <dnsserver 2>
- The SSH service might be disabled after an external Platform Services Controller converts to an embedded Platform Services Controller
If you convert an external Platform Services Controller to an embedded Platform Services Controller, the SSH service might be disabled based on Active Directory policies and restrictions.
Workaround: Manually enable the SSH service after the conversion is complete.
- After upgrade to ESXi 6.7, networking workloads on Intel 10GbE NICs cause higher CPU utilization
If you run certain types of networking workloads on an upgraded ESXi 6.7 host, you might see a higher CPU utilization under the following conditions:
- The NICs on the ESXi host are from the Intel 82599EB or X540 families
- The workload involves multiple VMs that run simultaneously and each VM is configured with multiple vCPUs
- Before the upgrade to ESXi 6.7, the VMKLinux ixgbe driver was used
Workaround: Revert to the legacy VMKLinux ixgbe driver:
- Connect to the ESXi host and run the following command:
# esxcli system module set -e false -m ixgben
- Reboot the host.
Note: The legacy VMKLinux ixgbe inbox driver version 3.7.x does not support Intel X550 NICs. Use the VMKLinux ixgbe async driver version 4.x with Intel X550 NICs.
- Initial install of DELL CIM VIB might fail to respond
After you install a third-party CIM VIB it might fail to respond.
Workaround: To fix this issue, enter the following two commands to restart sfcbd:
esxcli system wbem set --enable falseesxcli system wbem set --enable true
To collapse the list of previous known issues, click here.
