VMware Cloud Foundation 5.0 Release Notes

VMware Cloud Foundation 5.0 \| 01 JUN 2023 \| Build 21822418 VMware Cloud Foundation 5.0.0.1 \| 29 SEPT 2023 \| Build 22485660 Check for additions and updates to these release notes.

VMware Cloud Foundation 5.0 | 01 JUN 2023 | Build 21822418

VMware Cloud Foundation 5.0.0.1 | 29 SEPT 2023 | Build 22485660

Check for additions and updates to these release notes.

Upgrade Notice

Note: VCF+/VCF-S customers are now able to upgrade to VCF 5.0.

What's New

The VMware Cloud Foundation (VCF) 5.0 release includes the following:

In-place upgrade: VCF 4.3.x, VCF 4.4.x, VCF 4.5, and VCF 4.5.1 can upgrade to VCF 5.0 from SDDC Manager UI (with internet access) or using the Bundle Transfer Utility (without internet access).
Isolated Workload Domains: Create and manage workload domains that can each join the management domain's vCenter Single Sign-On domain or a new vCenter Single Sign-On domain that is not used by any other workload domain.
BOM component interoperability data checks: Auto-checks of BOM component interoperability within SDDC Manager.
BOM product re-licensing: Ability to perform relicensing of VCF BOM components from SDDC Manager UI.
Licensing update: SDDC Manager is now licensed as part of VCF. Therefore, a separate SDDC Manager license key is no longer required.
Improvements to upgrade prechecks: Upgrade prechecks now provides granular control over version-specific and individual component prechecks.
Improvements to Configuration Update workflow: Configuration Updates can be applied between upgrade applications or can be saved to be applied at a later time.
New upgrade option for ESXi Upgrade: If there are any powered-off virtual machines in the chosen cluster to be upgraded, they will only be evacuated during the upgrade process if this option is enabled. Otherwise, they will not be migrated specifically for the purpose of the upgrade.
Scaled support: Ability to upgrade 10 clusters (hosts) across 5 workload domains (5*10) simultaneously in ESXi, allowing for parallel upgrades to occur.
In-product feedback: SDDC Manager can prompt customers at random intervals to provide feedback related to VMware Cloud Foundation.
VMware NSX-T Data Center rebranding: Starting with version 4.0, VMware NSX-T Data Center is known as VMware NSX.
VMware Validated Solutions: All VMware Validated Solutions are updated to support VMware Cloud Foundation 5.0. Visit VMware Validated Solutions for the updated guides.
VMware Cloud Foundation with Skyline Health Diagnostics: New content available for Proactive Diagnostics of VMware Cloud Foundation with Skyline Health Diagnostics.
BOM updates: Updated Bill of Materials with new product versions.

Deprecation Notices

The VMware Imaging Appliance (VIA), included with the VMware Cloud Builder appliance to image ESXi servers, is deprecated in VMware Cloud Foundation 5.0 and will be removed in a future release.
In a future release, the "Connect Workload Domains" option from the vRealize Operations card located in SDDC Manager > Administration > vRealize Suite section will be removed. Additionally, related VCF Public API options will also be deprecated.

Starting with vRealize Operations 8.10, an improved functionality for connecting VCF Workload Domains to vRealize Operations is available directly from the vRealize Operations UI itself. Users are encouraged to use this method within the vRealize Operations UI for connecting VCF Workload Domains.
VMware Update Manager as the default option for management domain cluster will be deprecated in a subsequent release and vSphere Lifecycle Manager will become the default. This is to align with the vSphere deprecation of VMware Update Manager in future release as noted here: https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-703-release-notes.html

VMware Cloud Foundation Bill of Materials (BOM)

The VMware Cloud Foundation software product is comprised of the following software Bill-of-Materials (BOM). The components in the BOM are interoperable and compatible.

Software Component	Version	Date	Build Number
Cloud Builder VM	5.0	01 JUN 2023	21822418
SDDC Manager	5.0	01 JUN 2023	21822418
VMware vCenter Server Appliance	8.0 Update 1a	01 JUN 2023	21815093
VMware ESXi	8.0 Update 1a	01 JUN 2023	21813344
VMware vSAN Witness Appliance	8.0 Update 1	14 APR 2023	21495797
VMware NSX	4.1.0.2	01 JUN 2023	21761691
VMware vRealize Suite Lifecycle Manager	8.10 Patch 1	21 FEB 2023	21331275

VMware vSAN is included in the VMware ESXi bundle.
You can use vRealize Suite Lifecycle Manager to deploy vRealize Automation, vRealize Operations Manager, vRealize Log Insight, and Workspace ONE Access. vRealize Suite Lifecycle Manager determines which versions of these products are compatible and only allows you to install/upgrade to supported versions.
vRealize Log Insight content packs are installed when you deploy vRealize Log Insight.
The vRealize Operations Manager management pack is installed when you deploy vRealize Operations Manager.
You can access the latest versions of the content packs for vRealize Log Insight from the VMware Solution Exchange and the vRealize Log Insight in-product marketplace store.

VMware Software Edition License Information

The SDDC Manager software is licensed under the VMware Cloud Foundation license. As part of this product, the SDDC Manager software deploys specific VMware software products.

The following VMware software components deployed by SDDC Manager are licensed under the VMware Cloud Foundation license:

VMware ESXi
VMware vSAN
VMware NSX

The following VMware software components deployed by SDDC Manager are licensed separately:

VMware vCenter Server

NOTE Only one vCenter Server license is required for all vCenter Servers deployed in a VMware Cloud Foundation system.

For details about the specific VMware software editions that are licensed under the licenses you have purchased, see the VMware Cloud Foundation Bill of Materials (BOM).

For general information about the product, see VMware Cloud Foundation.

Supported Hardware

For details on supported configurations, see the VMware Compatibility Guide (VCG) and the Hardware Requirements section on the Prerequisite Checklist tab in the Planning and Preparation Workbook.

Documentation

To access the Cloud Foundation documentation, go to the VMware Cloud Foundation product documentation.

To access the documentation for VMware software products that SDDC Manager can deploy, see the product documentation and use the drop-down menus on the page to choose the appropriate version:

VMware vSphere product documentation, also has documentation about ESXi and vCenter Server
VMware vSAN product documentation
VMware NSX product documentation

Browser Compatibility and Screen Resolutions

The VMware Cloud Foundation web-based interface supports the latest two versions of the following web browsers:

Google Chrome
Mozilla Firefox
Microsoft Edge

For the Web-based user interfaces, the supported standard resolution is 1920 by 1080 pixels.

Installation and Upgrade Information

You can install VMware Cloud Foundation 5.0 as a new release or perform a sequential or skip-level upgrade to VMware Cloud Foundation 5.0.

Caution:

If you are upgrading to VMware Cloud Foundation 5.0 or above, read the critical information in KB article 91751.

Installing as a New Release

The new installation process has three phases:

Phase One: Prepare the Environment: The Planning and Preparation Workbook provides detailed information about the software, tools, and external services that are required to implement a Software-Defined Data Center (SDDC) with VMware Cloud Foundation, using a standard architecture model.
Phase Two: Image all servers with ESXi: Image all servers with the ESXi version mentioned in the Cloud Foundation Bill of Materials (BOM) section. See the VMware Cloud Foundation Deployment Guide for information on installing ESXi.
Phase Three: Install Cloud Foundation 5.0: See the VMware Cloud Foundation Deployment Guide for information on deploying Cloud Foundation.

Upgrading to Cloud Foundation 5.0

Important:

VMware Cloud Foundation 4.5.2 cannot upgrade to VMware Cloud Foundation 5.0 or 5.0.0.1. See KB article 94195 for more information.

You can perform a sequential or skip-level upgrade to VMware Cloud Foundation 5.0 from VMware Cloud Foundation 4.3.x or later. If your environment is at a version earlier than 4.3.x, you must upgrade the management domain and all VI workload domains to VMware Cloud Foundation 4.3.x or above and then upgrade to VMware Cloud Foundation 5.0. For more information see VMware Cloud Foundation Lifecycle Management.

Important:

Before you upgrade a vCenter Server, take a file-based backup. See Manually Back Up vCenter Server.

Note:

Since VMware Cloud Foundation 5.0.x disables the SSH service by default, scripts that rely on SSH being enabled on ESXi hosts will not work after upgrading to VMware Cloud Foundation 5.0.x. Update your scripts to account for this new behavior. See KB 86230 for information about enabling and disabling the SSH service on ESXi hosts.

VMware Cloud Foundation 5.0.0.1 Release Information

VMware Cloud Foundation 5.0.0.1 includes security updates, fixes for UI and Lifecycle Management issues, and enhancements to prechecks at the bundle level for vCenter.

You can perform a sequential or skip-level upgrade to VMware Cloud Foundation 5.0.0.1 from VMware Cloud Foundation VCF 4.3.x, VCF 4.4.x, VCF 4.5, VCF 4.5.1 and 5.0. If your environment is at a version earlier than 4.3.x, you must upgrade the management domain and all VI workload domains to VMware Cloud Foundation VCF 4.3.x, VCF 4.4.x, VCF 4.5 and VCF 4.5.1 and then upgrade to VMware Cloud Foundation 5.0.0.1. For more information see VMware Cloud Foundation Lifecycle Management.

VMware Cloud Foundation 5.0.0.1 contains the following BOM updates:

Software Component	Version	Date	Build Number
SDDC Manager	5.0.0.1	29 SEPT 2023	22485660

This release addresses the following issues in SDDC Manager:

UI fixes:
- Missing license information error message no longer occurs when:
  1. A domain creation operation is triggered from a workload domain screen through public API (outside of the SDDC UI)
  2. An Add Cluster workflow is triggered through the UI
- Banners now have a new "Dismiss" option.
- On a setup with management domain and VI workload domain, in SUBSCRIPTION/ACTIVE (OR UNSUBSCRIBED) state, this message was displayed to the user in error: You have reached the limit of 4 cloud-connected workload domains. To learn how to add more workload domains, go to the VMware knowledge base.
Lifecycle Management fixes:
- ESXi upgrade failed with error ESX UPGRADE VUM STAGE REGISTER UPLOADED FILES
- When upgrading VCF 4 to 5. upgrade, there was an issue with the license key input spec, which caused an issue with a vCenter Upgrade.
- LCM debug log (lcm-debug.log) spammed with error message does not have a valid product api version string. This excessive logging issue is resolved.
Deployment:
- Client consistently failed to commission host despite meeting all requirements
Precheck enhancements and fixes:
- NSXT BACKUP precheck failed due to lack of disk space and remediation message does not provide appropriate information.

Resolved Issues

The following issues are resolved in this release:

Bring-up Network Configuration Validation fails with "Gateway IP Address for Management is not contactable".
Name resolution fails when configuring the NTP server.

VMware Cloud Foundation Known Issues

Lifecycle Management Precheck does not throw an error when NSX Manager inventory is out of sync

The Lifecycle Management Precheck displays a green status and does not generate any errors for NSX Manager inventory.

Workaround: None
Missing license information error in UI
A Missing domain licensing information error is shown after:
1. A domain creation operation is triggered from a workload domain screen through public API (outside of the SDDC UI)
2. then an Add Cluster workflow is triggered through the UI
Workaround: Refresh the page and then trigger the "Add Cluster" workflow.
The upgrade screen displays the incorrect version when upgrading from VCF 4.3 to version 5.0.

The VCF UI shows the incorrect source version when upgrading from version 4.3 to 5.0. However, the actual upgrade is still correctly performing a direct skip upgrade from the source version to target version.

Workaround: None
Creating workload domains in parallel does not enforce maximum limit of compute managers per NSX Manager.

Creating workload domains in parallel can lead to one or more workload domain creations to fail since the compute managers limit per NSX Manager will be exceeded.

Workaround: Delete the failed workload domain
In VCF 5.0 Mixed BOM with vCenter Enhanced Linked Mode, you can see vCenter Server systems of version 8.0 from a 7.x vCenter Server instance

If you have a vCenter Enhanced Linked Mode group that contains vCenter Server instances of versions 8.x and 7.x, when you log in to a 7.x vCenter Server instance, in the vSphere Client, you can see vCenter Server systems of version 8.0. Since vCenter Server 8.0 introduces new functionalities, you can run workflows specific to vSphere 8.0 on the 8.0 vCenter Server, but this might lead to unexpected results when run from the 7.x vSphere Client.

Workaround: This issue is fixed in vSphere 7.0 P07.
Upgrade Pre-Check Scope dropdown may contain additional entries

When performing Upgrade Prechecks through SDDC Manager UI and selecting a target VCF version, the Pre-Check Scope dropdown may contain more selectable entries than necessary. SDDC Manager may appear as an entry more than once. It also may be included as a selectable component for VI domains, although it's a component of the management domain.

Workaround: None. The issue is visual with no functional impact.
Converting clusters from vSphere Lifecycle Manager baselines to vSphere Lifecycle Manager images is not supported.

vSphere Lifecycle Manager baselines (previously known as vSphere Update Manager or VUM) are deprecated in vSphere 8.0, but continue to be supported. See KB article 89519 for more information.

VMware Cloud Foundation 5.0 does not support converting clusters from vSphere Lifecycle Manager baselines to vSphere Lifecycle Manager images. This capability will be supported in a future release.

Workaround: None
Workload Management does not support NSX Data Center Federation

You cannot deploy Workload Management (vSphere with Tanzu) to a workload domain when that workload domain's NSX Data Center instance is participating in an NSX Data Center Federation.

Workaround: None.
NSX Guest Introspection (GI) and NSX Network Introspection (NI) are not supported on stretched clusters

There is no support for stretching clusters where NSX Guest Introspection (GI) or NSX Network Introspection (NI) are enabled. VMware Cloud Foundation detaches Transport Node Profiles from AZ2 hosts to allow AZ-specific network configurations. NSX GI and NSX NI require that the same Transport Node Profile be attached to all hosts in the cluster.

None.
Stretched clusters and Workload Management

You cannot stretch a cluster on which Workload Management is deployed.

None.

Upgrade Known Issues

New - The "Schedule Update/Retry Upgrade" button is not disabled when an existing upgrade is in progress.

When an upgrade is in progress for a cluster part of a domain, the UI still allows the user to select and start the upgrade for another cluster, but on the FINISH button, this action is blocked.

Workaround: Do not trigger another upgrade when an upgrade is in progress on a domain.
New - Deployment lock popup in updates page is blank during domain operations

If any domain operations are performed during an upgrade, such as add a host or add a cluster, upgrade related operations are blocked, and the tooltip does not show the text with the reason.

Workaround: None.
New - The FINISH button in the Upgrade UI allows for multiple upgrades to be initiated.

When the FINISH button is clicked more than once, multiple upgrades are initiated against the same bundle.

No workaround. You can see all of the triggered upgrades in the Task panel. All upgrades initiated after the first click of the Finish button will fail.
VCF 4.5.1 with the async patch for NSX 3.2.3 are unable to upgrade to VCF 5.0

NSX 3.2.3 to 4.1.0.2 upgrade is not supported. It supports only NSX 4.1.1 onwards.

Workaround: None.

This issue will be resolved in a future release.
VC upgrade fails during install

When running the workload VC-upgrade from VCF, it fails at the VC install with target vc upgrade precheck stage failing.

Workaround: Do not reuse the same temporary IP for sequential VC upgrades. Use a separate temporary IP for every VC undergoing upgrade.
NSX upgrade may fail if there are any active alarms in NSX Manager

If there are any active alarms in NSX Manager, the NSX upgrade may fail.

Workaround: Check the NSX Manager UI for active alarms prior to NSX upgrade and resolve them, if any. If the alarms are not resolved, the NSX upgrade will fail. The upgrade can be retried once the alarms are resolved.
Precheck for NSX has failed with ERROR message

If the Upgrade Prerequisite Backup SDDC Manager, all vCenter Servers, and NSX Managers is ignored and the STFPlocation defined for the NSX Configuration Backup does not have enough disk space, then the ERROR Message Precheck for NSX has failed will display. Although the root cause of the error sftp server disk is full appears in the Operations Manager Log, it is not currently displayed in SDDC Manager.

Workaround: Increase the amount of available disk space on the SFTP server and then complete the Upgrade Prerequisite before proceeding.
Lifecycle Management Precheck does not throw an error when NSX Manager inventory is out of sync

Workaround None.
The SDDC Manager bundle details should show 4.4.1.1 as the "Required Version" for the skip level upgrade

During a skip-level upgrade from VMware Cloud Foundation 4.4.1.1 to 5.0, there is an error in the SDDC Manager bundle details that incorrectly lists VMware Cloud Foundation 4.5.0 as the "Required Version."

Workaround: None

VCF ESXi upgrade fails during post validation due to HA related cluster configuration issue

The upgrade of ESXi Cluster fails with error that is similar to below error message:

Cluster Configuration Issue: vSphere HA failover operation in progress in cluster <cluster-name> in datacenter <datacenter-name>: 0 VMs being restarted, 1 VMs waiting for a retry, 0 VMs waiting for resources, 0 inaccessible vSAN VMs

Workaround: See KB article 90985.

SDDC Manager upgrade fails at "Setup Common Appliance Platform"

If a virtual machine reconfiguration task (for example, removing a snapshot or running a backup) is taking place in the management domain at the same time you are upgrading SDDC Manager, the upgrade may fail.

Workaround: Schedule SDDC Manager upgrades for a time when no virtual machine reconfiguration tasks are happening in the management domain. If you encounter this issue, wait for the other tasks to complete and then retry the upgrade.
Parallel upgrades of vCenter Server are not supported

If you attempt to upgrade vCenter Server for multiple VI workload domains at the same time, the upgrade may fail while changing the permissions for the vpostgres configuration directory in the appliance. The message chown -R vpostgres:vpgmongrp /storage/archive/vpostgres appears in the PatchRunner.log file on the vCenter Server Appliance.

Workaround: Each vCenter Server instance must be upgraded separately.
When you upgrade VMware Cloud Foundation, one of the vSphere Cluster Services (vCLS) agent VMs gets placed on local storage

vSphere Cluster Services (vCLS) ensures that cluster services remain available, even when the vCenter Server is unavailable. vCLS deploys three vCLS agent virtual machines to maintain cluster services health. When you upgrade VMware Cloud Foundation, one of the vCLS VMs may get placed on local storage instead of shared storage. This could cause issues if you delete the ESXi host on which the VM is stored.
Workaround: Deactivate and reactivate vCLS on the cluster to deploy all the vCLS agent VMs to shared storage.
1. Check the placement of the vCLS agent VMs for each cluster in your environment.
  1. In the vSphere Client, select Menu > VMs and Templates.
  2. Expand the vCLS folder.
  3. Select the first vCLS agent VM and click the Summary tab.
  4. In the Related Objects section, check the datastore listed for Storage. It should be the vSAN datastore. If a vCLS agent VM is on local storage, you need to deactivate vCLS for the cluster and then re-enable it.
  5. Repeat these steps for all vCLS agent VMs.
2. Deactivate vCLS for clusters that have vCLS agent VMs on local storage.
  1. In the vSphere Client, click Menu > Hosts and Clusters.
  2. Select a cluster that has a vCLS agent VM on local storage.
  3. In the web browser address bar, note the moref id for the cluster.
    
    For example, if the URL displays as https://vcenter-1.vrack.vsphere.local/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-c8:503a0d38-442a-446f-b283-d3611bf035fb/summary, then the moref id is domain-c8.
  4. Select the vCenter Server containing the cluster.
  5. Click Configure > Advanced Settings.
  6. Click Edit Settings.
  7. Change the value for config.vcls.clusters.<moref id>.enabled to false and click Save.
    
    If the config.vcls.clusters.<moref id>.enabled setting does not appear for your moref id, then enter its Name and false for the Value and click Add.
  8. Wait a couple of minutes for the vCLS agent VMs to be powered off and deleted. You can monitor progress in the Recent Tasks pane.
3. Enable vCLS for the cluster to place the vCLS agent VMs on shared storage.
  1. Select the vCenter Server containing the cluster and click Configure > Advanced Settings.
  2. Click Edit Settings.
  3. Change the value for config.vcls.clusters.<moref id>.enabled to true and click Save.
  4. Wait a couple of minutes for the vCLS agent VMs to be deployed and powered on. You can monitor progress in the Recent Tasks pane.
4. Check the placement of the vCLS agent VMs to make sure they are all on shared storage
SDDC Manager UI issues when running multiple parallel upgrade prechecks

If you initiate a precheck on more than one workload domain at the same time, the SDDC Manager UI may flicker and show incorrect information.

Workaround: Do not run multiple parallel prechecks. If you do, wait until the prechecks are complete to evaluate the results.
Upgrade precheck results for ESXi display the error "TPM 2.0 device detected but a connection cannot be established."

This issue can occur for ESXi hosts that have Trusted Platform Modules (TPM) chips partially configured.

Workaround: Ensure that the TPM is configured in the ESXi host's BIOS to use the SHA-256 hashing algorithm and the TIS/FIFO (First-In, First-Out) interface and not CRB (Command Response Buffer). For information about setting these required BIOS options, refer to the vendor documentation.
Performing parallel prechecks on multiple workload domains that use vSphere Lifecycle Manager images may fail

If you perform parallel prechecks on multiple workload domains that use vSphere Lifecycle Manager images at the same time as you are peforming parallel upgrades, the prechecks may fail.
Workaround: Use the following guidance to plan your upgrades and prechecks for workload domains that use vSphere Lifecycle Manager images.
- For parallel upgrades, VMware Cloud Foundation supports up to five workload domains with up to five clusters each.
- For parallel prechecks, VMware Cloud Foundation supports up to three workload domains with up to four clusters each.
- Do not run parallel upgrades and prechecks at the same time.
Using the /v1/upgrades API to trigger parallel cluster upgrades across workload domains in a single API call does not upgrade the clusters in parallel

When using the VMware Cloud Foundation API to upgrade multiple workload domains in parallel, including multiple resource upgrade specifications (resourceUpgradeSpec) in a single domain upgrade API (/v1/upgrades) call does not work as expected.

Workaround: To get the best performance when upgrading multiple workload domains in parallel using the VMware Cloud Foundation API, do not include multiple resource upgrade specifications (resourceUpgradeSpec) in a single domain upgrade call. Instead, invoke the domain upgrade multiple times with a single resourceUpgradeSpec for each workload domain.

You can also use the SDDC Manager UI to trigger multiple parallel upgrades across workload domains.
NSX Data Center upgrade fails at "NSX T PERFORM BACKUP"

If you did not change the destination of NSX Manager backups to an external SFTP server, upgrades may fail due to an out-of-date SSH fingerprint for SDDC Manager.
Workaround:
1. Log in to the NSX Manager UI.
2. Click System > Backup & Restore.
3. Click Edit for the SFTP Server.
4. Remove the existing SSH fingerprint and click Save.
5. Click Add to add the server provided fingerprint.
6. Click Save.
7. Retry the NSX Data Center upgrade from the SDDC Manager UI.
Cluster-level ESXi upgrade fails

Cluster-level selection during upgrade does not consider the health status of the clusters and may show a cluster's status as Available, even for a faulty cluster. If you select a faulty cluster, the upgrade fails.

Always perform an update precheck to validate the health status of the clusters. Resolve any issues before upgrading.
You are unable to update NSX Data Center in the management domain or in a workload domain with vSAN principal storage because of an error during the NSX transport node precheck stage

In SDDC Manager, when you run the upgrade precheck before updating NSX Data Center, the NSX transport node validation results with the following error.

No coredump target has been configured. Host core dumps cannot be saved.:System logs on host sfo01-m01-esx04.sfo.rainpole.io are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition.

Because the upgrade precheck results with an error, you cannot proceed with updating the NSX Data Center instance in the domain. VMware Validated Design supports vSAN as the principal storage in the management domain. However, vSAN datastores do no support scratch partitions. See VMware Knowledge Base article 2074026.
Disable the update precheck validation for the subsequent NSX Data Center update.
1. Log in to SDDC Manager as vcf using a Secure Shell (SSH) client.
2. Open the application-prod.properties file for editing: vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
3. Add the following property and save the file: lcm.nsxt.suppress.prechecks=true
4. Restart the life cycle management service: systemctl restart lcm
5. Log in to the SDDC Manager user interface and proceed with the update of NSX Data Center.
NSX-T upgrade may fail at the step NSX T TRANSPORT NODE POSTCHECK STAGE

NSX-T upgrade may not proceed beyond the NSX T TRANSPORT NODE POSTCHECK STAGE.

Contact VMware support.
ESXi upgrade fails with the error "Incompatible patch or upgrade files. Please verify that the patch file is compatible with the host. Refer LCM and VUM log file."

This error occurs if any of the ESXi hosts that you are upgrading have detached storage devices.

Workaround: Attach all storage devices to the ESXi hosts being upgraded, reboot the hosts, and retry the upgrade.
Update precheck fails with the error "Password has expired"

If the vCenter Single Sign-On password policy specifies a maximum lifetime of zero (never expires), the precheck fails.

Workaround: Set the maximum lifetime password policy to something other than zero and retry the precheck.

Bring-up Known Issues

Entering pNICs in non-lexicographic order in the deployment parameter workbook does not work as expected

When you deploy the management domain by uploading the deployment parameter workbook, the vmnic to uplink mapping for a vSphere distributed switch (vDS) is always done in lexicographic order, regardless of the order that you enter the vmnics in the deployment parameter workbook.

For example, if you enter vmnic0, vmnic2, vmnic3, vmnic1, the vmnic to uplink mapping will be:

vmnic0	Uplink 1
vmnic1	Uplink 2
vmnic2	Uplink 3
vmnic3	Uplink 4

Workaround: If you want to deploy the management domain with non-lexicographic vmnic to uplink mapping, create and upload a JSON file instead of using the deployment parameter workbook XLSX. See https://developer.vmware.com/apis/vcf/5.1.0/sddc/.

Entering more than two pNICs for the primary vDS in the deployment parameter workbook does not work as expected

If you deploy the management domain by uploading the deployment parameter workbook, there is an issue with converting the XLSX file to JSON when you enter more than two pNICs for the primary vSphere distributed switch (vDS). The result is that only the host overlay network uses all of the pNICs. The management, vSAN, and vMotion networks use only the first two pNICs.

Workaround: If you want to deploy the management domain with a primary vDS that uses more than two pNICs, create and upload a JSON file instead of using the deployment parameter workbook XLSX. See https://developer.vmware.com/apis/vcf/5.0.0/sddc/.
Bring-up Network Configuration Validation fails with "Gateway IP Address for Management is not contactable"

The following failure "Gateway IP Address for MANAGEMENT is not contactable" is reported as fatal error in Cloud Buider UI and bring-up cannot continue. In some cases the validation fails to validate connectivity because it uses as set of predefined ports, however, ping is working.

Workaround: See KB 89990 for more information.

SDDC Manager Known Issues

SDDC Manager incorrectly renders Dark Mode due to inheriting OS settings in VCF 5.0

SDDC Manager UI does not natively support Dark Mode. It attempts to render Dark Mode based on the OS settings and may cause issues with rendering some screens.
Workaround: There are two possible ways to workaround this issue:
- Turn off Dark Mode at the OS level and clear the browser cache, or
- Execute the following script in the Developer console of the browser you are using (replacing the Domain field with the FQDN of your system):
document.cookie = 'clarity-theme=Light; Max-Age=2147483647; path=/; Domain=sddc-manager.vrack.vsphere.local; SameSite=Lax'
SDDC Manager license key is not needed

A SDDC manager license key is not needed, and any errors seen related to an existing SDDC manager license key can be ignored.

Workaround: None.
vRealize Operations Manager admin account appears as disconnected

SDDC Manager incorrectly shows the vRealize Operations Manager admin account as disconnected due to an expired password. The admin account password used for logging into the vRealize Operations Manager UI never expires, but SDDC Manager is actually checking the virtual appliance (Photon OS) admin account password.
Workaround: To clear the expired password/disconnected alert in SDDC Manager:
1. Log in to the affected vRealize Operations Manager node and update the virtual appliance admin password.
2. In SDDC Manager, remediate (or rotate or update) the password for the expired account. Or, use the VMware Cloud Foundation API to run POST /v1/credentials/expirations.
Generate CSR task for a component hangs

When you generate a CSR, the task may fail to complete due to issues with the component's resources. For example, when you generate a CSR for NSX Manager, the task may fail to complete due to issues with an NSX Manager node. You cannot retry the task once the resource is up and running again.
1. Log in to the UI for the component to troubleshoot and resolve any issues.
2. Using SSH, log in to the SDDC Manager VM with the user name vcf.
3. Type su to switch to the root account.
4. Run the following command: systemctl restart operationsmanager
5. Retry generating the CSR.
SoS utility options for health check are missing information
Due to limitations of the ESXi service account, some information is unavailable in the following health check options:
- --hardware-compatibility-report: No Devices and Driver information for ESXi hosts.
- --storage-health: No vSAN Health Status or Total no. of disks information for ESXi hosts.
None.

Workload Domain Known Issues

SDDC Manager UI incorrectly shows "Owner" of Isolated Workload Doamin as the management domain SSO admin user.

When creating an isolated workload domain, the "Workload Domains" view in the UI incorrectly shows the "Owner" as the management domain SSO admin user.

Workaround: The domain info can be seen under "SSO Domain" as a column. To include the SSO Domain as a column, select it in the table configuration to show it as a column. The Owner column on the UI is not applicable for Isolated Workload Domains.
An unxpected replication agreement is still left after the successful decommission of the SSO node task

The decommission of the SSO node was successful, but the replication partner remained.

Workaround: Manually remove the replication agreement of the invalid SSO node, and restart the delete workload domain.

The vCenter command /usr/lib/vmware-vmdir/bin/vdcrepadmin can be used to add/remove replication agreements.
Heterogeneous operations "Cluster Creation" and "VI Creation" are not supported to be ran in parallel when they are operating against same shared NSX instance.

If there is a running VI Creation workflow operating on an NSX resource, then creating a cluster on domains that are sharing that NSX is not possible.

Workaround: None. The VI Creation workflow should complete before the cluster creation workflow can be started.
SDDC Manager UI allows you to select an unsupported NSX Manager instance

When you create a new VI workload domain, it cannot share an NSX Manager instance with a VI workload domain that is in different SSO domain. Although the SDDC Manager UI allows you to select an unsupported NSX Manager instance, the VI workload domain creation task will fail.

Workaround: None
Adding host fails when host is on a different VLAN

A host add operation can sometimes fail if the host is on a different VLAN.
1. Before adding the host, add a new portgroup to the VDS for that cluster.
2. Tag the new portgroup with the VLAN ID of the host to be added.
3. Add the Host. This workflow fails at the "Migrate host vmknics to dvs" operation.
4. Locate the failed host in vCenter, and migrate the vmk0 of the host to the new portgroup you created in step 1. For more information, see Migrate VMkernel Adapters to a vSphere Distributed Switch in the vSphere product documentation.
5. Retry the Add Host operation.
NOTE: If you later remove this host in the future, you must manually remove the portgroup as well if it is not being used by any other host.
Deploying partner services on an NSX workload domain displays an error

Deploying partner services, such as McAfee or Trend, on a workload domain enabled for vSphere Update Manager (VUM), displays the “Configure NSX at cluster level to deploy Service VM” error.

Attach the Transport node profile to the cluster and try deploying the partner service. After the service is deployed, detach the transport node profile from the cluster.
If the witness ESXi version does not match with the host ESXi version in the cluster, vSAN cluster partition may occur

vSAN stretch cluster workflow does not check the ESXi version of the witness host. If the witness ESXi version does not match the host version in the cluster, then vSAN cluster partition may happen.
1. Upgrade the witness host manually with the matching ESXi version using the vCenter VUM functionality.
2. Replace or deploy the witness appliance matching with the ESXi version.
vSAN partition and critical alerts are generated when the witness MTU is not set to 9000

If the MTU of the witness switch in the witness appliance is not set to 9000, the vSAN stretch cluster partition may occur.

Set the MTU of the witness switch in the witness appliance to 9000 MTU.
Adding a host to a vLCM-enabled workload domain configured with the Dell Hardware Support Manager (OMIVV) fails

When you try to add a host to a vSphere cluster for a workload domain enabled with vSphere Lifecycle Manager (vLCM), the task fails and the domain manager log reports "The host (host-name) is currently not managed by OMIVV." The domain manager logs are located at /var/log/vmware/vcf/domainmanager on the SDDC Manager VM.

Update the hosts inventory in OMIVV and retry the add host task in the SDDC Manager UI. See the Dell documentation for information about updating the hosts inventory in OMIVV.
Adding a vSphere cluster or adding a host to a workload domain fails

Under certain circumstances, adding a host or vSphere cluster to a workload domain fails at the Configure NSX Transport Node or Create Transport Node Collection subtask.
1. Enable SSH for the NSX Manager VMs.
2. SSH into the NSX Manager VMs as admin and then log in as root.
3. Run the following command on each NSX Manager VM: sysctl -w net.ipv4.tcp_en=0
4. Login to NSX Manager UI for the workload domain.
5. Navigate to System > Fabric > Nodes > Host Transport Nodes.
6. Select the vCenter server for the workload domain from the Managed by drop-down menu.
7. Expand the vSphere cluster and navigate to the transport nodes that are in a partial success state.
8. Select the check box next to a partial success node, click Configure NSX.
9. Click Next and then click Apply.
10. Repeat steps 7-9 for each partial success node.
When all host issues are resolved, transport node creation starts for the failed nodes. When all hosts are successfully created as transport nodes, retry the failed add vSphere cluster or add host task from the SDDC Manager UI.
The vSAN Performance Service is not enabled for vSAN clusters when CEIP is not enabled

If you do not enable the VMware Customer Experience Improvement Program (CEIP) in SDDC Manager, when you create a workload domain or add a vSphere cluster to a workload domain, the vSAN Performance Service is not enabled for vSAN clusters. When CEIP is enabled, data from the vSAN Performance Service is provided to VMware and this data is used to aid VMware Support with troubleshooting and for products such as VMware Skyline, a proactive cloud monitoring service. See Customer Experience Improvement Program for more information on the data collected by CEIP.

Enable CEIP in SDDC Manager. See the VMware Cloud Foundation Documentation. After CEIP is enabled, a scheduled task that enables the vSAN Performance Service on existing clusters in workload domains runs every three hours. The service is also enabled for new workload domains and clusters. To enable the vSAN Performance Service immediately, see the VMware vSphere Documentation.
Creation or expansion of a vSAN cluster with more than 32 hosts fails

By default, a vSAN cluster can grow up to 32 hosts. With large cluster support enabled, a vSAN cluster can grow up to a maximum of 64 hosts. However, even with large cluster support enabled, a creation or expansion task can fail on the sub-task Enable vSAN on vSphere Cluster.
1. Enable Large Cluster Support for the vSAN cluster in the vSphere Client. If it is already enabled skip to step 2.
  1. Select the vSAN cluster in the vSphere Client.
  2. Select Configure > vSAN > Advanced Options.
  3. Enable Large Cluster Support.
  4. Click Apply.
  5. Click Yes.
2. Run a vSAN health check to see which hosts require rebooting.
3. Put the hosts into Maintenance Mode and reboot the hosts.
For more information about large cluster support, see https://kb.vmware.com/kb/2110081.
Removing a host from a cluster, deleting a cluster from a workload domain, or deleting a workload domain fails if Service VMs (SVMs) are present

If you deployed an endpoint protection service (such as guest introspection) to a cluster through NSX Data Center, then removing a host from the cluster, deleting the cluster, or deleting the workload domain containing the cluster will fail on the subtask Enter Maintenance Mode on ESXi Hosts.
- For host removal: Delete the Service VM from the host and retry the operation.
- For cluster deletion: Delete the service deployment for the cluster and retry the operation.
- For workload domain deletion: Delete the service deployment for all clusters in the workload domain and retry the operation.
vCenter Server overwrites the NFS datastore name when adding a cluster to a VI workload domain

If you add an NFS datastore with the same NFS server IP address, but a different NFS datastore name, as an NFS datastore that already exists in the workload domain, then vCenter Server applies the existing datastore name to the new datastore.

If you want to add an NFS datastore with a different datastore name, then it must use a different NFS server IP address.

API Known Issues

LCM debug log (lcm-debug.log) spammed with error message "does not have a valid product api version string"

This issue is caused by the release version format issue of the vRealize interoperability data. As a result, the logs will reset frequently.

Workaround: None.

This issue will be resolved in a future release.
An API for NSX Clusters is listed on VMware Cloud Foundation Developer Center in error

Public API "2.36.6. Get the transport zones from the NSX cluster" is listed on VMware Cloud Foundation Developer Center in error.

Workaround: None
Stretch cluster operation fails

If the cluster that you are stretching does not include a powered-on VM with an operating system installed, the operation fails at the "Validate Cluster for Zero VMs" task.

Make sure the cluster has a powered-on VM with an operating system installed before stretching the cluster.