VMware Cloud Foundation 4.3.1 Release Notes

VMware Cloud Foundation 4.3.1 \| 21 SEP 2021 \| Build 18624509 VMware Cloud Foundation 4.3.1.1 \| 31 JAN 2022 \| Build 19235535 Check for additions and updates to these release notes.

VMware Cloud Foundation 4.3.1 | 21 SEP 2021 | Build 18624509

VMware Cloud Foundation 4.3.1.1 | 31 JAN 2022 | Build 19235535

Check for additions and updates to these release notes.

What's New

The VMware Cloud Foundation (VCF) 4.3.1 release includes the following:

Limited Support for in-place migration from VMware Cloud Foundation 3.10.1.2+ releases: In addition to existing migration methods, customers can now engage the VMware Professional Service Organization (PSO) to perform an assessment for a potential in-place migration from VMware Cloud Foundation 3.10.1.2+ releases to VMware Cloud Foundation 4.3.1. Contact your sales and channel teams for guidance on choosing the best method for migrating your environment
Supportability and Serviceability (SoS) Utility APIs: You can invoke VMware Cloud Foundation public APIs to run SoS Utility health checks and collect logs.
BOM updates: Updated Bill of Materials with new product versions.

VMware Cloud Foundation Bill of Materials (BOM)

The Cloud Foundation software product is comprised of the following software Bill-of-Materials (BOM). The components in the BOM are interoperable and compatible.

VMware Response to Apache Log4j Remote Code Execution Vulnerability: VMware Cloud Foundation is impacted by CVE-2021-44228, and CVE-2021-45046 as described in VMSA-2021-0028. To remediate these issues, see Workaround instructions to address CVE-2021-44228 & CVE-2021-45046 in VMware Cloud Foundation (KB 87095).

Software Component	Version	Date	Build Number
Cloud Builder VM	4.3.1	21 SEP 2021	18624509
SDDC Manager	4.3.1	21 SEP 2021	18624509
VMware vCenter Server Appliance	7.0 Update 2d	21 SEP 2021	18455184
VMware ESXi	7.0 Update 2c	24 AUG 2021	18426014
VMware Virtual SAN Witness Appliance	7.0 Update 2c	24 AUG 2021	18426014
VMware NSX-T Data Center	3.1.3.1	26 AUG 2021	18504668
VMware vRealize Suite Lifecycle Manager	8.4.1 Patch 2	6 SEP 2021	18537943
Workspace ONE Access	3.3.5	20 MAY 2021	18049997
vRealize Automation	8.5	19 AUG 2021	18472703
vRealize Log Insight	8.4.1	15 JUN 2021	18136317
vRealize Log Insight Content Pack for NSX-T	4.0.2	n/a	n/a
vRealize Log Insight Content Pack for vRealize Automation 8.3+	1.0	n/a	n/a
vRealize Log Insight Content Pack for Linux	2.1.0	n/a	n/a
vRealize Log Insight Content Pack for Linux - Systemd	1.0.0	n/a	n/a
vRealize Log Insight Content Pack for vRealize Suite Lifecycle Manager 8.0.1+	1.0.2	n/a	n/a
vRealize Log Insight Content Pack for VMware Identity Manager	2.0	n/a	n/a
vRealize Operations Manager	8.5	13 JUL 2021	18255622
vRealize Operations Management Pack for VMware Identity Manager	1.3	n/a	n/a

VMware vSAN is included in the VMware ESXi bundle.
You can use vRealize Suite Lifecycle Manager to deploy vRealize Automation, vRealize Operations Manager, vRealize Log Insight, and Workspace ONE Access.
vRealize Log Insight content packs are installed when you deploy vRealize Log Insight.
The vRealize Operations Manager management pack is installed when you deploy vRealize Operations Manager.
VMware Solution Exchange and the vRealize Log Insight in-product marketplace store only the latest versions of the content packs for vRealize Log Insight. The Bill of Materials table contains the latest versions of the packs that were available at the time VMware Cloud Foundation is released. When you deploy the Cloud Foundation components, it is possible that the version of a content pack within the in-product marketplace for vRealize Log Insight is newer than the one used for this release.

VMware Software Edition License Information

The SDDC Manager software is licensed under the Cloud Foundation license. As part of this product, the SDDC Manager software deploys specific VMware software products.

The following VMware software components deployed by SDDC Manager are licensed under the Cloud Foundation license:

VMware ESXi
VMware vSAN
VMware NSX-T Data Center

The following VMware software components deployed by SDDC Manager are licensed separately:

vCenter Server

NOTE: Only one vCenter Server license is required for all vCenter Servers deployed in a Cloud Foundation system.

For details about the specific VMware software editions that are licensed under the licenses you have purchased, see the Cloud Foundation Bill of Materials (BOM) section above.

For general information about the product, see VMware Cloud Foundation.

Supported Hardware

For details on supported configurations, see the VMware Compatibility Guide (VCG) and the Hardware Requirements section on the Prerequisite Checklist tab in the Planning and Preparation Workbook.

Documentation

To access the Cloud Foundation documentation, go to the VMware Cloud Foundation product documentation.

To access the documentation for VMware software products that SDDC Manager can deploy, see the product documentation and use the drop-down menus on the page to choose the appropriate version:

VMware vSphere product documentation, also has documentation about ESXi and vCenter Server
VMware vSAN product documentation
VMware NSX-T Data Center product documentation

Browser Compatibility and Screen Resolutions

The Cloud Foundation web-based interface supports the latest two versions of the following web browsers except Internet Explorer:

Google Chrome
Mozilla Firefox
Microsoft Edge
Internet Explorer: Version 11

For the Web-based user interfaces, the supported standard resolution is 1024 by 768 pixels. For best results, use a screen resolution within these tested resolutions:

1024 by 768 pixels (standard)
1366 by 768 pixels
1280 by 1024 pixels
1680 by 1050 pixels

Resolutions below 1024 by 768, such as 640 by 960 or 480 by 800, are not supported.

Installation and Upgrade Information

You can install VMware Cloud Foundation 4.3 as a new release or perform a sequential or skip-level upgrade to VMware Cloud Foundation 4.3.

Installing as a New Release

The new installation process has three phases:

Phase One: Prepare the Environment

The Planning and Preparation Workbook provides detailed information about the software, tools, and external services that are required to implement a Software-Defined Data Center (SDDC) with VMware Cloud Foundation, using a standard architecture model.

Phase Two: Image all servers with ESXi

Image all servers with the ESXi version mentioned in the Cloud Foundation Bill of Materials (BOM) section. See the VMware Cloud Foundation Deployment Guide for information on installing ESXi.

Phase Three: Install Cloud Foundation 4.3

See the VMware Cloud Foundation Deployment Guide for information on deploying Cloud Foundation. Note that there is no ISO published for ESXi 7.0 Update 2c that is included in the VMware Cloud Foundation 4.3.1 BOM. For information on how to proceed, seeCreate a Custom ISO Image for ESXi.

Upgrading to Cloud Foundation 4.3.1

You can perform a sequential or skip-level upgrade to VMware Cloud Foundation 4.3.1 from VMware Cloud Foundation 4.3, 4.2.1, 4.2, 4.1.0.1, or 4.1. If your environment is at a version earlier than 4.1, you must upgrade the management domain and all VI workload domains to VMware Cloud Foundation 4.1 and then upgrade to VMware Cloud Foundation 4.3.1. For more information, see VMware Cloud Foundation Lifecycle Management.

IMPORTANT: Before you upgrade a vCenter Server, take a file-based backup. See Manually Back Up vCenter Server.

VMware Cloud Foundation 4.3.1.1 Release Information

VMware Cloud Foundation 4.3.1.1 includes security fixes. You can perform a sequential or skip-level upgrade to VMware Cloud Foundation 4.3.1.1 from Cloud Foundation 4.3.1, 4.3, 4.2.1, 4.2, 4.1.0.1, or 4.1. If your environment is at a version earlier than 4.1, you must upgrade the management domain and all VI workload domains.

To upgrade the management domain, apply the following bundles, in order:

NOTE: Before triggering an upgrade to VCF 4.3.1.1, download all component upgrade/install bundles from VCF 4.3.1.0.

VMware Cloud Foundation bundle.
Configuration drift bundle.

NOTE: When you are upgrading from VMware Cloud Foundation from 4.3.1 to 4.3.1.1, no configuration drift bundle is required.

VMware Cloud Foundation 4.3.1.1 contains the following BOM updates:

Software Component	Version	Date	Build Number
SDDC Manager	4.3.1.1	31 JAN 2022	19235535

This release addresses the following issues in SDDC Manager:

Apache Log4j Remote Code Execution Vulnerability: (CVE-2021-44228 and CVE-2021-45046) as described in VMSA-2021-0028.
XML External Entity (XXE) Injection vulnerability as described in CVE-2021-23463
Credential logging vulnerability as described in VMSA-2022-0003. See KB 87050 for more information.

Resolved Issues

The following issues are resolved in this release.

vRealize Operations Manager: VMware Security Advisory VMSA-2021-0018
Generate CSR task for a component hangs
Supportability and Serviceability (SoS) Utility health checks fail with the error "Failed to get details"
Update precheck fails with the error "Password has expired"
vCenter Server: VMware Security Advisory VMSA-2021-0020 resolves resolves CVE-2021-22011 and CVE-2021-22018

VMware Cloud Foundation Known Issues

Stretched clusters and Workload Management

You cannot stretch a cluster on which Workload Management is deployed.

Workaround: None.
NSX-T Guest Introspection (GI) and NSX-T Service Insertion (SI) are not supported on stretched clusters

There is no support for stretching clusters where NSX-T Guest Introspection (GI) or NSX-T Service Insertion (SI) are enabled. VMware Cloud Foundation detaches Transport Node Profiles from AZ2 hosts to allow AZ-specific network configurations. NSX-T GI and NSX-T SI require that the same Transport Node Profile be attached to all hosts in the cluster.

Workaround: None

Upgrade Known Issues

New - VxRail Async patch 7.0.410 bundle visible in the Lifecycle Manager (LCM) bundle management UI page and availability status as "future"

If you connect to the VMware Depot, the VxRail async patch bundle 7.0.410 might be visible in the Lifecycle Manager(LCM) UI. This is a known issue caused by the async patch bundle information being added to the existing partner bundle metadata (PBM) file. However, this issue has been resolved in the latest PBM, which has already been published. If you are still seeing this bundle in the LCM UI I and have not used the Async Patch Tool to enable this patch, you can follow the workaround to remove the patch. After completing the workaround, the bundle will no longer be displayed in the LCM UI.

Workaround: Perform the bundle cleanup of VxRail async patch 7.0.410 by following the steps in KB 75050.
Network outage on NSX-T Data Center after upgrading VDS to 7.0.2

After upgrading NSX-T Data Center, if you upgrade the vSphere Distributed Switch (VDS) to version 7.0.2, you may experience network traffic disruption.

Workaround: Do not upgrade vSphere Distributed Switches to version 7.0.2.
Async Patch Tool Known Issues

The Async Patch Tool is a utility that allows you to apply critical patches to certain VMware Cloud Foundation components (NSX-T Manager, vCenter Server, and ESXi) outside of VMware Cloud Foundation releases. The Async Patch Tool also allows you to enable upgrade of an async patched system to a new version of VMware Cloud Foundation.

See the Async Patch Tool Release Notes for known issues.
NSX-T upgrade causing host PSOD

ESXi host can PSOD during NSX-T upgrade when there is a mass migration of DFW filters, where flows are being revalidated while configuration cycle is occurring.

See KB 87803 for more information. This issue is fixed in NSX-T 3.1.3.7.

If a VCF upgrade is tried post application of this workaround, the LCM pre-check on DRS configuration will fail. This is expected behavior.
Cluster-level ESXi upgrade fails

Cluster-level selection during upgrade does not consider the health status of the clusters and may show a cluster's status as Available, even for a faulty cluster. If you select a faulty cluster, the upgrade fails.

Workaround: Always perform an update precheck to validate the health status of the clusters. Resolve any issues before upgrading.
You are unable to update NSX-T Data Center in the management domain or in a workload domain with vSAN principal storage because of an error during the NSX-T transport node precheck stage.

In SDDC Manager, when you run the upgrade precheck before updating NSX-T Data Center, the NSX-T transport node validation results with the following error.

No coredump target has been configured. Host core dumps cannot be saved.:System logs on host sfo01-m01-esx04.sfo.rainpole.io are stored on non-persistent storage. Consult product documentation to configure a syslog server or a scratch partition.

Because the upgrade precheck results with an error, you cannot proceed with updating the NSX-T Data Center instance in the domain. VMware Validated Design supports vSAN as the principal storage in the management domain. However, vSAN datastores do no support scratch partitions. See KB article 2074026.
Workaround: Deactivate the update precheck validation for the subsequent NSX-T Data Center update.
1. Log in to SDDC Manager as vcf using a Secure Shell (SSH) client.
2. Open the application-prod.properties file for editing.
  
  vi /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties
3. Add the following property and save the file.
  
  lcm.nsxt.suppress.prechecks=true
4. Restart the life cycle management service.
  
  systemctl restart lcm
5. Log in to the SDDC Manager user interface and proceed with the update of NSX-T Data Center.
NSX-T upgrade may fail at the step NSX T TRANSPORT NODE POSTCHECK STAGE

NSX-T upgrade may not proceed beyond the NSX T TRANSPORT NODE POSTCHECK STAGE

Workaround: Contact VMware support.
ESXi upgrade fails with the error "Incompatible patch or upgrade files. Please verify that the patch file is compatible with the host. Refer LCM and VUM log file."

This error occurs if any of the ESXi hosts that you are upgrading have detached storage devices.

Workaround: Attach all storage devices to the ESXi hosts being upgraded, reboot the hosts, and retry the upgrade.
Skip level upgrades are not enabled for some product components after VMware Cloud Foundation is upgraded to 4.3
After performing skip level upgrade to VMware Cloud Foundation 4.3 from 4.1.x or 4.2.x, one or more of the following symptoms is observed:
- vRealize bundles do not show up as available for upgrade
- Bundles for previous versions of some product components (NSX-T Data Center, vCenter Server, ESXi) show up as available for upgrade
See KB 85505.
Domain prechecks for vRealize Suite products show incorrect health state

Domain pre-checks for vRealize Suite products may be wrongly marked as RED or GREEN. The issue only happens if pre-checks for some vRealize Suite products fail and then the precheck is retried at the resource level instead of the domain level.

If the precheck fails, do not click the Retry Precheck option on the failed resources. Instead, run the precheck again on the entire workload domain. If the precheck is already retried on the failed resource, a restart of lcm service is required in order to clear up the cache.
vRealize Suite upgrade bundles have an incorrect description

vRealize Suite upgrade bundles are incorrectly described as install bundles. You can ignore the incorrect description and proceed with the upgrade.

None.
vRealize Operations Manager upgrade fails on the step VREALIZE_UPGRADE_PREPARE_BACKUP with the error: Waiting for vRealize Operations cluster to change state timed out

When upgrading vRealize Operations Manager, SDDC Manager takes the vRealize Operations Manager cluster offline and takes snapshots of the vRealize Operations Manager virtual machines. In some circumstances, taking the cluster offline takes a long time and the operation times out.
Workaround: Take the vRealize Operations Manager cluster back online and retry the upgrade.
1. Log in to the vRealize Operations Manager Administration UI (https://<vrops_ip>/admin) using the admin credentials.
2. If the cluster status is offline, in the Cluster Status section click Take Cluster Online. Wait for the cluster to initialize and be marked as green.
3. In the SDDC Manager UI, the option to retry vRealize Operations Manager upgrade should be available. Retry the upgrade.
If the upgrade continues to fail, take the snapshots manually and retry the upgrade. Since the snapshots already exist, SDDC Manager will skip that step and proceed with the upgrade.
1. Log in to the vRealize Operations Manager Administration UI (https://<vrops_ip>/admin) using the admin credentials. Ensure that that the vRealize Operations Manager Cluster Status is offline. If it is online, click Take Cluster Offline in the Cluster Status section. Wait for the cluster to be marked as offline.
2. Log in to the management domain vCenter Server using the vSphere Client.
3. Navigate to the vRealize Operations Manager virtual machines and create a snapshot for each virtual machine in the vRealize Operations Manager cluster. Use the following prefix "vROPS_LCM_UPGRADE_MANUAL_BACKUP" for the snapshots. Please note that the prefix should match the letter casing.
4. After the snapshots are done, log in to the vRealize Operations Manager UI and take cluster online. Wait for the cluster initialization.
5. In the SDDC Manager UI, the option to retry vRealize Operations Manager upgrade should be available. Retry the upgrade.

Bring-up Known Issues

Bringup fails when creating NSX-T Data Center transport nodes

The bringup task "Create NSX-T Data Center Transport Nodes from Discovered Nodes" might fail if there's an ESXi host in the management cluster which is pending a reboot.

Workaround: Reboot all ESXi hosts that are pending reboot and retry bringup.
The Cloud Foundation Builder VM remains locked after more than 15 minutes

The VMware Imaging Appliance (VIA) locks out the admin user after three unsuccessful login attempts. Normally, the lockout is reset after fifteen minutes but the underlying Cloud Foundation Builder VM does not automatically reset.

Workaround: Log in to the VM console of the Cloud Foundation Builder VM as the root user. Unlock the account by resetting the password of the admin user with the following command.

pam_tally2 --user=<user> --reset

Workload Domain Known Issues

Cannot reuse a static IP pool that includes special characters in its name

If you chose Static IP Pool as the IP allocation method when creating a VI workload domain and you used special characters or spaces in the IP pool name, you are not able to reuse the IP pool when creating a new VI workload domain or adding a vSphere cluster to the workload domain.
Workaround: Use only supported characters when naming a static IP pool. Supported characters:
- a-z
- A-Z
- 0-9
- - and _
- No spaces
If you have an existing static IP pool that includes unsupported characters in its name, you can use the NSX Manager UI to rename it.
Unable to remove host from vSphere cluster in workload domain

The remove host workflow fails at the Delete Transport Nodes task. The NSX-T Data Center UI shows the deletion process stuck.
1. Remove the host by force from the NSX-T Data Center UI.
2. Restart the failed remove host workflow in SDDC Manager.
vCenter Server overwrites the NFS datastore name when adding a cluster to a VI workload domain

If you add an NFS datastore with the same NFS server IP address, but a different NFS datastore name, as an NFS datastore that already exists in the workload domain, then vCenter Server applies the existing datastore name to the new datastore.

Workaround: If you want to add an NFS datastore with a different datastore name, then it must use a different NFS server IP address.
Removing a host from a cluster, deleting a cluster from a workload domain, or deleting a workload domain fails if Service VMs (SVMs) are present

If you deployed an endpoint protection service (such as guest introspection) to a cluster through NSX-T Data Center, then removing a host from the cluster, deleting the cluster, or deleting the workload domain containing the cluster will fail on the subtask Enter Maintenance Mode on ESXi Hosts.
Workaround:
- For host removal: Delete the Service VM from the host and retry the operation.
- For cluster deletion: Delete the service deployment for the cluster and retry the operation.
- For workload domain deleting: Delete the service deployment for all clusters in the workload domain and retry the operation.
Creation or expansion of a vSAN cluster with more than 32 hosts fails

By default, a vSAN cluster can grow up to 32 hosts. With large cluster support enabled, a vSAN cluster can grow up to a maximum of 64 hosts. However, even with large cluster support enabled, a creation or expansion task can fail on the sub-task Enable vSAN on vSphere Cluster.
Workaround:
1. Enable Large Cluster Support for the vSAN cluster in the vSphere Client. If it is already enabled skip to step 2.
  1. Select the vSAN cluster in the vSphere Client.
  2. Select Configure > vSAN > Advanced Options.
  3. Enable Large Cluster Support.
  4. Click Apply.
  5. Click Yes.
2. Run a vSAN health check to see which hosts require rebooting.
3. Put the hosts into Maintenance Mode and reboot the hosts.
For more information about large cluster support, see KB 2110081.
The vSAN Performance Service is not enabled for vSAN clusters when CEIP is not enabled

If you do not enable the VMware Customer Experience Improvement Program (CEIP) in SDDC Manager, when you create a workload domain or add a vSphere cluster to a workload domain, the vSAN Performance Service is not enabled for vSAN clusters. When CEIP is enabled, data from the vSAN Performance Service is provided to VMware and this data is used to aid VMware Support with troubleshooting and for products such as VMware Skyline, a proactive cloud monitoring service. See Customer Experience Improvement Program for more information on the data collected by CEIP.

Workaround: Enable CEIP in SDDC Manager. See the VMware Cloud Foundation Documentation. After CEIP is enabled, a scheduled task that enables the vSAN Performance Service on existing clusters in workload domains runs every three hours. The service is also enabled for new workload domains and clusters. To enable the vSAN Performance Service immediately, see the VMware vSphere Documentation.
Adding a vSphere cluster or adding a host to a workload domain fails

Under certain circumstances, adding a host or vSphere cluster to a workload domain fails at the Configure NSX-T Transport Node or Create Transport Node Collection subtask.
Workaround:
1. Enable SSH for the NSX Manager VMs.
2. SSH into the NSX Manager VMs as admin and then log in as root.
3. Run the following command on each NSX Manager VM:
  
  sysctl -w net.ipv4.tcp_en=0
4. Login to NSX Manager UI for the workload domain.
5. Navigate to System > Fabric > Nodes > Host Transport Nodes.
6. Select the vCenter server for the workload domain from the Managed by drop-down menu.
7. Expand the vSphere cluster and navigate to the transport nodes that are in a partial success state.
8. Select the check box next to a partial successnode, click Configure NSX.
9. Click Next and then click Apply.
10. Repeat steps 7-9 for each partial success node.
When all host issues are resolved, transport node creation starts for the failed nodes. When all hosts are successfully created as transport nodes, retry the failed add vSphere cluster or add host task from the SDDC Manager UI.
vSAN partition and critical alerts are generated when the witness MTU is not set to 9000

If the MTU of the witness switch in the witness appliance is not set to 9000, the vSAN stretch cluster partition may occur.

Workaround: Set the MTU of the witness switch in the witness appliance to 9000 MTU.
If the witness ESXi version does not match with the host ESXi version in the cluster, vSAN cluster partition may occur

vSAN stretch cluster workflow does not check the ESXi version of the witness host. If the witness ESXi version does not match the host version in the cluster, then vSAN cluster partition may happen.
Workaround:
1. Upgrade the witness host manually with the matching ESXi version using the vCenter VUM functionality.
2. Replace or deploy the witness appliance matching with the ESXi version.
Deploying partner services on a workload domain displays an error

Deploying partner services, such as McAfee or Trend, on a workload domain enabled for vSphere Lifecycle Manager (vLCM) baselines, displays the “Configure NSX at cluster level to deploy Service VM” error.

Workaround: Attach the transport node profile to the cluster and try deploying the partner service. After the service is deployed, keep the transport node profile attached to the cluster. If you want to delete the cluster later, you must first undeploy the partner service and detach the transport node profile from the cluster.
Adding host fails when host is on a different VLAN

A host add operation can sometimes fail if the host is on a different VLAN.
Workaround:
1. Before adding the host, add a new portgroup to the vSphere Distributed Switch for that cluster.
2. Tag the new portgroup with the VLAN ID of the host to be added.
3. Add the Host. This workflow fails at the "Migrate host vmknics to dvs" operation.
4. Locate the failed host in vCenter, and migrate the vmk0 of the host to the new portgroup you created in step 1.
  
  For more information, see Migrate VMkernel Adapters to a vSphere Distributed Switch in the vSphere product documentation.
5. Retry the Add Host operation.
NOTE: If you later remove this host in the future, you must manually remove the portgroup as well if it is not being used by any other host.

SDDC Manager Known Issues

Rotating or updating vSphere Single-Sign On (PSC) password can cause issues

If you have multiple VMware Cloud Foundation instances that share a single SSO domain, rotating or updating the vSphere SSO password for the first VCF instance causes the second VCF instance to become inaccessible.

Workaround: See KB 85485.
SDDC Manager UI does not load correctly

If you log in to the SDDC Manager UI using an Active Directory user name that includes a space, the UI does not load correctly.

Workaround: None
SoS utility options for health check are missing information
Due to limitations of the ESXi service account, some information is unavailable in the following health check options:
- --hardware-compatibility-report: No Devices and Driverinformation for ESXi hosts.
- --storage-health: No vSAN Health Status or Total no. of disks information for ESXi hosts.
Workaround: None.
Deactivating CEIP on SDDC Manager does not deactive CEIP on vRealize Automation and vRealize Suite Lifecycle Manager

When you deactive CEIP on the SDDC Manager Dashboard, data collection is not deactivated on vRealize Automation and vRealize Suite Lifecycle Manager. This is because of API deprecation in vRealize Suite 8.x.

Workaround: Manually deactivate CEIP in vRealize Automation and vRealize Suite Lifecycle Manager. For more information, see VMware vRealize Automation Documentation and VMware vRealize Suite Lifecycle Manager Documentation.

Multi-Instance Management Known Issues

Join operation fails

A join operation may fail if a controller SDDC Manager has a public certificate with a depth greater than one (that is, it has intermediate certificates).

Workaround: Trust the intermediate certificate of the controller SDDC Manager. See KB 80986.
Multi-Instance Management Dashboard operation fails

After a controller joins or leaves a federation, Kafka is restarted on all controllers in the federation. It can take up to 20 minutes for the federation to stabilize. Any operations performed on the dashboard during this time may fail.

Workaround: Re-try the operation.
Federation creation information not displayed if you leave the Multi-Instance Management Dashboard

Federation creation progress is displayed on the Multi-Instance Management Dashboard. If you navigate to another screen and then return to the Multi-Instance Management Dashboard, progress messages are not displayed. Instead, an empty map with no Cloud Foundation instances are displayed until the federation is created.

Workaround: Stay on the Multi-Instance Dashboard till the task is complete. If you have navigated away, wait for around 20 minutes and then return to the dashboard by which time the operation should have completed.

API Known Issues

Updating DNS/NTP server does not apply the update to all NSX Managers

If you update the NTP or DNS server information for a VMware Cloud Foundation instance that includes more than one NSX Manager, only one of the NSX Managers gets updated with the new information.

Workaround: Use the NSX Manager API or CLI to manually update the DNS/NTP server information for the remaining NSX Manager(s).
Unable to download SoS bundles from SDDC Manager API Explorer
In the SDDC Manager UI, you are not able to download the SoS support bundle or SoS health summary bundle using the commands under Developer Center > API Explorer.
- GET "/v1/system/support-bundles/{id}/data
- GET "/v1/system/health-summary/{id}/data"
Use an alternative client such as curl to download the bundle. Example of downloading support bundle using curl:

$ curl --insecure -X GET https://10.0.0.4/v1/system/support-bundles/$ID/data -H "Content-Type: application/octet-stream" -H "Authorization: Bearer $TOKEN" -o sos-.tar % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6961k 100 6961k 0 0 41.0M 0 --:--:-- --:--:-- --:--:-- 41.2M

Example of downloading health summary bundle using curl:

$ curl --insecure -X GET https://10.0.0.4/v1/system/health-summary/$ID/data -H "Content-Type: application/octet-stream" -H "Authorization: Bearer $TOKEN" -o healthcheck-.tar % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 981k 100 981k 0 0 7778k 0 --:--:-- --:--:-- --:--:-- 7789k
The VMware Cloud Foundation API ignores NSX VDS uplink information for in-cluster expansion of an NSX Edge cluster

When you use the VMware Cloud Foundation API to expand an NSX Edge cluster and the new NSX Edge node is going to be hosted on the same vSphere cluster as the existing NSX Edge nodes (in-cluster), the edgeClusterExpansionSpec ignores any information you provide for firstNsxVdsUplink and secondNsxVdsUplink.

Workaround: None. This is by design. For in-cluster expansions, new NSX Edge nodes use the same NSX VDS uplinks as the existing NSX Edge nodes in the NSX Edge cluster.
Stretch cluster operation fails

If the cluster that you are stretching does not include a powered-on VM with an operating system installed, the operation fails at the "Validate Cluster for Zero VMs" task.

Workaround: Make sure the cluster has a powered-on VM with an operating system installed before stretching the cluster.

vRealize Suite Known Issues

vRealize Operations Manager: VMware Security Advisory VMSA-2021-0018

VMSA-2021-0018 describes security vulnerabilities that affect VMware Cloud Foundation.

Workaround: See KB 85452 for information about applying vRealize Operations Security Patches that resolve the issues.
Updating the DNS or NTP server configuration does not apply the update to vRealize Automation

Using the Cloud Foundation API to update the DNS or NTP servers does not apply the update to vRealize Automation due to a bug in vRealize Suite Lifecycle Manager.
Workaround: Manually update the DNS or NTP server(s) for vRealize Automation.

Update the DNS server(s) for vRealize Automation
1. SSH to the first vRealize Automation node using root credentials.
2. Delete the current DNS server using the following command:
  
  sed '/nameserver.*/d' -i /etc/resolv.conf
3. Add the new DNS server IP with following command:
  
  echo nameserver [DNS server IP] >> /etc/resolv.conf
4. Repeat this command if there are multiple DNS servers.
5. Validate the update with the following command:
  
  cat /etc/resolv.conf
6. Repeat these steps for each vRealize Automation node.
Update the NTP server(s) for vRealize Automation
1. SSH to the first vRealize Automation node using root credentials.
2. Run the following command to specify the new NTP server:
  
  vracli ntp systemd --set [NTP server IP]
  
  To add multiple NTP servers:
  
  vracli ntp systemd --set [NTP server 1 IP,NTP server 2 IP]
3. Validate the update with the following command:
  
  vracli ntp show-config
4. Apply the update to all vRealize Automation nodes with the following command:
  
  vracli ntp apply
5. Validate the update by running the following command on each vRealize Automation node:
  
  vracli ntp show-config
Connecting vRealize Operations Manager to a workload domain fails at the "Create vCenter Server Adapter in vRealize Operations Manager for the Workload Domain" step

When you connect vRealize Operations Manager to a workload domain, it fails at the Create vCenter Server Adapter in vRealize Operations Manager for the Workload Domain step with a message similar to Failed to configure vCenter <vcenter-hostname> in vROps <vrops-hostname>, because Failed to manage vROps adapter. This issue can occur when the vRealize Operations cluster is offline.
Workaround: Make sure that the vRealize Operations cluster is online.
1. Log in to the vRealize Operations Manager administration interface.
2. Click Administration > Cluster Management and check the cluster status.
3. If the vRealize Operations cluster is offline, bring the cluster online.
4. When the cluster status displays as online, retry connecting vRealize Operations Manager to a workload domain.
vRealize Operations Management Pack for VMware Identity Manager is not installed

If you install vRealize Operations Manager before you install Workspace ONE Access, then the vRealize Operations Management Pack for VMware Identity Manager is not installed.
Workaround:
1. Log in to the vRealize Suite Lifecycle Manager appliance.
2. Click VMware Marketplace.
3. Enter "Identity Manager" in the Search text box.
4. Download and install the vRealize Operations Management Pack for VMware Identity Manager.
5. Log in to vRealize Operations Manager.
6. On the main navigation bar, click Administration.
7. In the left pane, select Solutions > Other accounts.
8. Click Add account.
9. On the Account types page, click VMware Identity Manager adapter.
10. Configure the settings, choosing the default collector group.
11. In the Connection information section, click the Add icon.
12. In the Manage credential dialog box, configure the Workspace ONE Access credentials and click OK.
13. On the New account page, click Validate connection.
14. In the Info dialog box, click OK.
15. Click Add.
16. On the Other accounts page, verify that the collection status of the adapter is OK.
Deploying a second vRealize Suite Lifecycle Manager fails

If you have multiple instances of VMware Cloud Foundation in the same SSO domain and you try to deploy vRealize Lifecycle Manager on both, the second deployment will fail with the message Add vCenter Server and Data Center to vRealize Suite Lifecycle Manager Failed.

Workaround: Use a single vRealize Suite Lifecycle Manager to manage instances of VMware Cloud Foundation in the same SSO domain.
vRealize Suite Lifecycle Manager reports a "FAILED" inventory sync

After rotating a vCenter Server service account password in SDDC Manager, the inventory sync may fail for vRealize Suite environments managed by VMware Cloud Foundation.

Workaround: Log in to vRealize Suite Lifecycle Manager to identify and troubleshoot the failed environment(s).