check-circle-line exclamation-circle-line close-line

VMware Cloud Foundation 3.0.1 | 18 OCTOBER 2018 | Build 10426441

VMware Cloud Foundation is a unified SDDC platform that brings together VMware vSphere, vSAN, NSX and optionally, vRealize Suite components, into a natively integrated stack to deliver enterprise-ready cloud infrastructure for the private and public cloud. The Cloud Foundation 3.0.1 release continues to expand on the SDDC automation, VMware SDDC stack, and the partner ecosystem.

NOTE: VMware Cloud Foundation 3.0.1 must be installed as a new deployment or upgraded from Cloud Foundation 3.0.

What's in the Release Notes

The release notes cover the following topics:

What's New

The VMware Cloud Foundation 3.0.1 release includes the following:

  • Host imaging added to Cloud Foundation Builder VM.
    Users can now image ESXi hosts using the Cloud Foundation Builder VM. See the product documentation for more information.
    NOTE: Hosts are not imaged during the bring-up process.

  • Numerous issue fixes and feature enhancements, including:

    • Bringup improvements: greater Cloudbuilder integration and improved configuration file upload.

    • Security enhancements: updated support for OpenSSL, improved inline documentation for certificate replacement, and improved password update process.

    • Workload domain configuration: improved tooltips, field descriptions, and error messaging; corrected minor bugs that affected successful workload domain creation, and more comprehensive host validation.

    • vRealize Suite integration: numerous bug fixes that improve deployment with Cloud Foundation and component products, improved connectivity with workload domains.

  • Manual guidance for Stretched Cluster, Disaster Recovery, and cross-vCenter NSX connectivity is available in the  Cloud Foundation 3.0.1 documentation.
    The manual guidance is provided as part of the 3.0.1 documentation set, and applies also to the Cloud Foundation 3.0 release.

  • Supportability and Serviceability (SoS) Tool improvements, including: health-check for stretched vSAN clusters, in addition to numerous improvements to log bundle collection.

  • New Direct consumption of Deployment Parameter Sheet.
    The Cloud Foundation Builder VM can directly read the Deployment Parameter Sheet without using the JSON generator.

Cloud Foundation Bill of Materials (BOM)

The Cloud Foundation software product is comprised of the following software Bill-of-Materials (BOM). The components in the BOM are interoperable and compatible.

Software Component Version Date Build Number
Cloud Foundation Builder VM 3.0.1 18 OCT 2018 10426441
SDDC Manager 3.0.1 18 OCT 2018 10426441
VMware vCenter Server on vCenter Server Appliance 6.5 U2c 13 AUG 2018 9451637
VMware Platform Services Controller 6.5 U2c 13 AUG 2018 9451637
VMware vSphere (ESXi) 6.5 EP9 02 OCT 2018 10175896
VMware vSAN 6.6.1 EP9 02 OCT 2018 10175901
VMware NSX Data Center for vSphere 6.4.1 25 MAY 2018 8599035
VMware vRealize Automation 7.4 11 APR 2018 8229492
VMware vRealize Log Insight 4.6.1 07 JUN 2018 8597028
VMware vRealize Operations 6.7 11 APR 2018 8183617
vRealize Suite Lifecycle Manager 1.2 (patch) 11 SEP 2018 9145185
VMware NSX content pack for vRealize Log Insight 3.7 n/a


VMware vSAN content pack for vRealize Log Insight 2.0 n/a n/a

IMPORTANT: VMware Cloud Foundation downloads upgrade manifests for non-applicable upgrade bundles. For example, in VMware Cloud Foundation 3.x, in addition to the pertinent VMware Cloud Foundation 3.x manifest, you may also see update manifests present for VMware Cloud Foundation 2.x. For more information refer to Knowledge Base article 65045 VMware Cloud Foundation downloads upgrade manifests for non-applicable upgrade bundles.

VMware Software Edition License Information

The SDDC Manager software is licensed under the Cloud Foundation license. As part of this product, the SDDC Manager software deploys specific VMware software products.

The following VMware software components deployed by SDDC Manager are licensed under the Cloud Foundation license:

  • VMware ESXi
  • VMware vSAN
  • VMware NSX Data Center for vSphere

The following VMware software components deployed by SDDC Manager are licensed separately:

  • VMware vCenter Server
    NOTE Only one vCenter Server license is required for all vCenter Servers deployed in a Cloud Foundation system.
  • VMware vRealize Automation
  • VMware vRealize Operations
  • VMware vRealize Log Insight and content packs
    NOTE Cloud Foundation permits limited use of vRealize Log Insight for the management domain without purchasing full vRealize Log Insight licenses.

For details about the specific VMware software editions that are licensed under the licenses you have purchased, see the Cloud Foundation Bill of Materials (BOM) section above.

For more general information, see the Cloud Foundation product page.

Supported Hardware

For details on vSAN ReadyNodes in Cloud Foundation, see the VMware Compatibility Guide (VCG) for vSAN and the Hardware Requirements section in the VMware Cloud Foundation Planning and Preparation Guide.


To access the Cloud Foundation 3.0.1 documentation, go to the VMware Cloud Foundation documentation landing page.

To access the documentation for VMware software products that SDDC Manager can deploy, see their documentation landing pages and use the drop-down menus on the page to choose the appropriate version:

Browser Compatibility and Screen Resolutions

The Cloud Foundation web-based interface supports the following web browsers:

  • Google Chrome: Version 69.x or 68.x
  • Mozilla Firefox: Version 62.x or 61.x

For the Web-based user interfaces, the supported standard resolution is 1024 by 768 pixels. For best results, use a screen resolution within these tested resolutions:

  • 1024 by 768 pixels (standard)
  • 1366 by 768 pixels
  • 1280 by 1024 pixels
  • 1680 by 1050 pixels

Resolutions below 1024 by 768, such as 640 by 960 or 480 by 800, are not supported.

Installation and Upgrade Information

You can install Cloud Foundation 3.0.1 as a new release or upgrade from Cloud Foundation 3.0.

Installing as a New Release

The new installation process has three phases:

Phase One: Prepare the Environment

The VMware Cloud Foundation Planning and Preparation Guide provides detailed information about the software, tools, and external services that are required to implement a Software-Defined Data Center (SDDC) with VMware Cloud Foundation, using a standard architecture model.

Phase Two: Image all servers with ESXi

Image all servers with ESXi 6.5 EP9 (build 10175896). See Knowledge Base article 58715 Virtual Machines running on VMware vSAN 6.6 and later report guest data consistency concerns following a disk extend operation for details.

Phase Three: Install Cloud Foundation 3.0.1

Please refer to the following user documentation:

  1. The VMware Cloud Foundation Architecture and Deployment Guide, which provides a high-level overview of the VMware Cloud Foundation product and its architecture. This document also describes the deployment process for Cloud Foundation.
  2. The VMware Cloud Foundation Operations and Administration Guide, which provides information about managing a VMware Cloud Foundation system, including managing the system's virtual infrastructure, managing users, configuring and deploying service offerings, and upgrading and monitoring the system.

Upgrade to Cloud Foundation 3.0.1

You can upgrade to Cloud Foundation 3.0.1 only from Cloud Foundation 3.0.

NOTE: The upgrade process from VMware Cloud Foundation 3.0 to 3.0.1 causes a reconfiguration to the Adapter Teaming Policy and to the HA settings for the management cluster. You must manually correct these configurations after the upgrade. See the following known issues listed below in this Release Note:

The Cloud Foundation 3.0.1 update bundle includes the VMware software components described in the table below. This patch bundle is hosted on the VMware Depot site and available via the Lifecycle Management feature in SDDC Manager. See Lifecycle Management in the VMware Cloud Foundation Operations and Administration Guide.

Software Component Version Date Build Number
VMware Cloud Foundation Upgrade Bundle 3.0.1 18 OCT 2018 10426441
VMware vSphere (ESXi) 6.5 EP9 02 OCT 2018 10175896
VMware vSAN 6.6.1 EP9 02 OCT 2018 10175901

Preparing for Upgrade

Before upgrading, perform the following tasks:

  1. Download LCM upgrade bundles throught the SDDC Manager Dashboard or using the bundle transfer utility. See Patching and Upgrading Cloud Foundation in the VMware Cloud Foundation Operations and Administration Guide
  2. Before scheduling an upgrade, run the pre-upgrade-check utility with the following command from the SDDC Manager Dashboard.
    Navigate to Inventory > Workload Domains > [Workload Domain Name] > Update/Patches tab to access the pre-upgrade check control.

Resolved Issues

  • System not taking snapshot prior to certificate replacement

    By design, the certificate replacement process for VI domain resources should automatically take a snapshot in case the replacement process fails and the user needs to roll back. However, the system is failing to take this snapshot.

    This issue is fixed in this release (3.0.1).

  • SoS log collection fails with default options

    When running SoS log collection with default options, the discovery protocol check fails because it is trying to get details from ESXi nodes that are outside the current workload domain. By defaul,t SoS runs for the management domain if no other domain is specified with the --domain-name flag.
    This error occurs when more than one workload domain is present.

    This issue is fixed in this release (3.0.1).

  • SSO password rotation fails in NsxManagerSsoUpdater

    After replacing the certificate, the password rotation fails in NsxManagerSsoUpdater, with an error message that the certificate doesn't match any of the alternative names. This is caused by reverse lookup failing to resolve on the DNS server for the NSX Manager.

    This issue is fixed in this release (3.0.1).

  • Update depot login details do not appear in the Administration > Update Management page

    After manual restart of LCM service, the LCM depot enters a NOT_INITIALIZED state. This issue self-corrects during polling, but that may take up to an hour.

    This issue is fixed in this release (3.0.1).

Known Issues

The known issues are grouped as follows.

Bringup Known Issues
  • Bringup service logs are not accessible by admin user

    Although the admin user has permission on the Cloud Foundation Builder VM, bringup logs can only be accessed by the root user.

    Workaround: See Knowledge Base article 59247 How to download logs from the SDDC Manager VM or Cloud Foundation Builder VM in VMware Cloud Foundation 3.0.

  • vSAN disk validation returns cache tier.

    During the vSAN disk validation operation the system returns the error "Host 'X' does not contain the minimum VSAN SDD cache disk required. VSAN cache their not within 200GB +-13.0 percent specification - FAIL". This may be caused by the validation process not correctly identifying the correct cache tier capacity requirements for the vSAN capacity tier size.

    Workaround: Manually verify that the SSD Cache tier is the appropriate size for the specificed vSAN Capacity tier. The cache tier must be at least 10% of capacity tier. Once verified, error can be safely bypassed by clicking Acknowledge.

  • Cloud Foundation Builder fails to initiate with "[Admin/Root] password does not meet standards" message

    When configuring the Cloud Foundation Builder admin and root passwords, format restrictions are not validated. As a result, a user may create a password that does not comply with the restrictions. As a result, Cloud Foundation Builder will fail upon initiation.

    Workaround: When configuring Cloud Foundation Builder, ensure that the password meets the following restrictions:

    • Minimum eight characters long.
    • Must include both uppercase and lowercase letters
    • Must include digits and special characters
    • Must not include common dictionary words
  • Bring-up fails during SDDC Manager VMCA certificate installation task

    The bring-up may fail during SDDC Manager VMCA certificate installation task.

    Workaround: Wait a few minutes, then retry bring-up using the Retry feature.

  • Bringup process fails at task Disable TLS 1.0 on vRealize Log Insight Nodes

    The Bringup fails at the task Disable TLS 1.0 on vRealize Log Insight Nodes with the following error Connect to [/] failed: Connection refused (Connection refused). This issue has been observed on slow environments after restarting a vRealize Log Insight node. The node does not start correctly and its API is not reachable.

    Workaround: Use the following procedure to work around this issue.

    1. Restart the failed Bringup execution in the Cloud Foundation Builder VM and open the bringup logs.
      This will retry the failed the Bringup task, which might still fail on the initial attempt. The log shows an unsuccessful connection to the LogInsight node.
    2. While Bringup is still running, use SSH to log in to the Log Insight node that is shown as failed in the Bringup log.
      loginsight-node-2:~ # service loginsight status
    3. Execute the following command:
      loginsight-node-2:~ # mv /storage/core/loginsight/cidata/cassandra/data/system ~/cassandra_keyspace_files
    4. Reboot the Log Insight node and confirm that it is running.
      loginsight-node-2:~ # uptime
      18:25pm up 0:02, 1 user, load average: 3.16, 1.07, 0.39
      loginsight-node-2:~ # service loginsight status
      Log Insight is running.

    In a few minutes, the Bringup process should successfully establish a connection to the LogInsight node and proceed.

  • vSAN SSD capacity disks marked as eligible

    When running pre-bringup Audit Validation, an error displays because the audit shows the capacity to be under a terabyte, which is the recommended minimum capacity disk size.

    Workaround: The user has two options:

    • Ignore the error. It will does not prevent the user from completing the workflow. User can click Acknowledge to acknowledge the validation failure and proceed to bringup.
    • Execute the following command on each ESXi node in the deployment:
      esxcli storage core device list | grep -B 3 -e "Size: 3662830" | grep ^naa > /tmp/capacitydisks; for i in `cat /tmp/capacitydisks`; do esxcli vsan storage tag add -d $i -t capacityFlash; vdq -q -d $i; done
      NOTE: The size parameter in the above command will vary from customer to customer.
  • Bringup and VI workload domain workflows fail at VM deployments if any hosts are in maintenance mode

    Neither operation checks for host maintenance mode state. As a result, NSX controller deployments fail. This is expected because vSAN default policy requires a minimum of three ESXi nodes to be available for deployment.

    Workaround: If you encounter this error, do the following:

    1. Through either vCenter or the esxcli utility, take the affected hosts out of maintenance mode:
      esxcli system maintenanceMode set -e 0
    2. Restart the failed workflow.
  • Cloud Foundation Builder VM remains locked after more than 15 minutes.

    The VMware Imaging Appliance (VIA) locks out the user after three unsuccessful login attempts. Normally, the lockout is reset after fifteen minutes but the underlying Cloud Foundation Bundle VM does not automatically reset.

    Workaround: Using SSH, log in as root to the Cloud Foundation Builder VM. Reset the password of the admin user with the following command.
    pam_tally2 --user=<user> --reset

Upgrade Known Issues
  • Operationsmanager component fails to come up after RPM upgrade.

    After manually upgrading the operations manager RPM to the latest version, the operationsmanager fails to come up. The system returns INFO-level message: Waiting for changelog lock... This is likely caused by overlapping restarts of the service preventing any from succeeding. This can happen to any service (e.g. ) which is exercising liquibase, such as commonsvcs.

    Workaround: Clean the databasechangeloglock table from the database.

    1. Log in to the SDDC Manager VM as admin user "vcf".
    2. Enter su to switch to root user.
    3. Run the following commands:
      1. Open the postgres command prompt:
        # psql -h /home/postgresql/ -U postgres
      2. Open the password manager:
        \c password_manager opsmgr
      3. Delete the databasechangeloglock:
        delete from databasechangeloglock
        NOTE: You can combine the preceding steps into a single command:
        psql -h /home/postgresql/ -U postgres -d password_manager -c "delete from databasechangeloglock"
      4. Leave the password manager and exit from the postgres prompt.
      5. Restart the operationsmanager component:
        # systemctl restart operationsmanager
      6. Verify the operationsmanager is running:
        # curl http://localhost/operationsmanager/about
        It should return something like:
        "","description":"Operations Manager"}
  • Precheck returns Java Resource Access Exception

    The precheck operation returns the following Java error: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "https://nsxManager.qr13.vcf.local/api/1.0/appliance-management/backuprestore/backup". This issue has been observed after replacing the certificate for the PSC, vCenter, NSX, and vRealize Automation components, and may be caused by a loss of connection between the SDDC Manager virtual appliance and the NSX Manager node.

    Workaround: If you receive this error, do the following:

    1. Verify that the the SDDC Manager VM is reachable from the NSX Manager node, and that there are no SSL issues.
    2. Register the NSX Manager with the vCenter Server through the NSX Manager interface. (If applicable, restart the vSphere Web client before running an upgrade.)
    3. Replace the certificate again, and retry the precheck operation.
  • Post-Upgrade Configuration: User Must Reconfigure Adapter Teaming Policy for Management Portgroup

    The upgrade process from VMware Cloud Foundation 3.0 to 3.0.1 results in the teaming policy defaulting to a "Route based on originating virtual port" configuration. The policy for the Management, vMotion, and vSAN port groups should be set to the "Route based on physical NIC load" configuration.

    Workaround: Manually reconfigure the NIC teaming policy of the portgroups of already deployed clusters.

    In vCenter, click on the virtual distributed switch that corresponds to the cluster.

    1. Click on the management portgroup of the switch.
    2. Click the Configure tab.
    3. Choose Edit > Teaming & Failover.
    4. Change the Load Balancing setting to Route based on physical NIC Load.
    5. Repeat for vMotion and vSAN.
  • Post-Upgrade Configuration: User Must Reconfigure HA Settings for Management Cluster

    The upgrade process from VMware Cloud Foundation 3.0 to 3.0.1 results in the HA configuration settings for the management cluster defaulting to a "VM and Application Monitoring" configuration. The HA settings for the management cluster should be restored to the "VM Monitoring Only" setting.

    Workaround: Manually reconfigure the HA configuration for the management cluster.

    1. In the vSphere Web Client, navigate to the management cluster.
    2. Choose Configure > vSphere Availability > Edit.
    3. Under Failures and responses, change the setting for VM Monitoring from "VM and Application Monitoring" to "VM Monitoring Only".
  • Upgrade process returns invalid data error.

    Observed when upgrading from 3.0 to 3.0.1, and from 3.0.1 to 3.5. When upgrading through the Lifecycle Manager, the system returns this error: Scheduling immediate update of bundle failed. UPGRADE_SPEC_INVALID_DATA; User Input cannot be null/empty, User Input is required for this upgrade.

    Workaround: Refresh the browser page.

  • New Deploying vRealize Suite upgrade bundle may inadvertently pick up incorrect Cloud Foundation image.

    When upgrading to 3.0.1, the user may inadvertently download bundles for 3.5.x. If the user then deploy vRealize Operations Manager, it may pick up the incorrect Cloud Foundation image.

    Workaround: If you encounter this issue, you can resolve it by modifying the file before deploying vRealize Operations Manager.

    1. Using SSH, log in to the SDDC Manager VM.
    2. Locate and open in a text editor the /opt/vmware/vcf/lcm/lcm-app/conf/ file.
    3. Add the following section to the text:
      ############## VR suite install versions for Delaware release ###########
    4. Save and close the file.
    5. Restart the Lifecycle manager.
      systemctl restart lcm
    6. To verify, run the following commands.
      They should return the older vRealize product versions.:
      curl 'http://localhost/lcm/compliance/versions/VROPS?scope=DEPLOYMENT'
      curl 'http://localhost/lcm/compliance/versions/VRA?scope=DEPLOYMENT'
      curl 'http://localhost/lcm/compliance/versions/VRSLCM?scope=DEPLOYMENT'
vRealize Integration Known Issues
  • Bundle Transfer Utility does not download install bundles

    The Bundle Transfer Utility packaged with the SDDC Manager VM does not download vRealize install bundles.

    Workaround: Download the Bundle Transfer Utility from KB article 58838 to a computer with internet access. Then follow the instructions in the Manually Download Update Bundles section of the Operations and Administration guide.

  • vRealize Operations in vRealize Log Insight configuration fails when vRealize Operations appliances are in a different subdomain

    During vRealize Suite deployment in Cloud Foundation, the user provides FQDN values for vRealize load balancers. If these FQDNs are in a different domain than the one used during initial bringup, the deployment may fail.

    Workaround: To resolve this failure, you must add the vRealize Operations domain to the configuration in the vRealize Log Insight VMs.

    1. Log in to the vRealize Log Insight VM to modify the /etc/resolv.conf file.
      domain vrack.vsphere.local
      search vrack.vsphere.local vsphere.local 
    2. Add the domain used for vRealize Operations to the last line above.
    3. Repeat on each vRealize Log Insight VM.
  • Certificate replacement for vRealize Operations component requires load balancer reconfiguration to pass-through

    For vRealize Operations, the certificate resides by default on the load balancer, not on the individual nodes as with most other components. After replacing the certificate for the first time, you must ensure that the NSX load balancer for vRealize Operations nodes is configured for SSL pass-through since the new certificate will reside on the individual nodes.

    Workaround: Reconfigure the appropriate NSX Edge using the following procedure:

    1. Log in into the Management vCenter and navigate to Home > Networking & Security.
    2. Select NSX Edges in the Navigator.
    3. Confirm that the IP address in the NSX Manager field is same as the IP address for the NSX manager for the management domain.
    4. Double-click the NSX Edge labeled vrealize-edge.
    5. Select the Manage tab, then the Load Balancer tab.
    6. Open Application Profiles.
    7. Find and click the profile vrops-https profile with name and click Edit.
    8. Select the Enable SSL Passthrough option and click OK.
  • User must manually accept the vCenter Server certificate in vRealize Automation after connecting to workload domains.

    After deploying vRealize Automation and connecting it to workload domains in Cloud Foundation, the user must switch to the vRealize Automation interface and to manually accept the security certificate.

    Workaround: After integrating vRealize Automation with VI Workload Domain, accept the certificate as follows:

    1. Log in to vRealize Automation using the Tenant Administrator account.
    2. Go to the Infrastructure tab and select Endpoints.
    3. Click NSX endpoint.
    4. Click Test Connection.
    5. Click OK to accept the NSX certificate.
    6. Click OK to confirm the endpoint settings.
    7. Repeat steps 3.-6. for the vSphere endpoint.
  • vRealize Suite Lifecycle Manager is not properly cleaned-up during uninstall.

    vRealize Suite Lifecycle Manager is not cleaned-up during an uninstall of a failed vRealize Automation or vRealize Operations Manager deployment. This may happen if the deployment workflow for vRealize Automation or vRealize Operations fails under some specific conditions on step “ChangeVrslcmPasswords”. For example, if SDDC Manager services are restarted during this operation. As a result, vRealize Suite Lifecycle Manager VM is not be properly cleaned up from the system during the uninstall process of the failed vRealize Automation or vRealize Operations Manager deployment.

    Consecutive attempts to deploy vRealize Automation or vRealize Operations Manager will fail due to vRealize Suite Lifecycle Manager being left in a bad state.

    Workaround: Follow the steps outlined in Knowledge Base article 57917 to fix the issue.

  • vRealize Suite: The IaaS ManagerService stops running on the IaaS manager service nodes after certificate replacement.

    After replacing the certificate for the vRealize Automation resource, the Manager Service stops running on the IaaS manager service nodes. This can be observed by accessing the vRealize Automation appliance and opening the vRA Settings tab. Expand the IaaS manager service entries (for example, iaasms1.<serviceusername>.local or iaasms2.<serviceusername>.local) and the ManagerService shows a status of Stopped.

    Workaround: After completing certificate replacement, you must manually restart the ManagerService on the principal IaaS manager service node: for example, iaasms1.<serviceusername>.local. (The name of the node on your system may vary.) Access the node as described above and restart it. It may take five to ten minutes for the node to restart. It is recommended you return to verify the restart. You must also restart the VMware vCloud Automation Center Service.

    NOTE: You only need to restart one node. The other will restart as a peer.

  • Certificate replacement for the vRealize Automation component fails with 401 error

    Certificate replacement for the vRealize Automation component fails due to a 401 unauthorized error with the message "Importing certificate failed for VRA Cafe nodes." This issue is caused by a password lockout in the vRealize Automation product interface. For example, independently of Cloud Foundation, a user tried to log in to vRealize Automation with the wrong credentials too many times, causing the lockout.

    Workaround: The lockout period lasts for thirty minutes, after which the certification replacement process can succeed.

  • IP address for the load balancer VM shows as N/A in the SDDC Manager interface.

    In the Services tab in the MGMT domain page shows N/A as the IP address for the vRealize Log Insight load balancer VM.

    Workaround: The user can discover the IP as follows:

    1. Log in to the SDDC Manager VM, and change to root user.
    2. Run the following command to return the load balancer IP address: nslookup <vrli-hostname>.
Networking Known Issues
  • Platform audit for network connectivity validation fails

    The vSwitch MTU is set to the same MTU as the VXLAN VTEP MTU. However, if the vSAN and vMotion MTU are set to 9000, then vmkping fails.

    Workaround: Modify the nsxSpecs settings in the bring-up JSON by setting the VXLANMtu as a jumbo MTU because vSwitch is set with the VXLAN MTU value. This will prevent the error seen in the platform audit.

SDDC Manager Known Issues
  • No warning to prevent user from replacing certificates during updates

    The SDDC Manager Dashboard does not prevent the user from replacing certificates while product updates are in progress. This is unsupported.

    Workaround: Do not replace certificates if any update operations are in progress. Wait until updates complete before proceeding.

Workload Domain Known Issues
  • The vSAN HCL database does not update as part of workload domain creation

    When you create a workload domain, the vSAN HCL database should update as part of the process. As a result, database moves into a CRITICAL state, as observed from vCenter.

    Workaround: Manually update the vSAN HCL database as described in Knowledge Base article 2145116.

  • Adding host fails when host is in a different VLAN

    This operation should succeed as adding a host to workload domain cluster should succeed even though the new host is on a different VLAN than other hosts in the same cluster.


    1. Before attempting to add a host, add a new portgroup to the VDS for the cluster.
    2. Tag the new portgroup with the VLAN ID of the host to be added.
    3. Run the Add Host workflow in the SDDC Manager Dashboard.
      This will fail at the "Migrate host vmknics to dvs" operation.
    4. Locate the failed host in vCenter, and migrate the vmk0 of the host to the new portgroup you created in step 1.
    5. Retry the Add Host operation.
      It should succeed.

    NOTE: If you remove the host in the future, remember to manually remove the portgroup, too, if it is not used by any other hosts.

  • Add cluster to domain operation fails with error FAILED_TO_GET_COMPLIANT_ESXI_VERSIONS

    This error most likely occurs if you attempt the Add Cluster workflow shortly after an LCM update. Due to a separate issue, the LCM cache is taking longer to refresh than expected, thus returns the wrong component version information and resulting in the error.

    Workaround: Wait for at least 5 minutes after LCM updates complete before initiating the Add Cluster operation.

  • Add cluster to domain fails with error: FAILED_TO_GET_COMPLIANT_ESXI_VERSIONS

    This issue has been observed when a user attempts to add a cluster to a newly created workload domain. The domain creation workflow includes creating a cluster for that domain. However, even though the domain creation workflow may be completed, the new cluster may require up to five minutes to be recognized. This error results if a user tries to add an additional cluster during this five minute period.

    Workaround: After creating a new workload domain, all five minutes to pass before adding a new host cluster to that domain.

Security Operations Known Issues
  • Updating password policy failure results in UPDATE message when should be FAILED

    If the password policy fails, the system shows an UPDATE status and the transaction history shows the message "Operation failed in 'appliance update', for  credential update." In actuality, the operation has FAILED because the new password does meet requirements. A more appropriate message would read "Password update has failed due to unmet policy requirements" and recommend reviewing the policy.

    Workaround: Review the password policy for the component in question and modify the password configuration as necessary, and try again to update.

  • SSL Certificate Replacement for vCenter breaks vRealize Operations data collection

    After replacing the certificate for the vCenter Server component, both the vCenter and vSAN components in vRealize Operations Manager report a "Collection failed" error message. Testing the connection and attempting to accept the new certificate returns additional error messages: Unable to establish a valid connection to the target system. Adapter instance has been configured to trust multiple certificates, when only one is allowed. Please remove any old, unneeded certificates and try again.

    Workaround: If you encounter this issue, use the following procedure to resolve the situation.

    1. Delete the current vCenter and vSAN adapters.
    2. Re-create them using the same configuration and credentials originally set by vRealize Operations.
    3. Test the connection, accept the new certificates, and save the configuration.
Log Collection Known Issues
  • Unable to use WinSCP to download logs from the SDDC Manager VM

    WinSCP is a popular and frequently used tool for secure file download and is supported by Cloud Foundation. This permissions error has an easy workaround.

    Workaround: For a detailed solution, see Knowledge Base article 59247: How to download logs from the SDDC Manager VM or Cloud Foundation Builder VM in VMware Cloud Foundation 3.0.

  • SoS log collection returns AssertionError: "execute_api_and_save_output "

    If the user kicks off the SoS utility suite before the VI workload domain creation or a vRealize Suite LCM update runs, the system returns an AssertionError message. However, this error does not impact SoS functionality with log collection or other SoS operations such as health check.

    Workaround: There is no workaround, other than to avoid running the SoS utility during workload domain creation or LCM updates.