check-circle-line exclamation-circle-line close-line

VMware Cloud Foundation 3.7.2 | 28 MAY 2019 | Build 13774914

VMware Cloud Foundation is a unified SDDC platform that brings together VMware vSphere, vSAN, NSX and optionally, vRealize Suite components, into a natively integrated stack to deliver enterprise-ready cloud infrastructure for the private and public cloud. The Cloud Foundation 3.7.2 release continues to expand on the SDDC automation, VMware SDDC stack, and the partner ecosystem.

NOTE: VMware Cloud Foundation 3.7.2 must be installed as a new deployment or upgraded from VMware Cloud Foundation 3.7.1. For more information, see Installation and Upgrade Information below.

What's in the Release Notes

The release notes cover the following topics:

What's New

The VMware Cloud Foundation 3.7.2 release includes the following:

  • CEIP Enhancements: Provides richer insights into the customer environments including versions, host details, workload domain details, and the password manager using the additional CEIP data.
  • Backup and Restore: Supports the file-based backup and restore capability of VMware Cloud Foundation SDDC Manager through APIs.
  • Support for vSphere Express Patch 09: VMware ESXi 6.7 ESXi670-201905001, vCenter Server Appliance 6.7 Update 2a, and vSAN 6.7 Express Patch 09 were added to the BOM.

Cloud Foundation Bill of Materials (BOM)

The Cloud Foundation software product is comprised of the following software Bill-of-Materials (BOM). The components in the BOM are interoperable and compatible.

Software Component Version Date Build Number
Cloud Builder VM 2.0.2 28 MAY 2019

13774914

SDDC Manager 3.7.2 28 MAY 2019

13774914

VMware vCenter Server Appliance vCenter Server 6.7 Update 2a 14 MAY 2019

13643870

VMware vSphere (ESXi) ESXi670-201905001 14 MAY 2019

13644319

VMware vSAN

6.7 Express Patch 09

14 MAY 2019

13356300

VMware NSX Data Center for vSphere 6.4.4 13 DEC 2018 11197766
VMware NSX-T Data Center 2.4 28 FEB 2019 12456646
VMware vRealize Suite Lifecycle Manager 2.0.0 Patch 2 26 DEC 2018 11187327
VMware vRealize Log Insight 4.7 04 OCT 2018 9983377
vRealize Log Insight Content Pack for NSX for vSphere 3.8 n/a n/a
vRealize Log Insight Content Pack for Linux 1.0 n/a n/a
vRealize Log Insight Content Pack for vRealize Automation 7.3+ 2.1 n/a n/a
vRealize Log Insight Content Pack for vRealize Orchestrator 7.0.1+ 2.0 n/a n/a
vRealize Log insight Content Pack for NSX-T 3.2 n/a n/a
VSAN content pack for Log Insight 2.0 n/a n/a
vRealize Operations Manager 7.0 24 SEP 2018 10098133
vRealize Automation 7.5 20 SEP 2018 10053539
Horizon 7 7.7.0 13 DEC 2018 11038474

Note: 

  • vRealize Log Insight Content Pack for vRealize Automation is installed during the deployment of vRealize Automation.
  • vRealize Log Insight Content Pack for vRealize Orchestrator is installed during the deployment of vRealize Automation.
  • vRealize Log Insight Content Pack for NSX-T is installed alongside the deployment of the first NSX-T workload domain.
  • VMware Solution Exchange and the vRealize Log Insight in-product marketplace store only the latest versions of the content packs for vRealize Log Insight. The software components table contains the latest versions of the packs that were available and automation at the time VMware Cloud Foundation released. When you deploy the VMware Cloud Foundation components, it is possible that the version of a content pack within the in-product marketplace for vRealize Log Insight is newer than the one used for this release.

VMware Software Edition License Information

The SDDC Manager software is licensed under the Cloud Foundation license. As part of this product, the SDDC Manager software deploys specific VMware software products.

The following VMware software components deployed by SDDC Manager are licensed under the Cloud Foundation license:

  • VMware ESXi
  • VMware vSAN
  • VMware NSX Data Center for vSphere

The following VMware software components deployed by SDDC Manager are licensed separately:

  • VMware vCenter Server
    NOTE Only one vCenter Server license is required for all vCenter Servers deployed in a Cloud Foundation system.
  • VMware vRealize Automation
  • VMware vRealize Operations
  • VMware vRealize Log Insight and content packs
    NOTE Cloud Foundation permits limited use of vRealize Log Insight for the management domain without purchasing full vRealize Log Insight licenses.

For details about the specific VMware software editions that are licensed under the licenses you have purchased, see the Cloud Foundation Bill of Materials (BOM) section above.

For more general information, see VMware Cloud Foundation.

Supported Hardware

For details on vSAN Ready Nodes in Cloud Foundation, see VMware Compatibility Guide (VCG) for vSAN and the Hardware Requirements section in the VMware Cloud Foundation Planning and Preparation Guide.

Documentation

To access the Cloud Foundation 3.7.2 documentation, go to the VMware Cloud Foundation product documentation.

To access the documentation for VMware software products that SDDC Manager can deploy, see the product documentation and use the drop-down menus on the page to choose the appropriate version:

Browser Compatibility and Screen Resolutions

The Cloud Foundation web-based interface supports the following web browsers:

  • Google Chrome: Version 7.0.x or 69.x
  • Internet Explorer: Version 11
  • Mozilla Firefox: Version 63.x or 62.x

For the Web-based user interfaces, the supported standard resolution is 1024 by 768 pixels. For best results, use a screen resolution within these tested resolutions:

  • 1024 by 768 pixels (standard)
  • 1366 by 768 pixels
  • 1280 by 1024 pixels
  • 1680 by 1050 pixels

Resolutions below 1024 by 768, such as 640 by 960 or 480 by 800, are not supported.

Installation and Upgrade Information

You can install Cloud Foundation 3.7.2 as a new release or upgrade from VMware Cloud Foundation 3.7.1.

In addition to these Release Notes, see the VMware Cloud Foundation Operations and Administration Guide for information about the upgrade process.

Installing as a New Release

The new installation process has three phases:

Phase One: Prepare the Environment

The VMware Cloud Foundation Planning and Preparation Guide provides detailed information about the software, tools, and external services that are required to implement a Software-Defined Data Center (SDDC) with VMware Cloud Foundation, using a standard architecture model.

Phase Two: Image all servers with ESXi

Image all servers with ESXi 6.7 Uand update to ESXi 6.7 EP 09 (Build 13644319). See Knowledge Base article 58715 Virtual Machines running on VMware vSAN 6.6 and later report guest data consistency concerns following a disk extend operation for details. See also the VMware Cloud Foundation Architecture and Deployment Guide for information on installing ESXi.

Phase Three: Install Cloud Foundation 3.7.2

Refer to the VMware Cloud Foundation Architecture and Deployment Guide for information on deploying Cloud Foundation.

Upgrade to Cloud Foundation 3.7.2

You can upgrade to Cloud Foundation 3.7.2 only from 3.7.1. If you are at a version earlier than 3.7.1, refer to the 3.7.1 Release Notes for information on how to upgrade to 3.7.1.

For information on upgrading to 3.7.2, refer to the VMware Cloud Foundation Operations and Administration Guide.

 

Resolved Issues

  • The addCluster operation in the NSX-T workload domain fails with  the "Invalid parameter: {0}" error message.

    The addCluster operation fails during the NSX-T fabric node creation (host prep
    operation) during the transport Node creation in NSX-T Manager. In this state, if the cluster is removed, then the logical switches are not deleted. Also if the host preparation was done for all the hosts, they are not removed from the NSX-T fabric.

     

  • The unstretching of the cluster fails to put the ESXi server in the maintenance mode

    This issue occurs when automated unstretching of clusters is executed on the manually-enabled stretch cluster.

     

  • Removal of the host from the cluster does not lead to migration of the powered off VM

    The default behavior of the remove host workflow in VMware Cloud Foundation 3.7.1 and earlier is to leave powered-off or suspended virtual machines on the host.

     

  • NSXT VI creation fails while issuing the VMCA certificate for NSX-T manager

    This issue occurs with exception from NSXT manager error message.

    Some of appliance components is not functioning., details=Component health: Unknown, error_code=101, module_name=common-services.

    This happens because the NSX-T manager is not up. This issue is seen intermittently.


  • Add host fails with the "UNABLE_TO_CREATE_TRANSPORT_NODE_COLLECTION" error

    The add host workflow fails if the cluster has a dead host in it which is part of the NSX-T fabric.

     

  • There are intermittent failures while creating the VI Domain

    The NSX-T VI workload domain fails at the assignment of the Virtual IP address to the NSX-T Manager cluster.

     

  • The add cluster to NSXT WLD operation fails with the "Did not find compute collection with clusterMoid 'domain-c45' and clusterName 'clus1-ext'" error message

    The compute manager status is down after the vCenter certificate replacement.

    Workaround:
    1. Login to NSX Manager using root.

    2. Run/etc/init.d/cm-inventory restart

    3. Repeat the above steps on all the three NSX-T Managers.

  • The creation of Horizon VDI fails at the "Join VMs to Domain" task

    As part of the Horizon VDI creation, all the management virtual machines (connections server, App Volumes, composer server, and so on) are connected to the domain with the administrator user name. This issue occurs because the Horizon VDI template is created with the administrator@vsphere.local as user name and it fails at the "Join VMs to Domain" task.

     

  • NSX-T VI workload domain creation fails with the "Unable to create transport node collection" error

    While deploying NSX-T workload domain, the task fails intermittently due to the NSX VIB installation failure on the host.

     

Known Issues

The known issues are grouped as follows.

Bringup Known Issues
  • Clicking the help icon in the bring-up wizard opens the help for an older release.

    The help icon in the bring-up wizard links to the product help for an older version.

    Workaround: To open the help topic on deploying Cloud Foundation, perform the following steps:

    1. In a browser window, navigate to docs.vmware.com.

    2. Click Browse All Products and then click VMware Cloud Foundation.

    3. Click VMware Cloud Foundation Architecture and Deployment Guide.

    4. In the left navigation pane, click Deploying Cloud Foundation.

  • Cloud Foundation Builder fails to initiate with the "[Admin/Root] password does not meet standards" message

    When configuring the Cloud Foundation Builder admin and root passwords, the format restrictions are not validated. As a result, a user may create a password that does not comply with the restrictions. As a result, Cloud Foundation Builder will fail upon initiation.

    Workaround: When configuring the Cloud Foundation Builder, ensure that the password meets the following restrictions:

    • Minimum eight characters long.
    • Must include both uppercase and lowercase letters
    • Must include digits and special characters
    • Must not include common dictionary words
  • The bring-up process fails at task disable TLS 1.0 on the vRealize Log Insight nodes

    The bring-up fails at the task disable TLS 1.0 on the  vRealize Log Insight nodes with the following error Connect to 10.0.0.17:9543 [/10.0.0.17] failed: Connection refused (Connection refused). This issue has been observed in the slow environments after restarting a vRealize Log Insight node. The node does not start correctly and its API is not reachable.

    Workaround: Use the following procedure to work around this issue.

    1. Restart the failed bring-up execution in the Cloud Foundation Builder VM and open the bring-up logs.
      This will retry the failed the bring-up task which might still fail on the initial attempt. The log shows an unsuccessful connection to the vRealize Log Insight node.
    2. While bring-up is still running, use SSH to log in to the vRealize Log Insight node that is shown as failed in the bring-up log.
    3. Run the following command to determine the connection issue.
      loginsight-node-2:~ # service loginsight status
      It should confirm that the daemon is not running.
    4. Execute the following command:
      loginsight-node-2:~ # mv /storage/core/loginsight/cidata/cassandra/data/system ~/cassandra_keyspace_files
    5. Reboot the vRealize Log Insight node.
    6. Confirm that it is running.
      loginsight-node-2:~ # uptime
      18:25pm up 0:02, 1 user, load average: 3.16, 1.07, 0.39
      loginsight-node-2:~ # service loginsight status
      Log Insight is running.

    In a few minutes, the bring-up process should successfully establish a connection to the vRealize Log Insight node and proceed.

  • The bring-up and the Virtual Infrastructure workload domain workflows fail at VM deployments if any hosts are in maintenance mode

    No operation checks for host maintenance mode state. As a result, NSX controller deployments fail. This is expected because vSAN default policy requires a minimum of three ESXi nodes to be available for deployment.

    Workaround: If you encounter this error, do the following:

    1. Through either VMware vCenter or the esxcli utility, take the affected hosts out of maintenance mode:
      esxcli system maintenanceMode set -e 0
    2. Restart the failed workflow.
  • The Cloud Foundation Builder VM remains locked after more than 15 minutes.

    The VMware Imaging Appliance (VIA) locks out the user after three unsuccessful login attempts. Normally, the lockout is reset after fifteen minutes but the underlying Cloud Foundation Builder VM does not automatically reset.

    Workaround: Using SSH, log in as admin to the Cloud Foundation Builder VM, then switch to root user. Unlock the account by resetting the password of the admin user with the following command.
    pam_tally2 --user=<user> --reset

  • Unable to cancel JSON Spec Validations.

    This is observed during the Bringup process when there is an error in the JSON file. The user is unable to cancel the JSON validation, but it might take up to five minutes if there's no connectivity to the ESXi hosts

    Workaround: There is no workaround to enable the desired cancellation. However, if this occurs, after the validation fails, review the JSON for syntax or other errors. Correct these errors and try again.

  • Validation fails on SDDC Manager license.

    During the bringup process, the validation for the SDDC Manager license fails.

    Workaround: Enter a blank license and proceed. You can enter the correct license value later in the process.

  • During bring-up, the component sheet incorrectly lists the wrong NSX version.

    During the bring-up process, detailed information of the components to be deployed are displayed. This display incorrectly shows NSX version 6.4.3 but this is incorrect. The actual version being deployed is NSX 6.4.4.

    Workaround: None

  • Cloud Foundation Builder: Restart of imaging service fails.

    During the host imaging operation, the imaging service (imaging.service) fails when restarted.

    Workaround: If you encounter this issue, perform the following procedure:

    1. Stop the imaging service in the SDDC Manager VM.
      systemctl stop imaging.service
    2. Stop all processes related to imaging.
      ps - ef | grep imag
      kill <process_number>
    3. Sleep the system for five seconds.
      sleep 5
    4. Start the imaging service.
      systemctl start imaging.service
      It should restart correctly.
  • NSX-V Workload domain creation may fail at NSX-V Controller deployment

    The workload domain creation workflow may fail at NSX-V Manager deploying the controllers to the new domain. The user must reboot each of the ESXi hosts in the cluster of the new domain, and restart the workflow. The user will have an unusable domain until then.

    Workaround: The user must reboot each of the ESXi hosts in the cluster of the new domain and restart the workflow.

  • During VMware Cloud Foundation bring-up, the transport zone replication mode is set to Hybrid but no multicast address is set.

    A requirement in NSX deployment is that the transport zone replication mode is set to Hybrid with proper multicast address set. But in this issue, no multicast address is set.

    Workaround:

    1. From the vSphere client, launch Networking and Security.
    2. Click Installation and Upgrade and navigate to Logical Network Settings.
    3. Click Edit for the Segment IDs.
    4. Turn on Multicast addressing.
    5. For Multicast addresses type in the range 239.1.0.0-239.1.255.255.
    6. Ensure that the IGMP snooping is enabled on the ToR physical switch and an IGMP the querier is available.
       
  • The VCF Cloud Builder fails during bring-up when no vCenter License key is supplied in the Excel file

    If you do not supply a license key in the excel file that can be used by the Cloud Builder during the bring-up process, the deployment will fail while renaming the applied license key in the "Deploy and configure vCenter Server" task.

    Workaround: None

Upgrade Known Issues
  • Operations against an existing NSX-T workload domain fails after upgrading VMware Cloud Foundation to version 3.5.1

    Operations against an existing NSX-T workload domain fails on the "Gather input to add host to NSX-T Fabric and migrate networking from dvs to nvds" task.  These operations can include creating a new cluster, expanding a cluster or removing a host from a cluster.

    Workaround: See Knowledge Base article 67030.

  • The Operationsmanager component fails to come up after RPM upgrade.

    After manually upgrading the operations manager RPM to the latest version, the operationsmanager fails to come up. The system returns INFO-level message: Waiting for changelog lock... This is likely caused by overlapping restarts of the service preventing any from succeeding. This can happen to any service (e.g. ) which is exercising liquibase, such as commonsvcs.

    Workaround: Clean the databasechangeloglock table from the database.

    1. Log in to the SDDC Manager VM as admin user "vcf".
    2. Enter su to switch to root user.
    3. Run the following commands:
      1. In the postgres command prompt, open the password manager to delete the database change log lock:
        # psql -h /home/postgresql/ -U postgres -d password_manager -c "delete from databasechangeloglock"
      2. Restart the operationsmanager component:
        # systemctl restart operationsmanager
      3. Verify the operationsmanager is running:
        # curl http://localhost/operationsmanager/about
        It should return something like:
        {"id":"2cac9b7c-545f-4e6d-a66-6f81eef27601","name":"OPERATIONS_MANAGER",
        "version":"3.1.0-SNAPSHOT-9580592","status":"ACTIVE","serviceUrl":
        "http://127.0.0.1/operationsmanager","description":"Operations Manager"}
  • Some component upgrades fail due to disabled NTP servers.

    Observed when upgrading from 3.0.x to 3.5. The Platform Service Controller and vCenter Server component updates fail because they are unable to ping the disabled NTP server. The NTP server may be unavailable during the upgrade.

    Workaround: Before upgrading, reconfigure the Platform Service Controller and vCenter Server components to synchronize with the ESXi hosts. After the upgrade, you can restore the previous configuration to synchronize with the NTP server. See Configure the System Time Zone and Time Synchronization Settings in the vSphere documentation for details.

  • The upgrade process to version 3.5 may leave some hosts using previous ESXi version.

    The Lifecycle Manager by default displays only the latest ESXi version based on the bill of materials. However, with workload domains running on two ESXi versions, errors may result.

    Workaround: Manually update the ESXi version on all hosts.

  • Cloud Foundation 3.5 to 3.5.1 upgrade: operations manager upgrade fails at SERVICE UPGRADE INITIATION phase.

    The likely cause of this error is the presence of failed host commission workflows. To upgrade successfully, there can be no such failed worklfows.

    Workaround: Before starting the upgrade process from 3.5 to 3.5.1, check your deployment for any failed host commission workflows and resolve them.

  • VMware Cloud Foundation 3.5.1 cannot validate the Horizon install bundle

    The Horizon install bundle is required only for VMware Cloud Foundation 3.7. If you upload a Horizon bundle on VMware Cloud Foundation 3.5.1 setup ,it may not be recognized.

    Workaround: Upgrade to VMware Cloud Foundation 3.7 first before uploading the Horizon install bundle.

  • The Operation Manager Upgrade failed at timeout

    This issue is seen in the below scenario:

    1. Restart the Operation Manager.
    2. The Resource Aggregator inside Operation Manager makes the API calls to NSX Manager to build the cache at the start of the Operation       Manager.
    3.The NSX manager does not respond to the API calls because of its state or services are down.
    4. The Operations Manager restart is hung.

    Workaround: Restart NSX Manager.

  • The VMware Cloud Foundation 3.5.1 to 3.7 configuration drift bundle shows as available after the fresh install of the VMware Cloud Foundation 3.7 version

    You see a patch bundle after the fresh install of VMware Cloud Foundation 3.7 version instead of an install bundle.

    Workaround: Apply this bundle on VMware Cloud Foundation 3.7. This does not change LCM version. The future updates will become available after this bundle is applied.

  • The NSX-T Install Bundle generated as part of vcf-lcm-packages should consume the vRealize Log Insight Content Pack from the artifactory

    In the vRealize Log Insight UI, under Content Packs for NSX-T, you may see that there is an update available.

    Workaround: None. You can go ahead and update the content pack.

  • You cannot download all the bundles at once when you use the lcm-bundle-transfer util tool

    For downloading 3.7.1 bundles, follow the below steps:

    1. Use the bundle transfer-util tool from 3.7.0 and download the VCF bundle with LCM.

    2. Transfer this bundle to SDDC Manager.

    3.  Apply the bundle downloaded in Step 1.

    4.  Copy bundle-transfer-util from SDDC manager again and download the remaining bundles.

    5. Transfer the remaining bundles and upload to SDDC Manager

     

    Workaround: None

  • The SOS log collection failing for vRealize Suite Lifecycle Manager with the" Fail to create vRealize Suite Lifecycle Manager support bundle" error

    The SoS log collection fails for vRealize Suite Lifecycle Manager after the password is rotated or updated for the vRealize products (vRealize Automation/vRealize Operations).

    Workaround: When you rotate the vRealize Operations/vRealize Automation password (SSH or API), you have to log in to the vRealize Lifecycle Manager UI, delete the existing vRealize Operations environment, and re-create it.

  • The vCenter upgrade operation fails on the management domain and workload domain

    vCenter fails to be upgraded because lcm-bundle-repo nfs mount on the host is inaccessible.

    Workaround: Remove and remount the SDDC Manager NFS datastore on the affected ESXi hosts. Use the showmount command to check if all hosts are displayed in the SDDC manager mount list.

  • The VMware Cloud Foundation upgrade from the 3.7.1 version to 3.7.2 version may fail at the Lifecycle Manager self-service upgrade 

    This is very rare race condition which is happening during Lifecycle Manager self-service upgrade. You may see VMware Cloud Foundation upgrade failing at the Lifecycle Manager upgrade phase.In this case, the actual upgrade is successful. It is only an issue with the version update to the inventory.

    Workaround: Complete the failed upgrade and restart the available upgrade back from the management updates or the patches section.

  • The VMware Cloud Foundation 372 Lifecycle Manager bundle release date shows up as May 27 instead of May 28.

    The VMware Cloud Foundation bundle has “May 28, 2019 10:47:08.145 AM GMT+05:30” as the release date in the manifest . So it reflects this date as May 27 when you download the bundle from SDDC Manager due to the time zone difference as the bundles are created during the IST time.

    Workaround: Ignore the issue.

vRealize Integration Known Issues
  • vRealize Operations in vRealize Log Insight configuration fails when vRealize Operations appliances are in a different subdomain

    During vRealize Suite deployment in Cloud Foundation, the user provides FQDN values for vRealize load balancers. If these FQDNs are in a different domain than the one used during initial bringup, the deployment may fail.

    Workaround: To resolve this failure, you must add the vRealize Operations domain to the configuration in the vRealize Log Insight VMs.

    1. Log in to the first vRealize Log Insight VM.
    2. Open the /etc/resolv.conf file in a text editor, and locate the following lines:
      nameserver 10.0.0.250
      nameserver 10.0.0.250
      domain vrack.vsphere.local
      search vrack.vsphere.local vsphere.local 
    3. Add the domain used for vRealize Operations to the last line above.
    4. Repeat on each vRealize Log Insight VM.
  • SoS network cleanup utility fails on the hosts in two VDS switch configuration.

    This utility is not supported in a two VDS switch configuration in the current release.

  • Certificate replacement for the vRealize Automation component fails with 401 error

    Certificate replacement for the vRealize Automation component fails due to a 401 unauthorized error with the message "Importing certificate failed for VRA Cafe nodes." This issue is caused by a password lockout in the vRealize Automation product interface. For example, independently of Cloud Foundation, a user tried to log in to vRealize Automation with the wrong credentials too many times, causing the lockout.

    Workaround: The lockout period lasts for thirty minutes, after which the certification replacement process can succeed.

  • vRealize Automation integration fails with NSX-T workload domain.

    NSX-T does not yet support vRealize Automation integration.

    Workaround: None.

  • Upgrade to vRealize Suite Lifecycle Manager 2.0 removes Environments Cards and Request History

    The upgrade process from vRealize Suite Lifecycle Manager 1.2 to 2.0 replaces the deployed appliance and restores a baseline configuration. This action subsequently removes any previous environment cards from the user interface and the request history. This may create auditing concerns; however, the originating requests for vRealize Suite product actions (such as deployment, workload domain connections, and so on) are maintained in the SDDC Manager logs.

    Workaround: Verify that vRealize Log Insight is manually configured for log retention and archiving. This will help to ensure that the SDDC Manager logs with the historical and originating vRealize Suite product action requests are preserved. For example, see Configure Log Retention and Archiving for vRealize Log Insight in Region A in the VMware Validated Design 4.3 documentation.

  • You cannot log in to vROPS UI after the API password rotation.

    Description required

    Workaround:

    1. Whenever the user updates or rotates a password for vRealize Operations Manager, the user has to manually log in to vRealize Log Insight and navigate to Administration > Vrealize Operations > Update Password..
    2. If the user forgets step 1, vRealize Log Insight tries to connect with the wrong credentials and after certain failed attempts account gets locked. To unlock the login to vRealize Operations Manager UI, update vRealize Log Insight.
  • The vRealize port group is configured with the incorrect teaming policy.

    The vRealize network is configured with Route based on originating virtual port load balancing policy. This could cause uneven traffic distribution between the physical network interfaces.

    Workaround: Manually upgrade NIC teaming policy in the vCenter UI:

    1. Log into vCenter.
    2. Navigate to Networking.
    3. Locate the Distributed switch in management vCenter.
    4. Right click on vRack-DPortGroup-vRealize portgroup -> Edit Settings.
    5. Select Teaming and failover.
    6. Change Load balancing from Route based on originating virtual port to Route based on physical NIC load.
    7. Click OK and verify that the operation have completed successfully.
Networking Known Issues
  • Platform audit for network connectivity validation fails

    The vSwitch MTU is set to the same MTU as the VXLAN VTEP MTU. However, if the vSAN and vMotion MTU are set to 9000, then vmkping fails.

    Workaround: Modify the nsxSpecs settings in the bring-up JSON by setting the VXLANMtu as a jumbo MTU because vSwitch is set with the VXLAN MTU value. This will prevent the error seen in the platform audit.

  • NSX Manager is not visible in the vSphere Web Client.

    In addition to NSX Manager not being visible in the vSphere Web Client, the following error message displays in the NSX Home screen: "No NSX Managers available. Verify current user has role assigned on NSX Manager." This issue occurs when vCenter Server and the permission is not correctly configured for the account that is logged in.

    Workaround: To resolve this issue, follow the procedure detailed in Knowledge Base article 2080740 "No NSX Managers available" error in the vSphere Web Client.

  • The PSC and VC update fails if the ping to the NTP servers is disabled.

    The VC upgrade cannot check the NTP time with the server.

    Workaround: Have the NTP server without any security and ensure that it works correctly on the standard NTP port.

  • A warning is displayed on the validation page in the UI in case a host has an IP address ending with 0.

    This issue also occurs in some cases where it is a valid IP address and should be validated accordingly. The Configuration File Validation page displays a warning under "Host and IP DNS Records" category if an ESXi host has an IP address ending with 0. This would be a valid warning if a /24 subnet is used, but if a /22 subnet is used, this is not an issue and no warning should exist. 

    Workaround: None

SDDC Manager Known Issues
  • Cancelling "Network Connectivity" validations is not cleaning up the temporary vSwitch (adt_vSwitch_01) that is created by Platform Audit

    After cancelling "Network Connectivity" validations when user re-runs validation, "ESXi Host Readiness" validation fails with error "Physical NIC vmnic1 is connected to adt_vSwitch_01 on esxi host esxi-1 (10.0.0.100): should be disconnected".

    This is because temporary vSwitch (adt_vSwitch_01) which was created during previous "Network Connectivity" validations was not cleaned up after cancellation.

    Note: In the above error message, the ESXi host name and the IP address will vary depending up on customer's environment.

    Workaround: Run the platform audit validation again after the "ESXi Host Readiness" validation failure.

Workload Domain Known Issues
  • The vSAN HCL database does not update as part of workload domain creation

    When you create a workload domain, the vSAN HCL database should update as part of the process. As a result, database moves into a CRITICAL state, as observed from vCenter.

    Workaround: Manually update the vSAN HCL database as described in Knowledge Base article 2145116.

  • Adding host fails when host is in a different VLAN

    This operation should succeed as adding a host to workload domain cluster should succeed even though the new host is on a different VLAN than other hosts in the same cluster.

    Workaround:

    1. Before attempting to add a host, add a new portgroup to the VDS for the cluster.
    2. Tag the new portgroup with the VLAN ID of the host to be added.
    3. Run the Add Host workflow in the SDDC Manager Dashboard.
      This will fail at the "Migrate host vmknics to dvs" operation.
    4. Locate the failed host in vCenter, and migrate the vmk0 of the host to the new portgroup you created in step 1.
      For more information, see Migrate VMkernel Adapters to a vSphere Distributed Switch in the vSphere product documentation.
    5. Retry the Add Host operation.
      It should succeed.

    NOTE: If you remove the host in the future, remember to manually remove the portgroup, too, if it is not used by any other hosts.

  • In some cases, VI workload domain NSX Manager does not appear in vCenter.

    Observed in NFS-based workload domains. Although VI workload domain creation was successful, the NSX Manager VM is not registered in vCenter and as a result, not appearing in vCenter.

    Workaround: To resolve this issue, use the following procedure:

    1. Log in to NSX Manager (http://<nsxmanager IP>).
    2. Navigate to Manage > NSX Management Service.
    3. Un-register the lookup service and vCenter, then re-register.
    4. Close the browser and log in to vCenter.
  • Unable to delete VI workload domain enabled for vRealize Operations Manager from SDDC Manager.

    Attempts to delete the vCenter adapter also fail, and return an SSL error.

    Workaround: Use the following procedure to resolve this issue.

    1. Create a vCenter adapter instance in vRealize Operations Manager, as described in Configure a vCenter Adapter Instance in vRealize Operations Manager.
      This step is required because the existing adapter was deleted by the failed workload domain deletion.
    2. Follow the procedure described in Knowledge Base article 56946.
    3. Restart the failed VI workload domain deletion workflow from the SDDC Manager interface.
  • Unable to create the transport node collection through the NSX-T Manager

    The process of deletion of the NSX-T workload fails with the UNABLE_TO_RE_CREATE_TRANSPORT_NODE_COLLECTION error.

    Workaround:

    1. Restart the proton service. Log in to all the NSX-T Manager instances as root and execute the /etc/init.d/proton restart command.

    2. After the proton service is up, log in to NSX-T Manager UI, ensure that the transport nodes state is success for the cluster. If not, select Configure NSX from NSX-T Manager for the compute collection by providing the respective transport node profile.
    3. Detach the transport node profile for the compute collection from the NSX-T Manager.
    4. Retry the workflow.

  • Adding cluster to a NSX-T workload domain fails with the error message "Invalid parameter: {0}".

    If you try to add cluster to a NSX-T workload domain and it fails with the error message "Invalid parameter: {0}", the subtask that creates logical switches has failed. This is likely due to an issue where artifacts from previously removed workload domains and clusters are conflicting with the new switch creation process.

    Workaround: Delete the hosts from the NSX-T Manager if exists and then delete the stale logical switches. the logical switches are created in the following pattern:

     ls-<UUID>-management, ls-<UUID>-vsan, ls-<UUID>-vmotion

    For all these three logical switches, {UUID} remains the same. For these logical switches, the logical ports are shown as "0" (ZERO).  The user has to delete these logical switches.

  • Unable to delete cluster from any NSX workload domain.

    The likely cause of this error is the presence of a dead or deactivated host within the cluster. To resolve this, you edit the workflow entry to remove reference to the offending host.

    Workaround: To resolve this issue, perform the following procedure.

    1. In SDDC Manager (Inventory > Workload Domains > [workload domain] > [cluster]), try to delete the cluster in order to identify the problematic host.
    2. Using SSH, log in to the SDDC Manager VM to obtain the workflow ID.
      Run curl -s http://localhost:7200/domainmanager/workflows to return a JSON listing all workflows.
    3. Using the workflow ID, get the workflow input.
      curl -s http://localhost:7200/domainmanager/internal/vault/<workflow-id> \
      -XGET > remove_cluster_input.json
    4. Edit the remove_cluster_input.json file by removing the entry referencing the problematic host.
      1. Find the reference to the host under the heading:
        "RemoveClusterEngine____2__RemoveClusterFromNsx____0__NsxtRemoveCluster____0__removeNsxtModel"
      2. Delete the entire entry for the problematic host.
      3. Save the edited file as remove_cluster_input_deadhost_removed.json.
    5. Update the workflow with the new file.
      curl -s http://localhost:7200/domainmanager/internal/vault/<workflow-id> -XGET > \
      -XPUT -H "Content-type: text/plain" -d @ remove_cluster_input_deadhost_removed.json
    6. Return to SDDC Manager and retry the workflow.
      It will fail at the logical switches deletion task.
    7. In NSX Manager, clear the host as follows:
      1. Go to NSX > Nodes > Transport Nodes and delete the same host as a transport node.
      2. Go to NSX > Nodes > Hosts and delete the same host.
    8. Return to SDDC Manager and restart the workflow.
      It should succeed.
  • Removal of a dead host from the NSX-T workload domain fails and subsequently, the removal of the workload domain fails

    In some cases where the creation of NSX-T workload domain fails, the subsequent attempt to delete the dead hosts fails as well. The host is disconnected from vCenter. Therefore, manual intervention is required to connect back. This issue is also seen when the domain is created successfully but then one of the host goes dead.

    Workaround:

    If a host removal operation fails, since the host is dead (i.e host is disconnected from VC), then perform the following steps:

    1. Log in to NSX-T Manager.
    2. Identify the dead host under fabric->nodes->transport nodes.
    3. Edit the transport node and select the N-VDS tab.
    4. Under Physical NICs, assign vmnic0 to the uplink.
    5. From SDDC Manager, retry the failed domain deletion task.

  • If there is a dead host in the cluster, the subsequent task of adding a cluster or a host fails.

    If one of the hosts of the workload domain goes dead and the user tries to remove the host, the task fails. And then, that particular host is set to the deactivating state without providing an option to forcefully remove it.

    Workaround: Bring the dead host back to normal state, after which the add-cluster and add-host tasks succeed.

  • NSX-V Workload domain creation may fail at the NSX-V Controller deployment

    The workload domain creation workflow may fail at NSX-V Manager deploying controllers to the new domain. This issue occurs because the NSX Controllers do not get an IP address because the VM NIC is disconnected.

    Workaround: The user must reboot each of the ESXi hosts in the cluster of the new domain and restart the workflow.

  • The workflow fails during the vCenter deployment If you provide an IP address that is in use already.

    The duplicate IP address detection happens in later stages.

    Workaround: Ensure that the IP addresses that are given are unused ones.In case of duplicate IP addresses, the  workflow input needs to be modified to get the task running on a retry.

  • NSX-T workload domain creation fails at the 'Join vSphere hosts to NSX-T Fabric' task

    When NSX-T domain creation workflow fails at "Join vSphere hosts to NSX-T Fabric" task, it is generally due to the failure of NSX-T installation on one of the hosts. When seen in NSX-T manager web portal,the  installation failure is clearly indicated in the host view.

    This happens intermittently. When this happens, detection of it fails until some further point. Eventually task fails after a long wait.

    Workaround: The user has to log in to the NSX-T Manager, check the host that has the NSX-T installation failure, and delete it from the fabric.

  • NSX-T workload domain deployment fails if the host names have upper case characters

    Deploying a NSXT WLD using hosts with hostname having uppercase characters, fails with the following error message -

    Message: Unable to get input for VDS to N-VDS migration

    Remediation Message: Reference Token: 602SME Cause: Type: java.lang.IllegalArgumentException Message: Unable to find transport node with name <host name>

     

    Workaround:

    1. To continue from the failed task -
           - Log in to NSX-T Manager and edit the transport node name to lower case.
           - Update the DNS record and the host name of the hosts.
           - Retry the failed task.
    2. To deploy a new domain, use the lower case characters for the host name

  • The stretch cluster workflow does not report failure when the VMs are not compliant or out of date

    The stretch cluster workflow does not check for the VM compliant status after the policy is reapplied as it takes time for VMs to become compliant.

    Workaround: Try the reapply policy manually using the Web client.

  • Stretch cluster fails when the VSAN policy is reapplied

    This issue is observed on setups where the VSAN policy fails to apply in case of NSX EDGE VMs.

    Workaround: Disable reconfigure method of such VMs and restart workflow. Once the workflow is completed, enable the reconfigure method.

  • Unable to configure the fabric compute managers during the NSX-T workload creation

    POST : https://10.0.0.50/api/v1/fabric/compute-managers fails with the socket timeout when tried with NSX-T SDK client.

    Retry the workflow.

  • The "Failed to deploy NSX-T Controller" error message displays when NSX-T workload domain is created

    In of the tasks of NSX-T deployment the task name says "Deploy additional NSX-T Managers". But the progress message says "Failed to deploy NSX-T Controller". There are no controllers in the current release of NSX-T.

    Workaround: None

  • The stretch cluster specification validation fails

    This issue occurs when the hosts provided for the the stretch cluster expansion are from different network pools.

    Workaround: During the expansion of stretch cluster, provide hosts that are a part of the same network pool.

  • The certificate rotate operation fails during the installation phase

    The replacement of the certificate fails intermittently. The Apply certificate API of NSX_T returns the 500 http error code intermittently during the certificate replacement.

    Workaround: Retry the certificate replacement operation by using the same or the different CA.

  • Deletion of additional domain does not delete the NSXT vibs

    This issue is intermittent. You cannot re-purpose the same host for the workload domain creation as it has the older vibs.

    Workaround: Clean up the NSX-T vibs manually. Keep retrying the removal task for the NSX-Tvibs till it succeeds.

  • The host entries are still present on the  NSX Manager transport nodes list even after the successful removal of the host from the cluster.

    On the NSX Manager, under System->Fabric->Transport nodes list, the deleted ESXi hosts show up.

    Workaround: Clean up the entries manually through the UI.

  • The add host process fails while adding a host in the NSX-T workload domain if you have created a port group a name of your choice

    When one creates a port group in DVS that does not have any uplink for the traffic communication, then there is an issue in determining the corresponding segment during the add host workflow. Hence the workflow fails.

    Workaround: Delete the user-created port groups from DVS and rerun the failed workflow.

  • The NSX-T deployment fails at the task of deploying the second NSX-Manager

    The NSX-T deployment fails with the "The specified static ip address '<ip_address>' is already in use by another machine" error.

    Workaround: Manually delete the second NSX manager and retry the task.

  • VMware Cloud Foundation workflows fail due to missing images

    After restore of the SDDC Manager VM, there are no bundles/images in the VM. Due to that, VMware Cloud Foundation workflows that consume the images will fail.

    Workaround: Download the NSX-T bundle again from SDDC Manager.

  • The SoS clean-up fails in cleaning up the NSX-T consumed hosts

    The issue occurs in removing the NSX-T related vib from the ESXi hosts. Hence, the SOS cleanup fails at the network clean up stage.

    Workaround: Refer the step 4 in the Remove a Host From NSX-T or Uninstall NSX-T Completely section in the VMware NSX-T Installation Guide.

  • After the certificate rotation, the Horizon workload domain creation or expansion fails at the create local user operation in Platform Services Controller

    After the certificate rotation, the Horizon workload domain creation or expansion fails at the create local user operation in the Platform Services Controller task.

    Workaround: Log in to SDDC Manager and restart the solution manager service.

  • The VI workload domain restart task fails

    The VI domain creation failed with the "Image with product type vCenter with version 6.7.0-13010631 and image type INSTALL is not found" error. When you upload the install bundle and restart the task., the restart process also fails with the same error.

    Workaround: Start a new VI domain creation task.

  • The workload domain creation and/or cluster addition may fail at the NSX host preparation phase

    If the ESXi hosts that are used to create a workload domain or cluster have not been fully cleaned or are reporting LiveInstallationError for any reason, the workload domain creation and/or cluster addition may fail at the NSX host preparation because EAM is not able to install the NSX VIBs on the ESXi hosts.

    Workaround: Reboot the hosts, ensure that EAM does not show LiveInstallationError and restart the workflow from the UI.

Security Operations Known Issues
  • Updating password policy failure results in UPDATE message when should be FAILED

    If the password policy fails, the system shows an UPDATE status and the transaction history shows the message "Operation failed in 'appliance update', for  credential update." In actuality, the operation has FAILED because the new password does meet requirements. A more appropriate message would read "Password update has failed due to unmet policy requirements" and recommend reviewing the policy.

    Workaround: Review the password policy for the component in question and modify the password configuration as necessary, and try again to update.

  • SSL Certificate Replacement for vCenter breaks vRealize Operations data collection

    After replacing the certificate for the vCenter Server component, both the vCenter and vSAN components in vRealize Operations Manager report a "Collection failed" error message. Testing the connection and attempting to accept the new certificate returns additional error messages: Unable to establish a valid connection to the target system. Adapter instance has been configured to trust multiple certificates, when only one is allowed. Please remove any old, unneeded certificates and try again.

    Workaround: If you encounter this issue, use the following procedure to resolve the situation.

    1. Delete the current vCenter and vSAN adapters.
    2. Re-create them using the same configuration and credentials originally set by vRealize Operations.
    3. Test the connection, accept the new certificates, and save the configuration.
  • Unable to perform password management operations from the SDDC Manager Dashboard

    If the Cloud Foundation Operations Manager component is restarted while a password management operation is in progress, the password management operation ends up in an INCONSISTENT state. You will not be able to perform any password management operations until you cancel the INCONSISTENT operation.

    1.  SSH into the SDDC Manager VM as the vcf user.
    2. Type su to switch to the root account.
    3. Run curl http://localhost/security/password/vault/transactions | json pp
      This returns information about the INCONSISTENT operation, For example:
      [
      {
         "transactionStatus" : "INCONSISTENT",
         "workflowId" : "f6548e76-f3e9-4033-801d-36ccae893672",
         "transactions" : [
            {
               "oldPassword" : "AX1are276!",
               "username" : "administrator@vsphere.local",
               "entityName" : "psc-1.vrack.vsphere.local",
               "id" : 123,
               "newPassword" : "x%5N6H1A^p%wJ4N",
               "entityType" : "PSC",
               "credentialType" : "SSO",
               "workflowId" : "0daaac30-c88d-4407-8bf3-c791541ebbae",
               "transactionStatus" : "INCONSISTENT",
               "timestamp" : "2019-04-09T09:00:29.628+0000",
               "transactions" : [
                 
               ]
            }
         ],
         "type" : "ROTATE",
         "id" : 1
      }
      ]
    4. Using the ID of the INCONSISTENT transaction ("id" : 1 in the example above), run the following:
      curl -X DELETE http://localhost/security/password/vault/transactions/<ID> | json pp
      This returns something like the following:
      {
          "transactionId":1,
          "workflowId":"f6548e76-f3e9-4033-801d-36ccae893672",
          "status":"USER_CANCELLED"
      }
Known Issues Affecting Service Providers
  • Domain manager workflows fail when using SDDC Manager to manage an API-created cluster or domain.

    If you have a cluster or domain that was created through the API, and you try to manage it through the SDDC Manager dashboard, the workflow will fail. This affects the following domain manager workflows: Add/Remove Host, Add/Remove VI Workload Domain, and Add/Remove Cluster.

    Workaround: None. Clean up the failed workflow and try again using the API.