check-circle-line exclamation-circle-line close-line

VMware Validated Design for Software-Defined Data Center 5.0 Release Notes

VMware Validated Design for Software-Defined Data Center 5.0 | 22 JAN 2019
Updated VMware Cloud Builder 1.0.0.1 | 21 MAR 2019 | BUILD 12915705

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

About VMware Validated Design for Software-Defined Data Center 5.0

VMware Validated Design provides a set of prescriptive documents that explain how to plan, deploy, and configure a Software-Defined Data Center (SDDC). The architecture, the detailed design, and the deployment guides provide instructions about configuring a dual-region SDDC.

VMware Validated Design is tested by VMware to ensure that all components and their individual versions work together, scale, and perform as expected. Unlike Reference Architectures which focus on an individual product or purpose, a VMware Validated Design is a holistic approach to design, encompassing many products in a full stack for a broad set of use case scenarios in an SDDC.

This VMware Validated Design supports a number of use cases, and is optimized for integration, expansion, Day-2 operations, as well as future upgrades and updates. As new products are introduced, and new versions of existing products are released, VMware continues to qualify the cross-compatibility and upgrade paths of VMware Validated Design. Designing with a VMware Validated Design ensures that future upgrade and expansion options are available and supported.

VMware Software Components in the Validated Design

VMware Validated Design for Software-Defined Data Center 5.0 is based on a set of individual VMware products with different versions that are available in a common downloadable package.

The products included in VMware Validated Designs participate in the VMware's Customer Experience Improvement Program ("CEIP"). Join the CEIP because this program provides us with information used to improve VMware products and services, fix problems, and advise you on how best to deploy and use our products.

Details regarding the data collected through CEIP and the purposes for which it is used by VMware are set forth at the Trust & Assurance Center at http://www.vmware.com/trustvmware/ceip.html. To join or leave the CEIP for the products that are part of VMware Validated Design, see the documentation for each product.

Before you deploy or upgrade the SDDC, review the release notes of the software components that are used in this validated design.

VMware Software Components in VMware Validated Design 5.0
Product Group and Edition Product Name Product Version
VMware Validated Design Updated VMware Cloud Builder 1.0.0.1
VMware vSphere Enterprise Plus ESXi 6.7 Update 1
vCenter Server Appliance 6.7 Update 1
vSphere Update Manager 6.7 Update 1
vSphere Replication 8.1.1
VMware vSAN Standard or higher vSAN 6.7 Update 1
VMware NSX Data Center Advanced or higher NSX Data Center for vSphere 6.4.4
VMware vRealize Suite Lifecycle Manager vRealize Suite Lifecycle Manager 2.0.0 Patch 2
VMware vRealize Operations Manager Advanced or higher vRealize Operations Manager 7.0
vRealize Operations Management Pack for NSX for vSphere 3.5.2
vRealize Operations Management Pack for Storage Devices 7.0.0
vRealize Operations Management Pack for Site Recovery Manager 8.1.1
VMware vRealize Log Insight vRealize Log Insight 4.7
vRealize Log Insight Content Pack for NSX for vSphere 3.8
vRealize Log Insight Content Pack for vRealize Automation 7.3+ 2.1
vRealize Log Insight Content Pack for vRealize Orchestrator 7.0.1+ 2.0
vRealize Log Insight Content Pack for vRealize Business 1.3
vRealize Log Insight Content Pack for Linux 1.0
vRealize Log Insight Content Pack for Site Recovery Manager 1.7
VMware vRealize Automation Advanced or higher vRealize Automation 7.5
VMware vRealize Business for Cloud Advanced vRealize Business for Cloud 7.5
VMware Site Recovery Manager Enterprise Site Recovery Manager 8.1.1

VMware makes available patches and releases to address critical security and functional issues for several products. Verify that you are using the latest security and express patches or hotfixes for a given component after deploying VMware Validated Design. Scalability and functionally tests for individual patches, express patches, or hotfixes are not typically performed against VMware Validated Design. If a patch must be applied to your environment, follow the VMware published practices and VMware Knowledge Base articles for the specific patch. If an issue occurs during or after the process of applying a patch, contact VMware Technical Support.

VMware Solution Exchange and in-product marketplace store only the latest versions of the management packs for vRealize Operations Manager and the content packs for vRealize Log Insight. The software components table contains the latest versions of the packs that were available at the time this VMware Validated Design was validated. When you deploy the VMware Validated Design components, it is possible that the version of a management or content pack on VMware Solution Exchange and in-product marketplace is newer than the one used for this release.

For information on the lifecycle of the VMware software components in this VMware Validated Design, see Lifecycle of the VMware Software Components.

What's New

VMware Validated Design for Software-Defined Data Center 5.0 provides a list of new features:

  • Updated Bill of Materials that incorporates new product versions

  • Automated SDDC deployment by using VMware Cloud Builder

    The manual deployment guidance is now replaced with time- and effort- saving automated deployment of the SDDC. You prepare the environment, deploy a Cloud Builder instance, download the software bundle for VMware Validated Design, and input the environment specification to Cloud Builder. You perform only a minimal set of post-deployment tasks.

    For multiple availability zones in Region A, VMware Validated Design supports manual addition of the second availability zone.

  • Support for Microsoft Windows Server 2016 for the Windows virtual machines of vRealize Automation and Site Recovery Manager

  • Support for for Microsoft SQL Server 2017 for the external database of vRealize Automation.

  • Support for Ubuntu Server 18.04 for the virtual machines of vSphere Update Manager Download Service.

  • Multi-tenancy support in vRealize Orchestrator

  • Recommended use of 25 GbE NICs

  • Use of the HTML5-based vSphere Client for environment preparation, post-deployment configuration, and operational guidance in vSphere, NSX for vSphere, Site Recovery Manager, and vSphere Update Manager

  • New Optimized upgrade sequence

    For an efficient upgrade, the operations management layer is upgraded before the cloud management layer, and the upgrade of Site Recovery Manager and vSphere Replication is decoupled from the upgrade of Platform Services Controller and vCenter Server for both Region A and Region B.

  • New Validation in VMware Cloud Builder of the forward and reverse DNS resolution for the vSphere Update Manager Download Service nodes.

For more information, see the VMware Validated Design Documentation page.

Internationalization

This VMware Validated Design release is available only in English.

Compatibility

This VMware Validated Design guarantees that product versions in the VMware Validated Design for Software-Defined Data Center 5.0, and the design chosen, are fully compatible. Any minor known issues that exist are described in this release notes document.

Installation

To install and configure an SDDC according to this validated design, follow the guidance in the VMware Validated Design 5.0 documentation. For product download information, and guides access, see the VMware Validated Design Documentation page.

New Lifecycle of the VMware Software Components

This VMware Validated Design version is based on one or more VMware products whose versions eventually reach the End of Support Life (EOSL) stage as described by the VMware Lifecycle Policies.Тhose versions are no longer generally supported by VMware. In such a case, upgrade to a later version by using the upgrade procedures in the VMware Validated Design Upgrade documentation.

If you are using an earlier version in your environment, upgrade your environment according to the following scenarios:

Scenarios for Upgrade from a Version that Has Reached EOSL
Scenario Upgrade Approach
The version of VMware Validated Design that you are using has already entered the EOSL stage but the next VMware Validated Design version is still supported. Apply the VMware Validated Design Upgrade documentation to bring the VMware environment to a fully supported state
The version of VMware Validated Design that you are using and the next version have both already entered the EOSL stage Because the VMware Validated Design Upgrade documentation supports upgrade only from one release to the next one, the transition across multiple releases might be complex. Contact a VMware sales representative to plan and perform a custom upgrade procedure with the assistance of VMware Professional Services.

For more information about current and expired product releases, refer to the VMware Lifecycle Product Matrix.

Caveats and Limitations

Documentation Delivery Log

The VMware Validated Design documentation is published in several iterations. On the release date, the core documents that introduce VMware Validated Design and provide guidance on the SDDC design, planning, and deployment are published live. The remaining documents are released in groups until the entire set is compliant with the Bill of Materials of the VMware Validated Design release.

Log of Delivered VMware Validated Design Documentation
Delivery Date Documentation
22 JAN 2019
  • Introducing VMware Validated Designs
  • Standard SDDC
    • Architecture and Design
    • Planning and Preparation
    • Deployment of Region A
    • Deployment of Region B
    • Deployment of Multiple Availability Zones
5 MAR 2019
  • Standard SDDC
    • Upgrade
    • Operational Verification
19 MAR 2019
  • Consolidated SDDC
    • Architecture and Design
    • Planning and Preparation
2 APR 2019
  • Consolidated SDDC
    • Deployment

All documentation is available on the VMware Validated Design Documentation page. For details on the latest available guidance, see Documentation Map for VMware Validated Design.

Resolved Issues

The resolved issues are grouped as follows.

VMware Cloud Builder

New The following issues are resolved in VMware Cloud Builder 1.0.0.1.

  • An attempt to deploy an SSDC fails during the pre-deployment checks due to missing forward and reverse DNS records for the NSX Controllers

    Although DNS reverse records for the NSX Controllers are not required, the pre-deployment check tries to validate whether they exist. The pre-deployment check finishes with an error and the installation fails.

  • An attempt to deploy the SDDC fails at the step for configuring VM priorities in the recovery plans in Site Recovery Manager

    In the user interface of VMware Cloud Builder, you see that task Configure VM priorities in SRM recovery plans has failed.

    The following error message appears in the user interface:

    Could not set VM priority

    The /opt/vmware/bringup/logs/vcf-bringup.log log file in the Cloud Builder virtual appliance contains the following error stack trace:

    2019-01-21T07:46:02.797+0000
       ERROR [0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,threadPoolExecutor-4] [SP4J8B]
       CONFIGURE_VM_PRIORITIES_FAILED Configuration for VM priorities for SRM 172.16.11.124 failed
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Configuration for VM priorities for SRM 172.16.11.124 failed
           at com.vmware.evo.sddc.common.fsmplugins.srm.impl.ConfigureVmPriorities.execute(ConfigureVmPriorities.java:59)

    . . .

    Caused by: com.vmware.evo.sddc.common.services.srm.api.SrmConfigurationException:
      Could not set VM priority [VmPriorityMapping(vmName=vra01ims01b, priority=60),
    VmPriorityMapping(vmName=vra01svr01a, priority=40), VmPriorityMapping(vmName=vrb01svr01, priority=75),
       VmPriorityMapping(vmName=vra01iws01a, priority=50),
    VmPriorityMapping(vmName=vra01dem01a, priority=75), VmPriorityMapping(vmName=vra01iws01b, priority=50),
       VmPriorityMapping(vmName=vra01dem01b, priority=75),
    VmPriorityMapping(vmName=vra01svr01c, priority=40), VmPriorityMapping(vmName=vra01ims01a, priority=60),
       VmPriorityMapping(vmName=vra01svr01b, priority=40)] for recovery plan SDDC Cloud Management RP
           at com.vmware.evo.sddc.common.services.srm.instances.SrmInstance.setVmPriority(SrmInstance.java:382)
           at com.vmware.evo.sddc.common.services.srm.impl.SrmServiceImpl.setVmPriorityInRecoveryPlan(SrmServiceImpl.java:285)
           at com.vmware.evo.sddc.common.fsmplugins.srm.impl.ConfigureVmPriorities.configureVmPriorities(ConfigureVmPriorities.java:78)
           at com.vmware.evo.sddc.common.fsmplugins.srm.impl.ConfigureVmPriorities.execute(ConfigureVmPriorities.java:54) com.vmware.evo.sddc.common.fsmplugins.srm.impl.ConfigureVmPriorities.configureVmPriorities(ConfigureVmPriorities.java:78) at com.vmware.evo.sddc.common.fsmplugins.srm.impl.ConfigureVmPriorities.execute(ConfigureVmPriorities.java:54)

    . . .

    Caused by: java.lang.RuntimeException:
       Retriable operation 'Validate configuring VM recovery priority for vra01dem01a' failed to complete after 5 retries.
           at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:183)
           at com.vmware.evo.sddc.common.services.srm.instances.SrmInstance.setVmPriority(SrmInstance.java:378)

    . . .

    Caused by: com.vmware.evo.sddc.common.services.srm.api.SrmConfigurationException: VM priority is not configured for vra01dem01a
           at com.vmware.evo.sddc.common.services.srm.instances.SrmInstance$1.call(SrmInstance.java:368)

    . . .

  • After you deploy an SDDC, the event forwarding filter does not work as expected for the vRealize Log Insight instances in both Region A and Region B

    In the event forwarding configuration in Region A for the "sfo01 to lax01" destination, the filter value is *LAX01* instead of LAX01. In Region B, the value is *SFO01* instead of SFO01. As a result, the vRealize Log Insight instance in Region A filters out any site with a name that contains LAX01 and the vRealize Log Insight instance in Region B filters out sites with names that contain SFO01.

    For example, if the name of your site is LAX01-south, by using the site does not match *LAX01*  event forwarding filter in Region A, the vRealize Log Insight instance filters out both the LAX01 and the LAX01-south sites.

  • During the automated deployment of an SDDC, Cloud Builder cannot obtain the Active Directory forest name

    In Active Directory environments with more than one forest, Cloud Builder cannot obtain the Active Directory forest name and fails to register the Active Directory domain as an identity source in the Platform Services Controller.
    As a result, the Cloud Builder task Configure Active Directory domain as Identity Source in Platform Services Controller fails.

  • An attempt to deploy the Consolidated SDDC without including vRealize Operations Manager results in a deployment failure

    On the Run Parameters tab of the deployment parameters spreadsheet for the Consolidated SDDC, deselect vRealize Operations Manager. Then, start the deployment of the Consolidated SDDC. The deployment fails immediately.

  • After validation of a JSON deployment file, the Back button is disabled in the VMware Cloud Builder user interface

    On the Validate Environment tab in VMware Cloud Builder, after you perform a validation of a JSON deployment file, you cannot select a second JSON file for validation, because the Back button is disabled. .

  • VMware Cloud Builder generates a single Classless Inter-Domain Routing (CIDR) notation for the network configuration of the NFS storage for both the management cluster and the shared edge and compute cluster

    When generating the JSON deployment files for the management and the shared edge and compute clusters, VMware Cloud Builder generates a single CIDR notation for the two distinct NFS storage subnets. 

  • VMware Cloud Builder creates environments in vRealize Suite Lifecycle Manager although you deploy the SDDC without the respective vRealize Suite products

    On the Run Parameters tab of the deployment parameters spreadsheets for the Standard SDDC or Consolidated SDDC, deselect both vRealize Operations Manager and vRealize Automation. Then, start the deployment of the SDDC for the region. After the deployment finishes, vRealize Suite Lifecycle Manager contains the cross-region environment although Cloud Builder does not deploy the products for it. Similarly, if you deploy the SDDC without vRealize Log Insight, vRealize Suite Lifecycle Manager contains the region-specific environment.

Known Issues

The known issues are grouped as follows.

VMware Validated Design Content
  • New On the NSX load balancers for Standard SDDC and Consolidated SDDC,  the number of allowed simultaneous sessions to an analytics node in vRealize Operations Manager is lower than the supported maximum

    This version of VMware Validated Design provides the following guidance on the maximum number of simultaneous sessions to an analytics node in vRealize Operations Manager that is lower than the number that is supported in vRealize Operations Manager 7.0:

    • After you deploy or upgrade the SDDC, on the load balancers in the cross-region application virtual networks, the configuration of the vrops-svr-443 pool supports up to eight maximum simultaneous sessions to an analytics node.
    • The VMware Validated Design Architecture and Design and VMware Validated Design Architecture and Design for Consolidated SDDC deliverables provide guidance on a maximum of four simultaneous sessions per node.

    In vRealize Operations Manager 7.0, an analytics node can support up to 10 simultaneous session if the environment is operating close to its limits. As a result, by following the guidance that is available in this version of VMware Validated Design, users might be unable to connect to the operations interface of vRealize Operations Manager although the system can support these connections.

    Workaround: On the sfo01m01lb01 and lax01m01lb01 load balancers for Standard SDDC or on the sfo01w01lb01 load balancer for Consolidated SDDC, change the maximum number of simultaneous sessions per analytics node in vRealize Operations Manager.

    1. Log in to vCenter Server by using the vSphere Web Client.
      1. Open a Web browser and go to the following URL.
        SDDC Architecture vCenter Server URL
        Standard SDDC https://sfo01m01vc01.sfo01.rainpole.local/vsphere-client
        Consolidated SDDC https://sfo01w01vc01.sfo01.rainpole.local/vsphere-client
      2. Log in by using the following credentials.
        Setting Value
        User name administrator@vsphere.local
        Password vsphere_admin_password
    2. From the Home menu of the vSphere Web Client, select Networking & Security.
    3. In the Navigator pane, select NSX Edges.
    4. Open the network settings of the NSX load balancer.
      • For Region A for Standard SDDC, from the NSX Manager drop-down menu, select 172.16.11.65, and double-click the sfo01m01lb01 NSX Edge node.
      • For Region B for Standard SDDC, from the NSX Manager drop-down menu, select 172.17.11.65, and double-click the lax01m01lb01 NSX Edge node.
      • For Consolidated SDDC, from the NSX Manager drop-down menu, select 172.16.11.66, and double-click the sfo01w01lb01 NSX Edge node.
    5. On the Manage tab for the NSX Edge node, click the Load Balancer tab.
    6. Select Pools, select the vrops-svr-443 pool, and click the Edit icon.
      The Edit Pool dialog box appears.
    7. Increase the maximum number of simultaneous sessions to the analytics nodes.
      1. In the Members pane, select the vrops01svr01a analytics node and click the Edit icon.
      2. Set the Max Connections property to 10 and click OK.
      3. For Standard SDDC, repeat the previous steps for the vrops01svr01b and vrops01svr01c analytics nodes.
    8. In the Edit Pool dialog box, click OK.
vSphere
  • vRealize Automation converged blueprint provisioning fails with error: CloneVM : [CloneVM_Task] - A general system error occurred: vDS host error: see faultCause

    vRealize Automation converged blueprint provisioning fails because an attempt to perform a networking configuration operation on a vSphere Distributed Switch, such as creating a virtual machine adapter or a port group, causes the vSphere host to disconnect from the vCenter Server and results in the error message:

    Transaction has rolled back on the host.

    Workaround:

    Increase the network rollback timeout of vCenter Server from 30 to 60 seconds.
    See Networking Configuration Operation Is Rolled Back and a Host Is Disconnected from vCenter Server in the vSphere Troubleshooting documentation. 

  • You cannot upgrade VMware Tools on the Windows virtual machines because the verification for an operating system update fails

    When you try to upgrade VMware Tools to 10.3.x on the Windows guest operating system, you receive the following error message:

    Setup cannot verify that the required Microsoft update KB2919355 is installed.

    Workaround: Apply the latest service pack for Windows 2012 R2. See VMware KB 55798

  • New After you upgrade ESXi hosts using by vSphere Update Manager, one or more hosts might be in maintenance mode and the remaining ESXi hosts are not upgraded

    When you upgrade the ESXi hosts from version 6.5 Update 2 to 6.7 Update 1 by using vSphere Update Manager, one or more ESXi hosts might remain in maintenance mode after the upgrade process finishes. This causes vSphere Update Manager to skip the upgrade on the remaining hosts in the cluster and the Remediate entity task shows the following error message: 

    There are errors during the remediation operation. Check the events and log files for details

    Workaround: Exit maintenance mode for the ESXi hosts and remediate the remaining ESXi hosts that are not yet upgraded.

    1.  Exit maintenance mode for the ESXi hosts that are in maintenance mode after the upgrade process is completed.
      1. Open a Web browser and go to https://sfo01m01vc01.sfo01.rainpole.local/vsphere-client.
      2. Log in by using the following credentials.
        Setting Value
        User name administrator@vsphere.local
        Password

        vsphere_admin_password
      3. From the Home menu of vSphere Web Client, click Hosts and Clusters, and expand the sfo01m01vc01.sfo01.rainpole.local tree. 
      4. Right-click the host in maintenance mode and select Maintenance mode > Exit Maintenance Mode.
      5. Repeat this step for all ESXi hosts that are still in maintenance mode. 
    2. From the Home menu of vSphere Web Client, click Update Manager. 
    3. Remediate the ESXi hosts that are not upgraded by following Step 6 from the Use vSphere Update Manager to Upgrade the ESXi Hosts in the Management Cluster in Region A procedure. On the Select Target Objects page of the Remediate wizard, select all ESXi hosts that are not upgraded. 
vRealize Operations Manager
  • After you perform a failover operation of the vRealize Operations Manager or Cloud Management Platform virtual machines, the vRealize Automation Adapter might fail to collect statistics

    This issue might occur during both failover to Region B and failback to Region A of the Cloud Management Platform or the vRealize Operations Manager analytics cluster.

    After you perform disaster recovery or planned migration of the Cloud Management Platform or virtual machines of the vRealize Operations Manager analytics cluster, the collection state of the vRealize Automation Adapter cannot collect statistics. When you log in to the operations interface of vRealize Operations Manager at https://vrops01svr01.rainpole.local and navigate to the Solutions page, the collection state of the vRealize Automation Adapter is Failed.

    Workaround: To manually restart data collection in the vRealize Automation Adapter, click the Stop Collecting button and click the Start Collecting button.

  • After you perform a failover operation, the vRealize Operations Manager analytics cluster might fail to start because of an NTP time drift between the nodes
    • The vRealize Operations Manager operations interface might report that some of the analytics nodes are not coming online with the status message Waiting for Analytics.
    • The log information on the vRealize Operations Manager master or master replica node might contain certain NTP-related details.
      • The NTP logs in the /var/log/ folder might report the following messages:
        ntpd[9764]: no reply; clock not set
        ntpd[9798]: ntpd exiting on signal 15
      • The analytics-wrapper.log file in the /storage/log/vcrops/logs/ folder might report the following message:
        INFO | jvm 1 | YYYY/MM/DD | >>> AnalyticsMain.run failed with error: IllegalStateException: time difference between servers is 37110 ms. It is greater than 30000 ms. Unable to operate, terminating...

         

    Workaround: See VMware Knowledge Base article 2151266.

vRealize Automation and Embedded vRealize Orchestrator
  • Manual installation of an IaaS Website component using the IaaS legacy GUI installer fails with a certificate validation error

    The error message appears when you click Next on the IaaS Server Custom Install page with the Website component selected. This error message is a false negative and appears even when you select the right option. The error prevents the installation of a vRealize Automation IaaS Website component.

    Workaround: See Knowledge Base article 2150645.

  • Unable to log in to the vRealize Automation user interface after configuring a non-existing tenant as the authentication provider for the embedded vRealize Orchestrator.

    The vRealize Automation user interface becomes unavailable after you configure the authentication settings on the Configure Authentication Provider page in the embedded vRealize Orchestrator Control Center with a non-existing tenant. For example, if you enter a tenant name with a typo.

    On the Services tab at https://vra01svr01a.rainpole.local:5480, you see the following services as unavailable:

    Service State
    advanced-designer-service UNAVAILABLE
    o11n-gateway-service UNAVAILABLE
    shell-ui-app UNAVAILABLE
    vco null

    Workaround: Correct the tenant details and verify the service state on the vRealize Automation appliances.

    1. Log in to the vRealize Orchestrator Control Center. 
      1. Open a Web browser and go to https://vra01svr01.rainpole.local:8283/vco-controlcenter.
      2. Log in by using the following credentials. 
        Setting Value
        User name root
        Password deployment_admin_password
    2. On the Configure Authentication Provider page, update the authentication configuration with the correct tenant details. 
      The control center replicates the settings to all vRealize Orchestrator servers in the cluster after several minutes.
    3. Log in to the first vRealize Automation appliance.
      1. Open a Web browser and go to  https://vra01svr01a.rainpole.local:5480.
      2. Log in by using the following credentials. 
        Setting Value
        User name root
        Password deployment_admin_password
    4. On the Services tab, verify that the status of all services is REGISTERED.
    5. Repeat Step 4 and Step 5 on the other vRealize Automation appliances.
  • Converged blueprint provisioning requests in vRealize Automation might fail in environments that have high workload churn rate

    In environments that have a high curn rate for tenant workloads, requests for provisioning converged blueprints in vRealize Automation might fail with one of the following error messages.

    • Timeout Customizing machine

    Workaround: None.

  • After you perform disaster recovery of the Cloud Management Platform, the status of the shell-ui-app service might appear as Failed in the appliance management console of the vra01svr01b.rainpole.local node

    This issue might occur during both failover to Region B and failback to Region A of the Cloud Management Platform. After you perform disaster recovery of the Cloud Management Platform, you see the following symptoms when you verify the overall state of the platform:

    • In the appliance management console https://vra01svr01b.rainpole.local:5480, the status of the shell-ui-app service is Failed.
    • The statistics about the vra-svr-443 pool on the NSX load balancer shows that the vra01svr01b node is DOWN.
    • Trying to access the https://vra01svr01b.rainpole.local/vcac/services/api/health URL results with following error message:

      The service shell-ui-app was not able to register the service information with the Component Registry service! This might cause other dependent services to fail. Error Message: I/O error on POST request for "https://vra01svr01.rainpole.local:443/SAAS/t/vsphere.local/auth/oauthtoken?grant_type=client_credentials": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out"

    You can still log in to the vRealize Automation portal because the other vRealize Automation Appliance vra01svr01a can service your requests.

    Workaround: Restart the vcac-server service on the vra01svr01b.rainpole.local node.

    1. Open an SSH connection to the vra01svr01b.rainpole.local appliance and log in as root.
    2. Restart the vcac-server service by running the following command.
      service vcac-server restart
  • After failover or failback during disaster recovery, the login to the vRealize Automation Rainpole portal takes several minutes or fails with an error message

    This issue occurs during both failover to Region B and failback to Region A of the Cloud Management Platform when the root Active Directory is not available in the protected region. You see the following symptoms:

    • Login takes several minutes.

      When you log in to the vRealize Automation Rainpole portal at https://vra01svr01.rainpole.local/vcac/org/rainpole by using the ITAC-TenantAdmin user, the vRealize Automation portal loads after 2 to 5 minutes.

    • Login fails with an error.

      An attempt to log in to the vRealize Automation Rainpole portal fails with an error about incorrect user name and password.

    Workaround: Perform one of the following workarounds according to the type of the recovery operation.

    • Failover to Region B
      1. Open an SSH connection to the vra01svr01a.rainpole.local appliance and log in as root.
      2. Open the /usr/local/horizon/conf/domain_krb.properties file in a text editor.
      3. Add the following list of the domain-to-host values and save the domain_krb.properties file.
        Use only lowercase characters when you type the domain name.
        For example, as you have performed failover, you must map the rainpole.local domain to the controller in Region B.
        rainpole.local=dc51rpl.rainpole.local:389
      4. Change the ownership of the domain_krb.properties.
        chown horizon:www /usr/local/horizon/conf/domain_krb.properties
      5. Open the /etc/krb5.conf file in a text editor.
      6. Update the realms section of the krb5.conf file with the same domain-to-host values that you configued in the domain_krb.properties file, but omit the port number as shown in the following example.
        [realms]
        RAINPOLE.LOCAL = {
          auth_to_local = RULE:[1:$0\$1](^RAINPOLE\.LOCAL\\.*)s/^RAINPOLE\.LOCAL/RAINPOLE/
          auth_to_local = RULE:[1:$0\$1](^RAINPOLE\.LOCAL\\.*)s/^RAINPOLE\.LOCAL/RAINPOLE/
          auth_to_local = RULE:[1:$0\$1](^SFO01\.RAINPOLE\.LOCAL\\.*)s/^SFO01\.RAINPOLE\.LOCAL/SFO01/
          auth_to_local = RULE:[1:$0\$1](^LAX01\.RAINPOLE\.LOCAL\\.*)s/^LAX01\.RAINPOLE\.LOCAL/LAX01/
          auth_to_local = DEFAULT
          kdc = dc51rpl.rainpole.local
        }
      7. Restart the workspace service.
        service horizon-workspace restart
      8. Repeat this procedure on the vra01svr01b.rainpole.local and vra01svr01c.rainpole.local vRealize Automation appliances.
    • Failback to Region A
      ​If dc51rpl.rainpole.local becomes unavailable in Region B during failback, perform the steps for the failover case using dc01rpl.rainpole.local as the domain controller instead of dc51rpl.rainpole.local and restart the services.

    This workaround optimizes the synchronization with Active Directory by pointing to a specific domain controller that the vRealize Automation appliance can access if a disaster recovery operation occurs.

VMware Cloud Builder
  • After you refresh the browser on the Validation menu or Deploy menu, the In-Progress status is lost.

    The In-Progress status for validations or deployments is lost when you refresh the browser.

    Workaround: Click the Run Results button to restore the In-Progress status screen.

  • An attempt to deploy the SDDC fails at the step for integration of vRealize Business with vRealize Automation

    In the user interface of VMware Cloud Builder at https://sfo01cb01.sfo01.rainpole.local, you see that the Integrate vRealize Business for Cloud with vRealize Automation task has failed.

    The following error message appears on the Registration > vRA tab in the management console of the vRealize Business server appliance at https://vrb01svr01.rainpole.local:5480:

    Error starting ITBM data collector.

    The /opt/vmware/bringup/logs/vcf-bringup.log log file in the Cloud Builder appliance contains the following error stack trace:

    2019-01-17T21:07:52.503+0000 ERROR [0000000000000000,0000,operationToken=<20a0b100-1a3d-11e9-b244-49b4d2e5652c>] [c.v.v.c.v.v.s.i.VrbConfigurationServiceImpl,threadPoolExecutor-3] Status code was fTBM data collector.
    2019-01-17T21:07:52.503+0000 ERROR [0000000000000000,0000] [c.v.e.s.o.model.error.ErrorFactory,threadPoolExecutor-3] [EDUC0K] VRB_VRA_INTEGRATION_FAILURE Failed to integrate vRB 192.168.11.66 in vRA
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException:
      Failed to integrate vRB 192.168.11.66 in vRA vra01svr01.rainpole.local
            at com.vmware.evo.sddc.vrealize.vrb.VrbVraIntegration.execute(VrbVraIntegration.java:75)
            at com.vmware.evo.sddc.vrealize.vrb.VrbVraIntegration.execute(VrbVraIntegration.java:36)

    . . .
    Caused by: com.vmware.vcf.common.vrealize.vrb.service.impl.VrbApiException: Status code was failure. Status message: Error starting ITBM data collector.
            at com.vmware.vcf.common.vrealize.vrb.service.impl.VrbConfigurationServiceImpl.logApiStatusCodeMessageThrowException
              (VrbConfigurationServiceImpl.java:236)
    . . .

    Workaround:

    1. Log in to vCenter Server by using the vSphere Client.
      1. Open a Web browser and go to https://sfo01m01vc01.sfo01.rainpole.local/ui.
      2. Log in by using the following credentials.
        Setting Value
        User name administrator@vsphere.local
        Password vsphere_admin_password
    2. In the Navigator, click Hosts and Clusters.
    3. In the inventory, expand the entire sfo01m01vc01.sfo01.rainpole.local tree.
    4. Right-click the vrb01svr01 virtual machine and select Power > Restart Guest OS.
    5. After the virtual machine is powered on, retry the deployment in Cloud Builder.
  • An attempt to deploy the SDDC fails at the step for adding compute resources to the vRealize Automation fabric groups

    In the user interface of VMware Cloud Builder, you see that task Add Compute Resources to vRealize Automation Fabric Groups has failed.

    The following error message appears in the user interface: 

    Failed to add compute resources to fabric groups java.util.stream.ReferencePipeline$3@5e62dc99, for vRA vra01svr01.rainpole.local

    The /opt/vmware/bringup/logs/vcf-bringup.log log file in the Cloud Builder virtual appliance contains the following error stack trace:
    2019-01-18T11:44:54.616+0000 WARN [0000000000000000,0000,operationToken=<4f8ffc50-1b0a-11e9-b244-49b4d2e5652c>] [c.v.e.s.v.v.AddComputeResourcesToFabricGroup,threadPoolExecutor-4] No compute resources found, retrying...
    2019-01-18T11:44:54.617+0000 ERROR [0000000000000000,0000,operationToken=<4f8ffc50-1b0a-11e9-b244-49b4d2e5652c>] [c.v.e.s.v.v.AddComputeResourcesToFabricGroup,threadPoolExecutor-4]
      Failed to add compute resources to fabric groups java.util.stream.ReferencePipeline$3@5e62dc99, for vRA vra01svr01.rainpole.local.
    java.lang.RuntimeException: Retriable operation 'List compute resources' failed to complete after 6 retries.
            at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:183)
            at com.vmware.evo.sddc.vrealize.vra.AddComputeResourcesToFabricGroup.execute(AddComputeResourcesToFabricGroup.java:96)
    . . .
    Caused by: java.lang.IllegalStateException: No compute resources found, retrying...
            at com.vmware.evo.sddc.vrealize.vra.AddComputeResourcesToFabricGroup.lambda$execute$0(AddComputeResourcesToFabricGroup.java:88)
            at com.vmware.evo.sddc.common.util.RetriableCallable.call(RetriableCallable.java:142)

    . . .

    Workaround:

    1. Log in to the vra01ims01a virtual machine of the vRealize Automation IaaS Manager Service by using a Remote Desktop Protocol (RDP) client.
      1. Open an RDP connection to vra01ims01a.rainpole.local.
      2. Log in by using the following credentials.
        Setting Value
        User name rainpole\svc-vra
        Password svc-vra_password 
    2. In the search box of the Windows Start menu, enter services.msc and press Enter.
    3. Right-click the Distributed Transaction Coordinator service, and select Restart.
    4. After the service is started, log in to the vRealize Automation Rainpole portal.
      1. Open a Web browser and go to https://vra01svr01.rainpole.local/vcac/org/rainpole.
      2. Log in using the following credentials.
        Setting Value
        User name vra-admin-rainpole
        Password vra-admin-rainpole_password​
        Domain rainpole.local
    5. On the Infrastructure tab, click Endpoints > Fabric Groups.
    6. Click sfo01-fabric-group, verify that the sfo01-w01-shared01 compute resource is available, and click OK.
    7. Click lax01-fabric-group, and verify that lax01-w01-shared01 is available.
    8. Retry the deployment in Cloud Builder.
  • An attempt to deploy the SDDC intermittently fails when deploying Site Recovery Manager

    In the user interface of VMware Cloud Builder, you see that task Deploy Site Recovery Manager has failed.

    The following error message appears in the user interface:

    Failure occurred while installing SRM on 192.168.110.124 windows server VM

    Workaround:

    1. Verify that the Windows Server virtual machine is set up according to the VMware Validated Design Planning and Preparation documentation, and retry the SDDC deployment in Cloud Builder to retry the deployment of Site Recovery Manager.
    2. If the retry deployment fails, verify that the Windows Server 2016 virtual machine for Site Recovery Manager is running the latest version of VMware Tools and is configured with the latest virtual hardware for vSphere.
      If updates are required, retry the deployment in Cloud Builder to retry the deployment of Site Recovery Manager.
    3. If the retry deployment fails after updating VMware Tools and virtual hardware, contact VMware Technical Support.
  • An attempt to deploy the SDDC management components in Region B might fail at the step for configuring the SDDC Health Adapter instances in vRealize Operations Manager

    In the user interface of VMware Cloud Builder, you might see that the task Configure Sddc Health Adapter in vRealize Operations Manager has failed.

    The following error message appears in the user interface:

    Failed to configure SDDC Health Adapter for collector lax01vropsc01a in vROps 192.168.11.35

    The /opt/vmware/bringup/logs/vcf-bringup.log log file in the Cloud Builder virtual appliance contains the following error stack trace:

    2019-01-18T07:02:25.474+0000 DEBUG [0000000000000000,0000,operationToken=<f3ea63f0-19aa-11e9-89e3-5315012a81bf>] [c.v.e.s.c.s.v.impl.VropsServiceImpl,threadPoolExecutor-2] Could not find adapter with name: SDDC Health Adapter Instance - lax01vropsc01a in vrops 192.168.11.35
    2019-01-18T07:02:25.474+0000 DEBUG [0000000000000000,0000,operationToken=<f3ea63f0-19aa-11e9-89e3-5315012a81bf>]

     

    Workaround:

    1. Log in to the operations interface of vRealize Operations Manager.
      1. Open a Web browser and go to https://vrops01svr01.rainpole.local.
        1. Log in by using the following credentials.
          Setting Value
          User name admin
          Password vrops_admin_password
    2. On the main navigation bar, click Administration and select Solutions.
    3. On the Solutions page, under the SDDC Management Health solution, delete the SDDC Health Adapter instances for Region B.
    4. In the user interface of Cloud Builder, retry the deployment in Region B.