VMware Cloud Director 10.4 Release Notes

VMware Cloud Director 10.4 \| 14 JUL 2022 \| Build 20079248 (installed build 20079017) Check for additions and updates to these release notes.

VMware Cloud Director 10.4 | 14 JUL 2022 | Build 20079248 (installed build 20079017)

Check for additions and updates to these release notes.

What's New

VMware Cloud Director version 10.4 includes the following:

Networking Features
Static Routing for NSX-T Data Center edge gateways. Both service providers and tenants can manually modify the routing table for an NSX-T edge gateway and to configure a static route within an organization VDC by specifying the network, next hop, and any specific networks or interfaces to which to direct traffic. If the specified next hop is part of an existing subnet that is associated with the edge gateway, you can specify a scope for the static route. A scope is an organization VDC network in which the next hop of the static route is located.

Use static routing if you want to use a route that is not included in the routing table of your NSX-T edge gateway, or if you want to direct traffic trough a service edge gateway and a service interface, and to not use distributed routing. See Configure Static Routing.

VMware NSX Advanced Load Balancer (Avi Networks) Feature set configuration at the service engine group level. All Controller instances that you register with VMware Cloud Director are now licensed and deployed with the Premium (also known as Enterprise) feature set. As a service provider, you can choose to restrict a tenant's feature set to Standard (Basic) at the Service Engine Group level.

VMware NSX Advanced Load Balancer (Avi Networks) Standard feature set improvements for tenants
- You can configure a single virtual service to use more than one port protocol.
- You can configure multiple virtual services that are on the same edge gateway to be fronted by the same virtual IP address. The virtual services must use different ports.
- You can use IPv6 for virtual IP addresses and for load balancer server pools.
- You can configure a virtual service on the same subnet as an organization VDC network.
Fast Cross vCenter Server vApp Instantiation Utilizing Shared Storage

This solution provides added flexibility for enabling fast instantiation of vApps from a catalog across vCenter Server instances. If a VMware Cloud Director catalog is backed by a VDC in vCenter Server A, instantiating a vApp from the catalog to a VDC backed by a cluster in vCenter Server B triggers a time-consuming OVF export-import process. In previous releases, the only way to avoid this was to ensure that the source and destination vCenter Server instances have shared storage, and the templates in the source vCenter Server reside on the same shared storage where they are instantiated on the destination vCenter Server. Such configuration allows optimized copy of the templates, avoiding OVF export-import, and as a result leads to a significantly faster vApp instantiation. Starting with version 10.4, if vCenter Server A and vCenter Server B have a shared storage and the templates reside there, VMware Cloud Director performs fast vApp instantiation to any destination datastore in vCenter Server B.
Photon OS 3.0

VMware Cloud Director appliance is now based on Photon OS 3.0 which brings improved security and upgraded OS packages.
Service Accounts

You can use VMware Cloud Director to create and manage service accounts for applications interacting with VMware Cloud Director. To provide continuous access to VMware Cloud Director, service accounts use OAuth refresh tokens instead of passwords. When using service accounts, applications cannot perform certain tasks and have only View rights for some resources, such as users, groups, roles, and rights. See Managing Service Accounts.
Consolidated VM Console Proxy Endpoint

VM console proxy traffic is now using the same IP and port as the VMware Cloud Director UI and API. In previous releases, the console proxy traffic used separate IP and port. The unified access point eliminates the need to manage an additional endpoint and certificate, and allows for SSL termination at the load balancer level. The VMware Cloud Director HTTP request log records the VM console traffic. You can enable the LegacyConsoleProxy feature, however, before enabling the legacy console proxy implementation, verify that your system configuration has the necessary console proxy settings and see After you enable legacy console proxy mode, the VMware Cloud Director cell on which you enabled the… and If you run the cell management tool's clear-console-proxy-settings command while legacy console pro….
Enhanced trust management integration with vSphere

VMware Cloud Director 10.4 enhances SSL connectivity to all vSphere infrastructure components, including ESXi, by incorporating the vSphere Certificate Authority (CA) into the VMware Cloud Director trust mechanisms which also affects previously added vCenter Server instances.

IMPORTANT: Because of this enhancement to SSL connectivity, you must perform additional steps post-upgrade to ensure that VMware Cloud Director trusts all necessary vSphere certificates. Failure to perform these steps post-upgrade can disrupt the connection between VMware Cloud Director and vCenter Server instances.

Follow the procedure outlined in the advisory that appears upon upgrade. See also KB 78885 and The VMware Cloud Director console proxy, uploading OVFs and media, and powering on a VM fail.
VMware Cloud Director appliance certificate backup and restore
1. In addition to HTTP and console proxy certificates, VMware Cloud Director Appliance backup includes a vcd_ova certificate and keys.
2. VMware Cloud Director Appliance management UI supports optionally restoring the PostgreSQL and virtual appliance management UI certificates.
IOPS Limits and Reservation Visibility Enhancements
- Service providers can show or hide from tenants the IOPS limits and reservations on VM disks and named disks by using the new View Disk IOPS right. See Enabling the I/O Operations Per Second Setting.
- VM disks and named disks views display both IOPS limits and reservations for each disk. VMware Cloud Director also returns the data in the corresponding API calls.
- IOPS settings configured on a non-vSAN storage policy in vCenter Server appear in VMware Cloud Director named disk and VM disk views and in the corresponding API responses. By using the VMware Cloud Director UI, you cannot edit IOPS reservations and limits configured in vCenter Server.
Catalog Enhancements
1. If a catalog is subscribed to a published catalog, the synchronization of templates from the published to the subscribed catalog goes through a number of distinct steps. VMware Cloud Director now provides a detailed view of the currently running catalog synchronization task step and the progress percentage of that step. The currently running task step appears in the Recent Tasks panel and a more detailed view appears if a user clicks the Status of the item being synced. Distinct steps examples are: initializing library item sync, waiting for publisher to download files, transferring files from publisher to subscriber, importing OVF.
2. When the catalog item synchronization task is at the Transferring files from publisher to subscriber step, clicking the task details shows the Active File Transfers tab where you can see the file transfer progress of individual disk files.
3. If a catalog synchronization fails in file transfer phase due to VMware Cloud Director service restart or crash, when the cell restarts, the sync resumes from the previous transfer session, and VMware Cloud Director reuses the previously transferred content.
4. If you deactivate the Automatically download the content from an external catalog setting of a subscribed catalog, at catalog creation, VMware Cloud Director does not sync the items within the catalog. VMware Cloud Director synchronizes only the item metadata. VMware Cloud Director synchronizes the VMDKs when you request a sync on each item. Moreover, if a publisher does not have the content downloaded, the catalog subscription with this option does not trigger download of VMDKs at the publisher side.
API support for instantiating fenced vApp templates in NSX-T backed VDCs and for moving fenced vApps to VDCs backed by NSX-T Data Center

You can use the VMware Cloud Director API to instantiate a vApp template that contains fenced networks into a VDC backed by NSX-T Data Center. You can also move a fenced vApp from a VDC backed by NSX Data Center for vSphere to a VDC backed by NSX-T Data Center.

VMware Cloud Provider Blog

For more information about the new and updated features of this release, see What's New in VMware Cloud Director 10.4.

Security

VMware Cloud Director STIG Readiness Guide

In the United States Department of Defense (DoD), Security Technical Implementation Guides (STIGs) provide technical, standards-based hardening guidance. Officially published STIGs are mandatory in the DoD and fill a crucial role in systems accreditation as part of the Risk Management Framework (RMF). STIGs purpose is to deal with a variety of threats, vulnerabilities and remediations:

Intrusion Avoidance
Intrusion Detection
Response and Recovery
Security Implementation Guidance

VMware Cloud Director 10.4 includes the first release of the VMware Cloud Director STIG Readiness Guide. VMware produces the VMware Cloud Director STIG Readiness Guide. The guide does not have Defense Information Systems Agency (DISA) ownership. The guide provides Security Requirements Guidelines (SRGs) content that is ready to go through the DISA process but is not a DISA published STIG. See the VMware Cloud Director STIG Readiness Guide Overview document at the download link.

STIG work for VMware Cloud Director is ongoing and will continue over the next several releases. STIG Readiness Guides for VMware Cloud Director Appliance and its underlying components, such as PostgreSQL, NGINX, Photon OS, and VMware Cloud Director Application, are accessible through the VMware Cloud Director Readiness Guide ZIP file. You can use STIG viewing tools to read and review the XML documents and to create and manage security compliance checklists for your deployments.

VMware Cloud Director STIG Compliance Auditing profile and Hardening Playbook

For a compliance auditing profile based on Chef InSpec and CINC Auditor to perform an automated check for STIG compliance of the VMware Cloud Director 10.4 STIG Readiness Guide, see the VMware Cloud Director 10.4 STIG Readiness Guide Chef InSpec Profile. For a hardening playbook that uses Ansible to perform automated remediation for STIG compliance of the VMware Cloud Director 10.4 STIG Readiness Guide, see the VMware Cloud Director 10.4 STIG Readiness Guide Ansible Playbook.

Product Support Notices

NSX Data Center for vSphere Support

VMware Cloud Director will continue to operate with NSX Data Center for vSphere to support migrations to VMware NSX-T Data Center until January 2023. At that time VMware will evaluate the future of NSX Data Center for vSphere as a backing networking platform in VMware Cloud Director. It is strongly recommended that VMware Cloud Director customers migrate to VMware NSX-T Data Center before January 2023 to avoid any disruptions in support.
VMware Cloud Director API versions 31.0 and 32.0 are not supported.
VMware Cloud Director API versions 33.0, 34.0, 35.0 and 35.2 are deprecated and will be unsupported starting with the next major VMware Cloud Director release.

Upgrading from Previous Releases

For more information on upgrading to VMware Cloud Director 10.4, upgrade and migration paths and workflows, see Upgrading and Migrating the VMware Cloud Director Appliance or Upgrading VMware Cloud Director on Linux.

System Requirements and Installation

Ports and Protocols
Compatibility Matrix
Supported VMware Cloud Director Server Operating Systems
Supported AMQP Servers
Supported Databases for Storing Historic Metric Data
Disk Space Requirements
Memory Requirements
CPU Requirements
Required Linux Software Packages
SDK/Plugin Support
Identity Provider Support
Supported Security Protocols and Cipher Suites
Supported Browsers
Supported Guest Operating Systems and Virtual Hardware Versions

Ports and Protocols

For information on the network ports and protocols that VMware Cloud Director 10.4 uses, see VMware Ports and Protocols.

Compatibility Matrix

See the VMware Product Interoperability Matrixes for current information about:

VMware Cloud Director interoperability with other VMware platforms
Supported VMware Cloud Director databases

Supported VMware Cloud Director Server Operating Systems

CentOS 7
CentOS 8
Red Hat Enterprise Linux 7
Red Hat Enterprise Linux 8

Supported AMQP Servers

VMware Cloud Director uses AMQP to provide the message bus used by extension services, object extensions, and notifications. This release of VMware Cloud Director requires RabbitMQ version 3.8.x or 3.9.x.

For more information, see the VMware Cloud Director Installation, Configuration, and Upgrade Guide.

Supported Databases for Storing Historic Metric Data

VMware Cloud Director supports Apache Cassandra versions 3.11.x and 4.0.x.

Disk Space Requirements

Each VMware Cloud Director server requires approximately 2100MB of free space for the installation and log files.

Memory Requirements

Please consult VMware Cloud Director Installation, Configuration, and Upgrade Guide for memory requirements.

CPU Requirements

VMware Cloud Director is a CPU-bound application. CPU over-commitment guidelines for the appropriate version of vSphere should be followed. In virtualized environments, regardless of the number of cores available to VMware Cloud Director, there must be a sensible vCPU to physical CPU ratio, that does not result in extreme over-committing.

Required Linux Software Packages

Each VMware Cloud Director server must include installations of several common Linux software packages. These packages are typically installed by default with the operating system software. If any of the packages are missing, the installer fails with a diagnostic message.

In addition to the installer required packages, several procedures for configuring the network connections and creating SSL certificates require the use of the Linux nslookup command, which is available in the Linux bind-utils package.

SDK/Plugin Support

If you plan to build custom service plugins to run against VMware Cloud Director API version 37.0.0-alpha, use @vcd/sdk version 0.12.2-alpha.5 or later.

Identity Provider Support

VMware Cloud Director 10.4 supports LDAP, SAML, and OpenId Connect (OIDC) identity providers.

Supported Security Protocols and Cipher Suites

VMware Cloud Director requires the client connections to be secure. SSL version 3 and TLS version 1.0 and 1.1 have been found to have serious security vulnerabilities and are no longer included in the default set of protocols that the server offers to use when making a client connection. System administrators can enable more protocols and cipher suites. See the Cell Management Tool section in the VMware Cloud Director Installation, Configuration, and Upgrade Guide. The following security protocols are supported:

TLS version 1.2
TLS version 1.1 (disabled by default)
TLS version 1.0 (disabled by default)

To enable the disabled versions, see KB 88929.

Supported cipher suites enabled by default:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Supported cipher suites disabled by default:

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA

System administrators can use the cell management tool to explicitly enable the supported cipher suites that are disabled by default.

Supported Browsers

VMware Cloud Director is compatible with the current major and previous major release of the following browsers:

Google Chrome
Mozilla Firefox
Microsoft Edge

Supported Guest Operating Systems and Virtual Hardware Versions

VMware Cloud Director supports all guest operating systems and virtual hardware versions supported by the ESXi hosts that back each resource pool.

Note:

In VMware Cloud Director 10.4.1 and earlier, you cannot add or manage vTPM devices. When creating Windows 11 VMs or any other VMs that require vTPM devices, you might need to add the vTPM devices using vCenter Server. VMware Cloud Director does not interfere with vTPM devices added to VMs through vCenter Server.

Resolved Issues

New - The VMware Cloud Director console proxy, uploading OVFs and media, and powering on a VM fail

VMware Cloud Director 10.4 enhances SSL connectivity to all vSphere infrastructure components, including ESXi, by incorporating the vSphere Certificate Authority (CA) into the VMware Cloud Director trust mechanisms. In certain cases, the vSphere endpoint and the vSphere CA use different trust anchors and VMware Cloud Director must trust more than one trust anchor from vSphere. If the vSphere CA is not trusted, some VMware Cloud Director features do not work.
New - You cannot login to the VMware Cloud Director portals by using Single Sign On

When customizing the VMware Cloud Director portals, if you change the default configuration for the portalColor parameter, you cannot login to the portals by using Single Sign On. On the login page of the tenant portal, when you click on Sign In with Single Sign-On, the system redirects you back to the login page.
New - All tenants in a multi-tenant VMware Cloud Director environment can see the port profiles for a specific organization

In a multi-tenant VMware Cloud Director environment, if an organization administrator enables the load-balancing services for an organization by using the capabilities of VMware NSX Advanced Load Balancer (Avi Networks), the application port profiles become visible for all tenants in this multi-tenant VMware Cloud Director environment.
New - Fast cross vCenter Server vApp instantiation using shared storage fails when a cluster which does not have access to the source VM template's datastore backs the target VDC

When performing fast cross vCenter Server vApp instantiation using shared storage, the source VM template is always registered as a multi-vc-vm in the target vCenter Server instance. This VM is registered in a child resource pool of the target VDC's backing provider VDC. However, if the target resource pool resides in a cluster which does not have access to the source VM template's datastore, the registration is impossible and fails.

If you attempt an instantiation that is eligible for fast cross vCenter Server vApp instantiation and the target VDC backing cluster cannot access the source datastore, the instantiation always fails unless you disabled the feature.
You can add new VMs to a vApp by using a user without Create and Reconfigure a vApp permissions

As a user without Create and Reconfigure a vApp permissions, you can add a new VM to an existing vApp.
Creating a vApp from a vApp template or by using an OVF deploys the new vApp with an incorrect configuration for the administrator password on the guest operating system

If you create a new vApp by using an OVF or a vApp template that is disabled for guest customization, VMware Cloud Director deploys the new vApp with enabled option to allow setting an administrator password on the guest operating system.
After upgrading to VMware Cloud Director 10.3.2, you cannot execute commands with the cell management tool when using credentials from an external LDAP identity provider

After upgrading to VMware Cloud Director 10.3.2, when attempting to run commands by using a user from an external LDAP identity provider, the cell management tool returns the Unable to connect to the cell: Invalid credentials. Exiting. error message.
An attempt to migrate tenant storage fails with an Internal Server Error error message

In the HTML5 UI, using the Migrate Tenant Storage option to migrate all the items stored on a datastore to other datastores in an SDRS Clusterfails to migrate the VMs with an errors message.

Internal Server ErrorCaused by: java.lang.RuntimeException: The operation failed because no suitable resource was found. Out of x candidate hubs:x hubs eliminated because: No valid storage containers found for VirtualMachine "{vm-uuid}". All x available storage containers were filtered out as being invalid.
When you use VMware Cloud Director API version 35.2 or earlier to access a powered off and deployed VM, or a suspended and deployed VM, the power states of the VMs appear as PARTIALLY_POWERED_OFF and PARTIALLY_SUSPENDED, respectively

When you use a version of VMware Cloud Director API version 35.2 or earlier to access a VM that is powered off and deployed or a VM that is suspended and deployed, the power states of the VMs appear as PARTIALLY_POWERED_OFF and PARTIALLY_SUSPENDED, respectively. This happens because of a backward incompatible change in VMware Cloud Director API version 36.0 which introduced these new power states. As a result, API calls from versions 35.2 and earlier that attempt to process these states fail. This issue is resolved in this release. If you are using an API client version 35.2 or earlier, the states of the VMs appear as POWERED_OFF and SUSPENDED, respectively.
Using the VMware Cloud Director API 36.2 or later to power off or discard the suspended state of a VM also results in undeploying the VM

In VMware Cloud Director API 36.2 and later versions, when you make the following API requests, this results in undeploying the VM besides powering it off or discarding its suspended state.

POST /vApp/{vm-id}/action/powerOff

POST /vApp/{vm-id}/action/discardSuspendedState

This change creates a backward incompatibility with API versions 36.1 and 36.0, in which these API calls result only in powering off or discarding the suspended state of the VM, respectively. This issue is resolved in this release - if you are using an API client version 36.1 or 36.0, the API request results only in powering off or discarding the suspended state of the VM, respectively.
When performing a database upgrade for VMware Cloud Director, the upgrade fails with insert or update on table error

The issue occurs due to stale information in tables associated with a foreign key constraint. Missing data in one of the tables causes a conflict with the foreign key constraint.

Known Issues

New - When using the CloudAPI to create or update an organization, you cannot set to true the canPublish flag

When using the CloudAPI to create an organization or update an organization enabling it to publish catalogs, the canPublish field remains false, despite you setting the value to true. The legacy API is not affected.

Workaround: Use the VMware Cloud Director UI to activate or deactivate the option to Publish catalog externally for an organization.
New - VMware Cloud Director backup fails

If you use Ubuntu or Linux distributions that are based on Debian as NFS for the VMware Cloud Director appliance, the NFS server cannot be configured appropriately to support the creation of backups through the PostgreSQL user.
Workaround: Depending on the file that the appliance has, run the following commands from appliance's secure shell as the root user.
- If the appliance has the /opt/vmware/appliance/bin/create-db-backup file, run the following command.
sed -i '/PG_BACK_UP() {/,/}/ { /PG_BACK_UP() {/!{ /}/!d }}; /PG_BACK_UP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dump -v -Fc \$DBNAME" > \$DB_DUMP_PATH 2>> \$LOG_FILE' /opt/vmware/appliance/bin/create-db-backup
- If the appliance has the /opt/vmware/appliance/bin/create-backup.sh file, run the following commands.
sed -i '/DB_BACKUP() {/,/}/ { /DB_BACKUP() {/!{ /}/!d }}; /DB_BACKUP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dump -v -Fc \$DBNAME" > \$DB_DUMP_PATH 2>> \$LOG_FILE' /opt/vmware/appliance/bin/create-backup.sh

sed -i '/DB_USER_BACKUP() {/,/}/ { /DB_USER_BACKUP() {/!{ /}/!d }}; /DB_USER_BACKUP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dumpall --roles-only | grep -e '\''CREATE ROLE vcloud;\\|ALTER ROLE vcloud WITH'\''" > \$BACKUP_DIR\/vcloud-user.sql' /opt/vmware/appliance/bin/create-backup.sh
New - After upgrading a VMware Cloud Director appliance, the management API and the management UI report an incorrect older version of the appliance

The problem occurs because the VMware Cloud Director appliance management API uses a different source of truth for obtaining the current version of the VMware Cloud Director appliance than the vamicli version --appliance command. This alternate source of truth is not always being updated during the appliance upgrade causing incorrect information to appear.

Workaround: Use the vamicli version --appliance command to verify the VMware Cloud Director appliance version.
New - When you deploy a VM from a template with a storage policy that includes a configured IOPS limit, after deployment, the VM disks do not have an IOPS limit configured or have a different IOPS limit

The problem occurs because the the I/O Operations Per Second (IOPS) limit set in the VM template overrides the storage policy's IOPS limit. For example, if the VM template does not have a configured IOPS value, after deployment, the VM disks do not have a configured IOPS limit.

Workaround: You can use vApp templates, or edit the VM after deployment.
New - When sharing vApps with users, you can navigate to nonexistent pages

When sharing vApps with users, the buttons to go to the previous or next page are available even though you are already on the respective first or last page. As a result, you can navigate to pages that do not exist and do not have content.

Workaround: None.
New - Deleting an organization in VMware Cloud Director UI fails with a You must delete this Organization's Application Port Profiles before you can delete the organization error
If application port profiles are created on an edge gateway associated with an organization, attempting to delete the organization fails. The issue occurs because VMware Cloud Director deletes the edge gateways before deleting the port profiles, which causes the following error.
```
com.vmware.vcloud.api.presentation.service.InvalidStateException: You must delete this Organization's Application Port Profiles before you can delete the organization.
```
Workaround: Use the VMware Cloud Director API to force delete an organization and to delete the stranded application port profiles associated with it. See Delete Stranded Application Port Profiles from VMware Cloud Director.
New - You cannot access the Service Provider Admin Portal and the VMware Cloud Director Tenant Portal after rebooting the VMware Cloud Director VM

If you reboot the VMware Cloud Director VM by using a method other than using the vSphere Client, for example, by using vSphere High Availability or VMware Host Client, you cannot access the Service Provider Admin Portal and the VMware Cloud Director Tenant Portal. The problem occurs because after the reboot, the deployment OVF parameters are deleted from the ovfEnv.xml file, and the cell cannot be accessed.

Workaround: Power off and then power on the VMware Cloud Director VM by using the vSphere Client.
New - VM does not receive the DNS Server IP addresses from the DHCP scope that is defined in the vApp network

When you connect a VM to a routed vApp network in DHCP IP mode, the VM does not receive the DNS addresses defined in the DHCP scope.

Workaround: Using NSX Manager, manually configure the DNS servers in the routed vApp network segment.
New - API clients throw Invalid mime type errors for responses from multisite VMware Cloud Director APIs
If the multisite field in the response header values specifies a list of organizations, the API client generates the following error.
```
org.springframework.util.InvalidMimeTypeException: Invalid mime type 
```
The issue occurs because the VMware Cloud Director API returns an illegal @ character in the MIME (Multipurpose Internet Mail Extensions) type headers of the response. You can ignore the error because VMware Cloud Director continues to function properly.
Workaround: None.
New - VMware Cloud Director UI and tasks are slow to load and complete
The Artemis message bus communication is not working and when you trigger operations from the UI, they can take up to 5 minutes to complete or might time out. The performance issues can affect operations such as powering on VMs and vApps, provider VDC creation, vApp deployment, and so on.

The log files might contain an error message, such as:
- a) Connection failure to <VCD Cell IP Address> has been detected: AMQ229014: Did not receive data from <something> within the 60,000ms
- b) Connection failure to /<VCD Cell IP Address>:61616 has been detected: AMQ219014: Timed out after waiting 30,000 ms
- c) Bridge is stopping, will not retry
- d) Local Member is not set at on ClusterConnection ClusterConnectionImp
Workaround:

For a) and b):
1. Verify that the VMware Cloud Director cells have network connectivity and can communicate with each other.
2. Restart the VMware Cloud Director cell that contains the error message.
For c) and d), restart the VMware Cloud Director cell that contains the error message.
New - The VMware Cloud Director appliance database disk resize script might fail if the backing SCSI disk identifier changes

The database disk resize script runs successfully only if the backing database SCSI disk ID remains the same. If the ID changes for any reason, the script might appear to run successfully but fails. The /opt/vmware/var/log/vcd/db_diskresize.log shows that the script fails with a No such file or directory error.
Workaround:
1. Log in directly or by using an SSH client to the primary cell as root.
2. Run the lsblk --output NAME,FSTYPE,HCTL command.
3. In the output, find the disk containing the database_vg-vpostgres partition and make note of its ID. The ID is under the HCTL column and has the following sample format 2:0:3:0.
4. In the db_diskresize.sh script, modify the partition ID with the ID from Step 3. For example, if the ID is 2:0:3:0, in line
```
echo 1 > /sys/class/scsi_device/2\:0\:2\:0/device/rescan
```
  you must change the ID to 2:0:3:0.
```
echo 1 > /sys/class/scsi_device/2\:0\:3\:0/device/rescan
```
5. Аfter saving the changes, manually re-invoke the resize script or reboot the appliance.
New - Fast cross vCenter Server instantiation of a vApp template with memory state fails

If you attempt an instantiation that is eligible for fast cross vCenter Server vApp template instantiation while preserving the memory state of the VMs within the vApp, the operation fails with an error message.

java.util.concurrent.ExecutionException: com.vmware.ssdc.util.LMException: Internal Server Error

Workaround: Move the template to a datastore that is not shared between the vCenter Server instances to avoid VMware Cloud Director performing fast vApp template instantiation.
New - Publishing a vRealize Orchestrator workflow to the VMware Cloud Director service library fails with an error message

When you attempt to publish a vRealize Orchestrator workflow, the operation fails with a 500 Server Error error message.

This happens because the API returns a large number of links for each individual tenant to which the workflow is published and causes an overflow in the HTTP headers.

Workaround: To publish the workflow, use CURL or POSTMAN to run an API request with increased HTTP header size limit.
New - VMware Cloud Director operations, such as powering a VM on and off takes longer time to complete

VMware Cloud Director operations, such as powering a VM on or off takes longer time to complete. The task displays a Starting virtual machine status and nothing happens.

The jms-expired-messages.logs log file displays an error.

RELIABLE:LargeServerMessage & expiration=

Workaround: None.
New - Runtime defined entity (RDE) modify event entries cause the Audit_trail database table to grow at an uncontrollable rate

Runtime defined entities (RDE) modify event entries cause the Audit_trail database table to grow uncontrollably. This happens because the database backs up the complete RDE and not only the changes.

Workaround: Set the audit.rde.diffOnly config property to true.
New - Switching to a vApp or a VM using the Quick Search option while updating another vApp or VM, might result in changed object settings

Using the Quick Search feature to switch between objects such as vApps or VMs while the task of updating another object is not finished might result in renaming the vApp or VM that you are updating or changing some of its other settings.

Workaround: Before you use the Quick Search feature to switch to another object, wait for the ongoing update task to finish.
New - Migrating VMs between organization VDCs might fail with an insufficient resource error

If VMware Cloud Director is running with vCenter Server 7.0 Update 3h or earlier, when relocating a VM to a different organization VDC, the VM migration might fail with an insufficient resource error even if the resources are available in the target organization VDC.

Workaround: Upgrade vCenter Server to version 7.0 Update 3i or later.
New - Suspending a VM through the VMware Cloud Director UI results in a partially suspended state of the VM

In the VMware Cloud Director Tenant Portal, when you suspend a VM, VMware Cloud Director does not undeploy the VM, and the VM becomes Partially Suspended instead of Suspended.

Workaround: None.
New - Role name and description are localized in the VMware Cloud Director UI and can cause duplication of role names

The problem occurs because the UI translation does not affect the back end and API. You might create roles with the same names as the translated names which results in perceived duplicate roles in the UI and conflicts with the API usage of role names when creating service accounts.

Workaround: None.
New - The Customer Experience Improvement Program (CEIP) status is Enabled even after deactivating it during the installation of VMware Cloud Director

During the installation of VMware Cloud Director, if you deactivate the option to join the CEIP, after the installation completes, the CEIP status is active.

Workaround: Deactivate the CEIP by following the steps in the Join or Leave the VMware Customer Experience Improvement Program procedure.
New - When starting the VMware Cloud Director appliance, the message [FAILED] Failed to start Wait for Network to be Configured. See 'systemctl status systemd-networkd-wait-online.service' for details appears.

The message appears incorrectly and does not indicate an actual problem with the network. You can disregard the message and continue to use the VMware Cloud Director appliance as usual.

Workaround: None.
New - The VMware Cloud Director Tenant Portal UI does not display the IOPS limits and reservations for a vSAN storage policy

vSAN manages itself the IOPS limits on vSAN storage policies. As a result, the VMware Cloud Director Tenant Portal UI does not display the IOPS reservations and limits for a vSAN storage policy and you cannot modify their values.

Workaround: None.
VMware Cloud Director appliance upgrade fails with an invalid version error when FIPS mode is enabled

For VMware Cloud Director versions 10.3.x and later, when FIPS mode is enabled, VMware Cloud Director appliance upgrade fails with the following error.

Failure: Installation failed abnormally (program aborted), the current version may be invalid.
Workaround:
1. Before you upgrade the VMware Cloud Director appliance, deactivate FIPS Mode on the cells in the server group and the VMware Cloud Director appliance. See Enable or Disable FIPS Mode on the VMware Cloud Director Appliance.
2. Verify that the /etc/vmware/system_fips file does not exist on any appliance.
3. Upgrade the VMware Cloud Director appliance.
4. Enable FIPS mode again.
Restore from an appliance backup might fail with an Invalid command-line arguments. Missing argument for option: consoleproxy-cert error

If you run the clear-console-proxy-settingsCMT command before you take an appliance backup, then, if you choose to restore the console proxy certificate from the backup, the restore process fails with an Invalid command-line arguments. Missing argument for option: consoleproxy-cert error.

The issue occurs because the command to clear the console proxy settings removes the console proxy certificate, and the console proxy settings are missing for the backup. If the console proxy certificate is not in the backup, you cannot restore it.

If the console proxy settings were cleared, run the appliance restore without selecting to restore the console proxy certificate.
If you use fast cross vCenter Server vApp instantiation and then you delete the VM with prefix multi-vc-vm- that was created during the instantiation, this also deletes the VMX and VMDK files of the original VM source template of the instantiation

If you use fast cross vCenter Server vApp instantiation and then you delete the VM with prefix multi-vc-vm- that was created during the instantiation, this also deletes the VMX and VMDK files of the original VM source template that you used for the instantiation. This happens because when you use fast cross vCenter Server vApp instantiation to instantiate a VM, the source VM template which is located on vCenter Server A is registered with vCenter Server B, creating a VM with prefix multi-vc-vm- which spans across the two vCenter Server instances, while its VMX and VMDK files are stored with the original VM template on vCenter Server A. The multi-vc-vm- VM will be deleted either if it's deleted directly from vCenter Server or if it's imported to VMware Cloud Director with the Delete Source check box selected, and then deleted from VMware Cloud Director.

Workaround: Do not delete or import multi-vc-vm- VMs.
In the VMware Cloud Director UI, clicking Help in the top navigation bar does not lead you to the relevant product documentation

In the VMware Cloud Director UI, clicking Help in the top navigation bar does not lead you to the relevant product documentation. This happens because in VMware Cloud Director 10.4, the Help menu link is retrieved from a custom link, and the default value for the custom link is null.

Use the branding vCloud OpenAPI methods to modify the custom link in the Help menu. See Customizing the VMware Cloud Director Portals and Getting Started with VMware Cloud Director OpenAPIat https://developer.vmware.com.
In the VMware Cloud Director UI, clicking Download VMRC does not lead you to the relevant download page

In the VMware Cloud Director UI, clicking Download VMRC does not redirect you to https://my.vmware.com to download VMRC. This happens because in VMware Cloud Director 10.4, the Download VMRC link is retrieved from a custom link, and the default value for the custom link is null.

Use the branding vCloud OpenAPI methods to modify the custom link in the Download VMRC menu item. See Customizing the VMware Cloud Director Portals and Getting Started with VMware Cloud Director OpenAPI at https://developer.vmware.com.
After you enable legacy console proxy mode, the VMware Cloud Director cell on which you enabled the feature fails to start

After you enable legacy console proxy mode from the Feature Flag settings menu, the VMware Cloud Director cell on which you enabled the feature fails to start. This happens because in VMware Cloud Director 10.4, the console proxy uses the same IP address and port as the REST API. Because the IP address and the port are not available, the cell cannot start with its default configuration.
Workaround 1: If the legacy console proxy properties in the global.properties and response.properties files are missing, rerun the configure command /opt/vmware/vcloud-director/bin/configure --unattended-installation and specify the following properties.
- --consoleproxy-host-https
- --consoleproxy-keystore-password
- --user-consoleproxy-certificate-path
- --user-consoleproxy-key-path
- --user-consoleproxy-key-password
- --consoleproxy-port-https
If you are not using the same IP address for the console proxy and for the REST API, specifying a port is optional. If you don't specify a port, the console proxy will use the default 443. See Unattended Configuration Reference.

Workaround 2: Use the PostgreSQL interactive terminal to run the following command.

UPDATE feature_flags SET is_enabled = false WHERE name='LegacyConsoleProxy'

This disables the legacy console proxy feature that you enable from the Feature Flag settings menu and VMware Cloud Director reverts to using its default 10.4 console proxy implementation.
If you run the cell management tool's clear-console-proxy-settings command while legacy console proxy mode is enabled, you can't restart the cell on which you ran the command

When you run the clear-console-proxy-settings subcommand of the cell management tool, it removes the console proxy certificates and properties regardless of whether the legacy console proxy feature flag is enabled or not. If you clear the settings while the legacy console proxy feature flag is enabled, this prevents the cell from restarting.
Workaround 1: If the legacy console proxy properties in the global.properties and response.properties files are missing, rerun the configure command /opt/vmware/vcloud-director/bin/configure --unattended-installation and specify the following properties.
- --console-proxy-ip
- --consoleproxy-cert
- --consoleproxy-key
- --consoleproxy-key-password
- --console-proxy-port-https
If you are not using the same IP address for the console proxy and for the REST API, specifying a port is optional. If you don't specify a port, the console proxy will use the default 443. See Unattended Configuration Reference.

Workaround 2: Use the PostgreSQL interactive terminal to run the following command.

UPDATE feature_flags SET is_enabled = false WHERE name='LegacyConsoleProxy'

This disables the legacy console proxy feature that you enable from the Feature Flag settings menu and VMware Cloud Director reverts to using its default 10.4 console proxy implementation.
You can't view and edit the license type for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API

You can't view and edit the license for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API. This happens because in VMware Cloud Director 10.4, the Controller license type was replaced by a selection between a Standard and a Premium feature set at the Service Engine Group level to provide more flexibility.

Workaround: Use the supportedFeatureSet path for service engine groups and on edge gateways to enable and disable the available features.
When you attempt to delete a stranded item in VMware Cloud Director by clicking OK on the Delete Standed Item window, the window becomes unresponsive

When you attempt to delete a stranded item in VMware Cloud Director by clicking OK on the Delete Standed Item window, the window becomes unresponsive. This issue occurs when your network connection to the VMware Cloud Director instance is slow. Fetching a stranded item might take up to five minutes, during which the UI is unresponsive. If you click the Cancel button, the window closes, but the deletion of the item is not cancelled.

Workaround: Wait for the window to close on its own.
You can't create and use VMware Cloud Director VDC templates in VMware Cloud Director service environments that use VMware Cloud on AWS network pools

If you are using only a provider network pool that is backed by VMware Cloud on AWS for your provider VDC, you cannot create a VDC template and instantiate a VDC from a template. This happens because creating and instantiating VDC templates is supported only for provider VDCs backed by NSX-T Data Center and by NSX Data Center for vSphere. You can use VMware Cloud Director VDC templates with on-premises, Microsoft Azure VMware Solution, Oracle Cloud VMware Solution, or Google Cloud VMware Engine SDDCs.

Workaround: None.
Creating a new VM with encrypted vSAN storage policy fails with an Invalid storage policy for encryption operation error message

When creating a new VM, if you specify the storage policy of the VM as vSAN encrypted and the storage policy for the VM hard disk as both non-encrypted and non-vSAN, the operation fails with an error message.

Invalid storage policy for encryption operation
1. Specify the storage policies for the VM and the VM hard disk as vSAN encrypted.
2. After the VM deploys successfully, update the hard disk storage policy for the VM to non-encrypted and non-vSAN. For information, see Edit Virtual Machine Properties.
You cannot connect to VMware Cloud Director through VMware OVF Tool version 4.4.3 or earlier

When you attempt to connect to VMware Cloud Director through OVF Tool version 4.4.3 or earlier, this results in the following error. Error: No supported vCloud version was found. This happens because of an API behavior change in VMware Cloud Director 10.4 where the API does not return links to all the VDCs in an organization.

Workaround: Upgrade to OVF Tool 4.5.0. See VMware OVF Tool Release Notes.
You are unable to log in to VMware Cloud Director by using VMware PowerCLI 12.7.0 or earlier

When you attempt to log in to VMware Cloud Director by using VMware PowerCLI 12.7.0 or earlier, this results in the following error. NOT_ACCEPTABLE: The request has invalid accept header: Invalid API version requested. This happens because VMware PowerCLI version earlier than 13.0.0 do not support VMware Cloud Director API versions later than 33.0. See VMware Product Interoperability Matrix.

Workaround: Upgrade VMware PowerCLI to version 13.0.0.
VMware Cloud Director displays the old version for an upgraded vCenter Server instance

After you upgrade a vCenter Server instance to a newer version, in the list of vCenter Server instances, VMware Cloud Director still displays the old version for the upgraded instance.

Reset the connection between the vCenter Server instance and VMware Cloud Director. See Reconnect a vCenter Server Instance in VMware Cloud Director Service Provider Admin Portal Guide.
Refreshing the LDAP page in your browser does not take you back to the same page

In the Service Provider Admin Portal, refreshing the LDAP page in your browser takes you to the provider page instead of back to the LDAP page.

Workaround: None.
Mounting an NFS datastore from NetApp storage array fails with an error message during the initial VMware Cloud Director appliance configuration

During the initial VMware Cloud Director appliance configuration, if you configure an NFS datastore from NetApp storage array, the operation fails with an error message.

Backend validation of NFS failed with: is owned by an unknown user

Workaround: Configure the VMware Cloud Director appliance by using the VMware Cloud Director Appliance API.
The synchronization of a subscribed catalog times out while synchronizing large vApp templates

If an external catalog contains large vApp templates, synchronizing the subscribed catalog with the external catalog times out.Theissue occurs when the timeout setting is set to its default value of five minutes.

Workaround: Using the manage-config subcommand of the cell management tool, update the timeout configuration setting.

./cell-management-tool manage-config -n transfer.endpoint.socket.timeout -v [timeout-value]
In an IP prefix list, configuring any as the Network value results in an error message

When creating an IP prefix list, if you want to deny or accept any route and you configure the Network value as any, the dialog box displays an error message.

"any" is not a valid CIDR notation. A valid CIDR is a valid IP address followed by a slash and a number between 0 and 32 or 64, depending on the IP version.

Workaround: Leave the Network text box blank.
If you use vRealize Orchestrator 8.x, hidden input parameters in workflows are not populated automatically in the VMware Cloud Director UI

If you use vRealize Orchestrator 8.x, when you attempt to run a workflow through the VMware Cloud Director UI, hidden input parameters are not populated automatically in the VMware Cloud Director UI.
Workaround:To access the values of the workflow input parameters, you must create a vRealize Orchestrator action that has the same input parameter values as the workflow that you want to run.
1. Log in to the vRealize Orchestrator Client and navigate to Library>Workflows.
2. Select the Input Form tab and click Values on the right-hand side.
3. From the Value options drop-down menu, select External source,enter the Action inputs, and click Save.
4. Run the workflow in the VMware Cloud Director UI.
The vpostgres process in a standby appliance fails to start

The vpostgres process in a standby appliance fails to start and the PostgreSQL log shows an error similar to the following. FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the master server (its value was 16). This happens because PostgreSQL requires standby nodes to have the same max_worker_processes setting as the primary node. VMware Cloud Director automatically configures the max_worker_processes setting based on the number of vCPUs assigned to each appliance VM. If the standby appliance has fewer vCPUs than the primary appliance, this results in an error.

Workaround: Deploy the primary and standby appliances with the same number of vCPUs.
Upgrading from VMware Cloud Director 10.2.x to VMware Cloud Director 10.3 results in an Connection to sfcbd lost error message

If you upgrade from VMware Cloud Director 10.2.x to VMware Cloud Director 10.3, the upgrade operation reports an error message.

Connection to sfcbd lost. Attempting to reconnect

Workaround: You can ignore the error message and continue with the upgrade.
When using FIPS mode, trying to upload OpenSSL-generated PKCS8 files fails with an error

OpenSSL cannot generate FIPS-complaint private keys. When VMware Cloud Director is in FIPS mode and you try to upload PKCS8 files generated using OpenSSL, the upload fails with a Bad request: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: ... not available: No such algorithm: ...error or salt must be at least 128 bits error.

Workaround: Disable FIPS mode to upload the PKCS8 files.
Creation of Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in fails

When you create a Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in, you must select a Kubernetes version. Some of the versions in the drop-down menu are not compatible with the backing vSphere infrastructure. When you select an incompatible version, the cluster creation fails.

Workaround: Delete the failed cluster record and retry with a compatible Tanzu Kubernetes version. For information on the incompatibilities between Tanzu Kubernetes and vSphere, see Updating the vSphere with Tanzu Environment.
If you have any subscribed catalogs in your organization, when you upgrade VMware Cloud Director, the catalog synchronization fails

After upgrade, if you have subscribed catalogs in your organization, VMware Cloud Director does not trust the published endpoint certificates automatically. Without trusting the certificates, the content library fails to synchronize.

Workaround: Manually trust the certificates for each catalog subscription. When you edit the catalog subscription settings, a trust on first use (TOFU) dialog prompts you to trust the remote catalog certificate.

If you do not have the necessary rights to trust the certificate, contact your organization administrator.
After upgrading VMware Cloud Director and enabling the Tanzu Kubernetes cluster creation, no automatically generated policy is available and you cannot create or publish a policy

When you upgrade VMware Cloud Director to version 10.3.1 and vCenter Server to version 7.0.0d or later, and you create a provider VDC backed by a Supervisor Cluster, VMware Cloud Director displays a Kubernetes icon next to the VDC. However, there is no automatically generated Kubernetes policy in the new provider VDC. When you try to create or publish a Kubernetes policy to an organization VDC, no machine classes are available.

Workaround: Manually trust the corresponding Kubernetes endpoint certificates. See VMware knowledge base article 83583.
Entering a Kubernetes cluster name with non-Latin characters disables the Next button in the Create New Cluster wizard

The Kubernetes Container Clusters plug-in supports only Latin characters. If you enter non-Latin characters, the following error appears.

Name must start with a letter and only contain alphanumeric or hyphen (-) characters. (Max 128 characters).

Workaround: None.
NFS downtime can cause VMware Cloud Director appliance cluster functionalities to malfunction

If the NFS is unavailable due to the NFS share being full, becoming read only, and so on, can cause appliance cluster functionalities to malfunction. HTML5 UI is unresponsive while the NFS is down or cannot be reached. Other functionalities that might be affected are the fencing out of a failed primary cell, switchover, promoting a standby cell, and so on. For more information about setting up correctly the NFS shared storage, see Preparing the Transfer Server Storage for the VMware Cloud Director Appliance.
Workaround:
- Fix the NFS state so that it is not read-only.
- Clean up the NFS share if it is full.
Trying to encrypt named disks in vCenter Server version 6.5 or earlier fails with an error

For vCenter Server instances version 6.5 or earlier, if you try to associate new or existing named disks with an encryption enabled policy, the operation fails with a Named disk encryption is not supported in this version of vCenter Server. error.

Workaround: None.
A fast-provisioned virtual machine created on a VMware vSphere Storage APIs Array Integration (VAAI) enabled NFS array, or vSphere Virtual Volumes (VVols) cannot be consolidated

In-place consolidation of a fast provisioned virtual machine is not supported when a native snapshot is used. Native snapshots are always used by VAAI-enabled datastores, as well as by VVols. When a fast-provisioned virtual machine is deployed to one of these storage containers, that virtual machine cannot be consolidated .

Workaround: Do not enable fast provisioning for an organization VDC that uses VAAI-enabled NFS or VVols. To consolidate a virtual machine with a snapshot on a VAAI or a VVol datastore, relocate the virtual machine to a different storage container.
If you add an IPv6 NIC to a VM and then you add an IPv4 NIC to the same VM, the IPv4 north-south traffic breaks

Using the HTML5 UI, if you add an IPv6 NIC first or configure an IPv6 NIC as the primary NIC in a VM, and then you add an IPv4 NIC to the same VM, the IPv4 north-south communication breaks.

Workaround: First you must add the IPv4 NIC to the VM and then the IPv6 NIC.