VMware Cloud Director 10.4.1 Release Notes

VMware Cloud Director 10.4.1 \| 08 DEC 2022 \| Build Build 20912720 (installed build 20912624) Check for additions and updates to these release notes.

VMware Cloud Director 10.4.1 | 08 DEC 2022 | Build Build 20912720 (installed build 20912624)

Check for additions and updates to these release notes.

What's New

VMware Cloud Director version 10.4.1 includes the following:

Support for vSphere 8.0

VMware Cloud Director 10.4.1 brings support for vSphere 8.0.
Support for NSX 4.0.1.1.

VMware Cloud Director 10.4.1 brings support for NSX 4.0.1.1.
Support for UEFI Boot and Secure Boot

You can use VMware Cloud Director to create a VM with either BIOS or EFI boot firmware. If you select EFI, you can activate Secure boot. Secure boot in combination with UEFI provides strong assurance that the PC manufacturer verified and trusts the firmware code.
Solution Add-On Management

VMware Cloud Director 10.4.1 introduces several new concepts that facilitate creating, deploying, running, and managing extensions. Solution Add-Ons are an evolution of VMware Cloud Director extensions that are built, implemented, packaged, deployed, instantiated, and managed following a new extensibility framework. Solution Add-Ons contain custom functionality or services and can be built and packaged by a cloud provider or by an independent software vendor. VMware also develops and publishes its own VMware Cloud Director Solution Add-Ons.

The Solution Add-On Landing Zone is a part of the provider management plane that represents a container of compute, storage and networking resources dedicated to hosting, managing, and running solution add-ons on behalf of the cloud provider.

In VMware Cloud Director 10.4.1, cloud providers can create and configure the Solution Add-On Landing Zone, as well as deploy, instantiate, and delete add-ons. For more information, see Using Solution Add-Ons with VMware Cloud Director.

You can use the VMware Cloud Director Extension SDK to build, install and manage VMware Cloud Director Solution Add-Ons in a unified and consistent manner. For details, see VMware Cloud Director Extension SDK.

Important:
To secure a successful installation of the Solution Add-On Management UI, verify that before you install or upgrade to VMware Cloud Director 10.4.1, you configured your public addresses and uploaded the necessary certificates for a secure connection to the VMware Cloud Director API. See Configure Your Solution Add-On Landing Zone.
Embedded database upgraded to PostgreSQL 14
- The VMware Cloud Director appliance version 10.4.1 uses PostgreSQL 14 database.
Note:
The final release of PostgreSQL 10 occurred on November 10, 2022. PostgreSQL version 10 is currently unsupported. If you are using an external PostgreSQL configuration, consider upgrading to a later major version.
- Because of the PostgreSQL database upgrade, for all appliances in the cluster, you must shut down the VMware Cloud Director service earlier in the upgrade process. See Upgrade the VMware Cloud Director Appliance by Using an Update Package and Upgrade the VMware Cloud Director Appliance by Using the VMware Update Repository.
- Upgrading to version 10.4.1 removes any custom parameter settings of the postgresql.auto.conf file. See You loose the custom changes of the parameter settings from the postgresql.auto.conf file after upg….
- The PostgreSQL database upgrade from version 10 to version 14 involves cloning the existing database contents to a new local instance, which results in a temporary increase in disk utilization. Because of this, verify that before starting the upgrade to VMware Cloud Director 10.4.1, you have sufficient free space on the database disk. Because the cloning process only includes the database contents and not the database logs nor the write-ahead logs that are used for replication, verify that the database logs are backed up in compliance with your organization's log retention policies. See Verify the Embedded PostgreSQL Database is Ready for Upgrade to VMware Cloud Director 10.4.1.
- Increased database password length. Starting with version 10.4.1, the minimum database password length is at least 14 characters.
- The default PostgreSQL configuration does not support MD5 hashing for password authentication. The VMware Cloud Director appliance does not support MD5 hashing for PostgreSQL password authentication. The default password authentication is SCRAM-SHA-256. During the upgrade to version 10.4.1, to convert the password to the SCRAM-SHA-256 hashing, the appliance performs an automated reset of the vcloud user's PostgreSQL password. Irrespective of the previous length of the password, the appliance randomizes it to meet the minimum length requirements.
  
  After the upgrade, you can change the password manually to a custom password with at least 14 characters. See, Change the PostgreSQL Database Password.
IP Spaces

VMware Cloud Director 10.4.1 introduces IP Spaces, an improved IP address management service. IP spaces provide a structured approach to allocating public and private IP addresses across organizations with a specific focus on the provider (tier-0) gateway. IP spaces are designed to simplify the allocation and consumption of IP ranges and prefixes by preventing overlapping IP addresses across organizations or organization VDCs.

IP spaces add tenant-level observability to provider gateways and facilitate service providers in the allocation of public IP addresses to each organization in the form of IP prefixes for networks and IP ranges for network services. As a service provider, you can create public, shared, and private IP spaces. Public IP spaces are quota-based and can be used by multiple organizations. An organization can consume public IP addresses until either its quota is reached or the supply gets exhausted. You can use a shared IP space for services and management networks that are required in the tenant space, but as a service provider, you don't want to expose it to organizations in your environment. A private IP space can be used by only one organization for which no quotas are assigned. Tenants can also create private IP spaces to address their private IP address management needs.
Tool to Detect and Fix Inconsistent Resource Pool Information in VMware Cloud Director

The cell management tool includes a command to detect inconsistent information about resource pools between VMware Cloud Director and vCenter Server. The cell-management-tool detect-rp-mismatches command detects and attempts to resolve inconsistencies between resource pools known to vCenter Server and VMware Cloud Director by updating the VMware Cloud Director database. See Detect and Fix Resource Pool Mismatches between VMware Cloud Director and vCenter Server.
API to Remap Users Between Identity Providers

You can use the user management API to remap local users or users from an existing IDP to a new IDP source. You can use this feature to remap local users to any IDP of your choice as per the product support notice that VMware Cloud Director Starts the Deprecation Process for Local Users. For more information about migrating users between IDPs, see the Remap a User Between Identity Providers topic in the Service Provider Admin Guide or the Remap a User Between Identity Providers topic in the Tenant Guide.
New Branding and Theming Experience

VMware Cloud Director has a new UI and API where providers can create, manage, and assign themes for their tenants. The UI provides a live preview of the look and feel of the VMware Cloud Director Portal when changing the theming, branding, links, and menus. You can migrate the themes created with the old branding APIs.

Note:
The 10.4.1 branding and theming is an Alpha feature and providers must activate it from the Branding API feature flag. When you activate the feature flag, VMware Cloud Director assigns the default light theme to the provider and across all organizations.
Transparent Load Balancing

VMware Cloud Director 10.4.1 adds support for transparent load balancing for NSX edge gateways. The purpose of transparent load balancing is to pass the IP address of the client in incoming packets to the real server. This controls the flow of data in such a way that all communication from the client to the real server flows through the load balancer.

To use transparent load balancer, enable Transparent Mode while configuring your newly created edge gateway. If you have upgraded to VMware Cloud Director 10.4.1 from an earlier version, you must deactivate the load balancer setting on existing edge gateways and activate it again. See Enable Load Balancer on an NSX Edge Gateway.

When you create a virtual service in the Tenant Portal, you can enable Transparent Mode for it to preserve the client's IP address. VMware Cloud Director now allows a variety of real service groups to be added to NSX-backed virtual services. These include IP sets, dynamic, and static groups.
OpenAPI Certificate Format Changes

All OpenAPI certificate APIs format the incoming certificates by removing all characters aside from PEM content enclosed within PEM headers and footers, including the leading and trailing whitespace.
Service Accounts Work Across Sites in a Multisite Environment

A service account token that is granted access in a VMware Cloud Director site also works in associated VMware Cloud Director sites. Only access tokens work across sites, while refresh tokens do not.
Reduced Maximum pageSize of the Service Account Query

The maximum pageSize of the service account query GET {host}/cloudapi/1.0.0/serviceAccounts changes from 128 to 32. The change is not backward compatible.
Legacy Console Proxy Is No Longer Available

Legacy Console Proxy is no longer available and does not appear in the list of features under Feature Flags.
Updated Audit Log Entries for Defined Entity Upgrades

To avoid unnecessary growth of the audit log when using extensions and plugins that perform frequent defined entity updates, the audit log entries for defined entity upgrades include only the differences between the old and the new values. The audit log entries for defined entity creation remain the same and continue to include the full initial defined entity value.

Product Support Notices

VMware Cloud Director Starts the Deprecation Process for Local Users for Production Use

Authentication for local users in VMware Cloud Director does not make use of modern authentication technologies, security best practices, and compliance requirements, such as password policies, 2FA, or MFA support. By using the VMware Cloud Director integration with external identity providers, you can take advantage of all existing and future advancements in the authentication technologies.

VMware Cloud Director continues to fully support the use of local users while they are under deprecation. In version 10.4.1, you can use the provided API-based tools to remap any local users to users backed by an external identity provider of your choice. In future releases, VMware Cloud Director will provide UI-based tools for bulk remapping and other necessary emergency access mechanisms. VMware Cloud Director supports LDAP, SAML, and OIDC external identity provider protocols. See the Remap a User Between Identity Providers topic in the Service Provider Admin Guide or the Remap a User Between Identity Providers topic in the Tenant Guide. After you remap a user, the user can log in using the new login flow, continuing to see all their previously owned assets. You can also consider replacing local users created for automation purposes with service accounts.

VMware Cloud Director will continue to support local users for evaluation use. Production use of local users will continue to be fully supported until June 1, 2024 or later, if necessary.
APIs Under Accelerated Deprecation

VMware Cloud Director API 37.1 (VMware Cloud Director 10.4.1) contains APIs that are under accelerated deprecation and will be removed in future releases. See VMware Cloud Director API Programming Guide API 37.0.
VMware Cloud Director 10.4.1 Does Not Support vCenter Server 6.7 and ESXi 6.7

Security Fixes

Photon OS 3.0 Security Updates

VMware Cloud Director appliance version 10.4.1 includes Photon OS 3.0 security updates for advisories up to and including PHSA-2022-0480. See the Photon OS 3.0 Security Advisories.

System Requirements and Installation

For information about system requirements and installation instructions, see VMware Cloud Director 10.4 Release Notes.

Supported VMware Cloud Director Server Operating Systems
Supported AMQP Servers

Supported VMware Cloud Director Server Operating Systems

CentOS 7
CentOS 8
CentOS 9
Red Hat Enterprise Linux 7
Red Hat Enterprise Linux 8
Red Hat Enterprise Linux 9

Supported AMQP Servers

VMware Cloud Director uses AMQP to provide the message bus used by extension services, object extensions, and notifications. This release of VMware Cloud Director requires RabbitMQ version 3.8.x ,3.9.x, 3.10.x, or 3.11.x. For more information, see the VMware Cloud Director Installation, Configuration, and Upgrade Guide.

Documentation

To access the full set of product documentation, go to VMware Cloud Director Documentation.

Previous Releases of VMware Cloud Director 10.4.x

VMware Cloud Director 10.4 Release Notes

Resolved Issues

New - Changing an existing SNAT rule to DNAT fails to set the source IP address to Any

In the VMware Cloud Director Tenant Portal, if you change the interface type of an existing SNAT rule to DNAT, VMware Cloud Director fails to reset the source IP address to Any.
New - Runtime defined entity (RDE) modify event entries cause the Audit_trail database table to grow at an uncontrollable rate

Runtime defined entities (RDE) modify event entries cause the Audit_trail database table to grow uncontrollably. This happens because the database backs up the complete RDE and not only the changes.
New - In the VMware Cloud Director UI, clicking Download VMRC does not lead you to the relevant download page

In the VMware Cloud Director UI, clicking Download VMRC does not redirect you to https://my.vmware.com to download VMRC. This happens because in VMware Cloud Director 10.4, the Download VMRC link is retrieved from a custom link, and the default value for the custom link is null.

Use the branding vCloud OpenAPI methods to modify the custom link in the Download VMRC menu item. See Customizing the VMware Cloud Director Portals and Getting Started with VMware Cloud Director OpenAPI at https://developer.vmware.com.
Testing the SMTP settings results in an Error: Bad Request: Failed to send an email: Failed messages: SMTPSendFailedException STARTTLS is required to send mail error message

If you configure the SMTP server secure mode to Start TLS, when attempting to test the SMTP settings for your organization, the operation fails with an error message.

Error: Bad Request: Failed to send an email: Failed messages: SMTPSendFailedException STARTTLS is required to send mail
In a multisite deployment, VMware Cloud Director fails to automatically log you in to the tenant portal when attempting to open a local site organization

In a multisite deployment, when setting the global options for the load balancer within /provider/administration/settings/multisite, if you configure a URL for the load balancer, upon an attempt to open a local site organization, VMware Cloud Director fails to automatically log you in to the tenant portal.

This happens because the backend validation of the load balancer URL is case sensitive and VMware Cloud Director does not recognize the URL as a perfect match.
An attempt to migrate a VMware NSX Data Center for vSphere (NSX-V) backed provider VDC to an organization VDC of NSX-T Data Center backed provider VDC fails with an Internal Server Error error message

If you disconnect an isolated network from an organization VDC, attempting to migrate the same organization VDC from NSX-V backed provider VDC to an organization VDC of NSX-T fails with an error message.

Internal Server Error
Editing and deleting a metadata key fails with a Bad Request: The request was rejected because the URL contained a potentially malicious String error message

If a metadata key contains special characters, an attempt to update or delete it fails with an error message.

The request was rejected because the URL contained a potentially malicious String
The storage value of a VM template does not match the storage value of the source VM

If you create a VM template from a powered on VM, the Storage column of the resulted template displays only the disk storage size value instead of a summary of the disk storage size and the memory size.
VMware Cloud Director displays the password auto generation as active after deactivating it by using the HTML5 UI

For a VM with specified administrator password, if you deactivate the password auto generation as a tenant, the HTML5 UI continues displaying the option as active.
VMware Cloud Director does not display the default IOPS storage policy value for a named disk

When creating a named disk, if you do not specify the IOPS storage policy, after the creation operation completes, VMware Cloud Director does not display the default IOPS storage policy value.
The Actions menu for a vApp dissappears while VMware Cloud Director is running an operation on another vApp

If you initiate an operation for a vApp and you attempt to open the Actions menu for a different vApp, the menu dissappears.
VMware Cloud Director does not apply the configured maximum compute policy when converting a VDC into a flex organization VDC

When converting a VDC into a flex organization VDC, VMware Cloud Director does not apply the value you set as a maximum compute policy for the flex organization VDC.
An attempt to update an existing or adding a new organization VDC network backed by NSX-T Data Center with an imported distributed port group sets the network in an Invalid state

If you set global default segment profiles to be applied to all VDC networks, an attempt to update an existing or add a new organization VDC network backed by NSX-T Data Center with an imported distributed port group sets the network in an Invalid state.
Moving a VM to a different provider VDC fails with an Internal Server Error message

If two provider VDCs are backed by different vCenter Server instances and you configure different names for their storage profiles, moving a VM between the provider VDCs fails with the following error.

Internal Server Error
Deleting a standalone VM fails with a The requested operation could not be executed on vApp error message after powering off the same VM in vCenter Server

If you power off a standalone VM directly in vCenter Server, an attempt to delete the same VM in VMware Cloud Director fails with an error message.

The requested operation could not be executed on vApp. Stop the vApp and try again.
VMware Cloud Director displays vApp templates that do not reside in the selected catalog

If you configure a local and a shared catalog with the same name and you attempt to review the vApp templates and media files from the local catalog, VMware Cloud Director displays the same from the shared catalog and vice versa.
An attempt to review the IPsec VPN Screen results in an Http failure response error message

If you configure more than 50 IPsec VPN tunnels on an edge gateway, when you navigate to the IPsec VPN screen, the UI displays an error message.

Http failure response for https://<public-url>/cloudapi/1.0.0/edgeGateways/urn:vcloud:gateway:5b240e42-3b84-4fb0-be13-d9344866b561/ipsec/tunnels?pageSize=128: 500 Server Error
The Edit Port Groups for <external-network> dialog box is empty

In a multisite environment, when you attempt to edit the configured distributed port group on an external network, the Edit Port Groups for <external-network> dialog box is empty and does not display any information.
An attempt to delete an organization that is configured with a data center group results in a Failed status

For an organization, if you configure a data center group, an attempt to delete this organization fails with a Failed status.
A user that is imported from an OIDC identity provider has read-only access

When logging in to VMware Cloud Director as a user that is imported from an OIDC identity provider, you have read-only access.
Suspending a VM through the VMware Cloud Director UI results in a partially suspended state of the VM

In the VMware Cloud Director Tenant Portal, when you suspend a VM, VMware Cloud Director does not undeploy the VM, and the VM becomes Partially Suspended instead of Suspended.
When you attempt to delete a stranded item in VMware Cloud Director by clicking OK on the Delete Standed Item window, the window becomes unresponsive

When you attempt to delete a stranded item in VMware Cloud Director by clicking OK on the Delete Standed Item window, the window becomes unresponsive. This issue occurs when your network connection to the VMware Cloud Director instance is slow. Fetching a stranded item might take up to five minutes, during which the UI is unresponsive. If you click the Cancel button, the window closes, but the deletion of the item is not cancelled.
The Customer Experience Improvement Program (CEIP) status is Enabled even after deactivating it during the installation of VMware Cloud Director

During the installation of VMware Cloud Director, if you deactivate the option to join the CEIP, after the installation completes, the CEIP status is active.
VMware Cloud Director displays incorrect values for the Application, Source, and Destination parameters of a firewall rule

If you create a firewall rule and add the rule to an NSX-T Data Center tier-1 edge gateway, VMware Cloud Director does not inherit the correct values for the Application, Source, and Destination parameters and displays them as Any.
Role name and description are localized in the VMware Cloud Director UI and can cause duplication of role names

The problem occurs because the UI translation does not affect the back end and API. You might create roles with the same names as the translated names which results in perceived duplicate roles in the UI and conflicts with the API usage of role names when creating service accounts.
Subscribing to an external catalog by using a proxy fails with an Unknown remote catalog error error message

If you use a proxy when attempting to subscribe to an external catalog whose IP address is not directly reachable, or is outside of the DMZ, the operation fails with an error message.

Unknown remote catalog error: Connect to <name>:443 failed: connect timed out.
Updating the storage policy for a VM fails with an Internal Server Error error message

If a VM resides on a single host cluster and the VM network is not backed by a distributed virtual port group, an attempt to update the storage policy of the VM fails with an error message.

Internal Server Error error
The left panel displays duplicate menu items for an edge gateway

If you log in to the VMware Cloud Director portals as a user with no Load Balancer View Only rights, the left panel portal displays duplicate menu items.
Creating an anti-affinity rule fails with a No valid hosts could be found due to datastore accessibility error message

When you attempt to create an anti-affinity rule, the operation fails with a No valid hosts could be found due to datastore accessibility error message.

This happens because the placement engine attempts to create the rule on a read-only datastore.
Editing the IPSec VPN tunnel fails with an IPSec VPN Tunnel with id XXXX not found on Edge Gateway error message

On an edge gateway, if you configure more than 25 IPSec VPN tunnels, editing one of the IPSec VPN tunnels fails with an error message.

IPSec VPN Tunnel with id XXXX not found on Edge Gateway
If you use fast cross vCenter Server vApp instantiation and then you delete the VM with prefix multi-vc-vm- that was created during the instantiation, this also deletes the VMX and VMDK files of the original VM source template of the instantiation

If you use fast cross vCenter Server vApp instantiation and then you delete the VM with prefix multi-vc-vm- that was created during the instantiation, this also deletes the VMX and VMDK files of the original VM source template that you used for the instantiation. This happens because when you use fast cross vCenter Server vApp instantiation to instantiate a VM, the source VM template which is located on vCenter Server A is registered with vCenter Server B, creating a VM with prefix multi-vc-vm- which spans across the two vCenter Server instances, while its VMX and VMDK files are stored with the original VM template on vCenter Server A. The multi-vc-vm- VM will be deleted either if it's deleted directly from vCenter Server or if it's imported to VMware Cloud Director with the Delete Source check box selected, and then deleted from VMware Cloud Director.
In the VMware Cloud Director UI, clicking Help in the top navigation bar does not lead you to the relevant product documentation

In the VMware Cloud Director UI, clicking Help in the top navigation bar does not lead you to the relevant product documentation. This happens because in VMware Cloud Director 10.4, the Help menu link is retrieved from a custom link, and the default value for the custom link is null.

Known Issues

New - When using the CloudAPI to create or update an organization, you cannot set to true the canPublish flag

When using the CloudAPI to create an organization or update an organization enabling it to publish catalogs, the canPublish field remains false, despite you setting the value to true. The legacy API is not affected.

Workaround: Use the VMware Cloud Director UI to activate or deactivate the option to Publish catalog externally for an organization.
New - VMware Cloud Director backup fails

If you use Ubuntu or Linux distributions that are based on Debian as NFS for the VMware Cloud Director appliance, the NFS server cannot be configured appropriately to support the creation of backups through the PostgreSQL user.
Workaround: Depending on the file that the appliance has, run the following commands from appliance's secure shell as the root user.
- If the appliance has the /opt/vmware/appliance/bin/create-db-backup file, run the following command.
sed -i '/PG_BACK_UP() {/,/}/ { /PG_BACK_UP() {/!{ /}/!d }}; /PG_BACK_UP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dump -v -Fc \$DBNAME" > \$DB_DUMP_PATH 2>> \$LOG_FILE' /opt/vmware/appliance/bin/create-db-backup
- If the appliance has the /opt/vmware/appliance/bin/create-backup.sh file, run the following commands.
sed -i '/DB_BACKUP() {/,/}/ { /DB_BACKUP() {/!{ /}/!d }}; /DB_BACKUP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dump -v -Fc \$DBNAME" > \$DB_DUMP_PATH 2>> \$LOG_FILE' /opt/vmware/appliance/bin/create-backup.sh

sed -i '/DB_USER_BACKUP() {/,/}/ { /DB_USER_BACKUP() {/!{ /}/!d }}; /DB_USER_BACKUP() {/ a\su - postgres -c "$VMWARE_POSTGRES_BIN\/pg_dumpall --roles-only | grep -e '\''CREATE ROLE vcloud;\\|ALTER ROLE vcloud WITH'\''" > \$BACKUP_DIR\/vcloud-user.sql' /opt/vmware/appliance/bin/create-backup.sh
New - After upgrading a VMware Cloud Director appliance, the management API and the management UI report an incorrect older version of the appliance

The problem occurs because the VMware Cloud Director appliance management API uses a different source of truth for obtaining the current version of the VMware Cloud Director appliance than the vamicli version --appliance command. This alternate source of truth is not always being updated during the appliance upgrade causing incorrect information to appear.

Workaround: Use the vamicli version --appliance command to verify the VMware Cloud Director appliance version.
New - When you deploy a VM from a template with a storage policy that includes a configured IOPS limit, after deployment, the VM disks do not have an IOPS limit configured or have a different IOPS limit

The problem occurs because the the I/O Operations Per Second (IOPS) limit set in the VM template overrides the storage policy's IOPS limit. For example, if the VM template does not have a configured IOPS value, after deployment, the VM disks do not have a configured IOPS limit.

Workaround: You can use vApp templates, or edit the VM after deployment.
New - When sharing vApps with users, you can navigate to nonexistent pages

When sharing vApps with users, the buttons to go to the previous or next page are available even though you are already on the respective first or last page. As a result, you can navigate to pages that do not exist and do not have content.

Workaround: None.
New - The VMware Cloud Director API does not return some provider VDCs as merge candidates

If you attempt to get merge candidates for a provider VDC and there are more provider VDCs in the system than the value specified in the page size query parameter, the merge candidate API only processes the first page size number of provider VDCs to check if they are merge candidates and ignores the other provider VDCs in the system.

Workaround: To ensure the VMware Cloud Director API processes all the provider VDCs, specify a page size greater than or equal to the number of provider VDCs in the system.
New - Deleting an organization in VMware Cloud Director UI fails with a You must delete this Organization's Application Port Profiles before you can delete the organization error
If application port profiles are created on an edge gateway associated with an organization, attempting to delete the organization fails. The issue occurs because VMware Cloud Director deletes the edge gateways before deleting the port profiles, which causes the following error.
```
com.vmware.vcloud.api.presentation.service.InvalidStateException: You must delete this Organization's Application Port Profiles before you can delete the organization.
```
Workaround: Use the VMware Cloud Director API to force delete an organization and to delete the stranded application port profiles associated with it. See Delete Stranded Application Port Profiles from VMware Cloud Director.
New - You cannot access the Service Provider Admin Portal and the VMware Cloud Director Tenant Portal after rebooting the VMware Cloud Director VM

If you reboot the VMware Cloud Director VM by using a method other than using the vSphere Client, for example, by using vSphere High Availability or VMware Host Client, you cannot access the Service Provider Admin Portal and the VMware Cloud Director Tenant Portal. The problem occurs because after the reboot, the deployment OVF parameters are deleted from the ovfEnv.xml file, and the cell cannot be accessed.

Workaround: Power off and then power on the VMware Cloud Director VM by using the vSphere Client.
New - The VMware Cloud Director proxy does not work when the proxy is configured for a vCenter instance registered with a URL containing port 443

When a vCenter instance is registered with a URL containing port 443, for example, https://vcenter.com:443, and you configure a proxy for that vCenter instance, VMware Cloud Director does not use the proxy and the java.net.SocketTimeoutException: connect timed out error appears in the logs.

Workaround: Remove the 443 port from the vCenter URL.
New - Changing the primary IP address of an NSX edge gateway fails with a Cannot allocate multiple primary IP addresses for NSX-T Edge Gateway error
When an NSX edge gateway has multiple IPv4 and IPv6 subnets, changing the primary IP address of the edge gateway using the VMware Cloud Director UI fails with an error similar to the following.
```
Error: [ e4dc8a76-86ea-408b-b2aa-27ea42305ec2 ] Cannot allocate multiple primary IP addresses for NSX-T Edge Gateway edgw1(com.vmware.vcloud.entity.gateway:8d17e279-2e4a-4452-b0c7-4a410a304374).
```
Workaround: You can use the VMware Cloud Director API.
New - VM does not receive the DNS Server IP addresses from the DHCP scope that is defined in the vApp network

When you connect a VM to a routed vApp network in DHCP IP mode, the VM does not receive the DNS addresses defined in the DHCP scope.

Workaround: Using NSX Manager, manually configure the DNS servers in the routed vApp network segment.
New - If you shut down the guest OS before deleting a VM, VMware Cloud Director cannot reuse the VM's IP address from the static pool of IP addresses
If you shut down the guest OS of a VM with a static pool IP allocation before deleting the VM, VMware Cloud Director does not release the IP address back into the IP pool, resulting in IP exhaustion. Creating a scale group or a VM with this IP address might fail with the following error.
```
No Static IP Pools or no free IP within any Static IP Pool to allocate to VM nic at index 0.
```
Workaround: Instead of shutting down the guest OS, power off the VM and then, delete it.
New - API clients throw Invalid mime type errors for responses from multisite VMware Cloud Director APIs
If the multisite field in the response header values specifies a list of organizations, the API client generates the following error.
```
org.springframework.util.InvalidMimeTypeException: Invalid mime type 
```
The issue occurs because the VMware Cloud Director API returns an illegal @ character in the MIME (Multipurpose Internet Mail Extensions) type headers of the response. You can ignore the error because VMware Cloud Director continues to function properly.
Workaround: None.
New - An attempt to view the task details for a deleted item as a user with custom role fails with a This operation is denied error message

In VMware Cloud Director Tenant Portal, if you login as a user with custom role that includes Administrator View rights and no manage rights, an attempt to view the task details for a deleted item by another user fails with an error message.

This operation is denied
Workaround:
- You must grant General: Administrator Control right to the custom user role.
- Alternatively, you can upgrade to VMware Cloud Director 10.4.2.2.
New - VMware Cloud Director UI and tasks are slow to load and complete
The Artemis message bus communication is not working and when you trigger operations from the UI, they can take up to 5 minutes to complete or might time out. The performance issues can affect operations such as powering on VMs and vApps, provider VDC creation, vApp deployment, and so on.

The log files might contain an error message, such as:
- a) Connection failure to <VCD Cell IP Address> has been detected: AMQ229014: Did not receive data from <something> within the 60,000ms
- b) Connection failure to /<VCD Cell IP Address>:61616 has been detected: AMQ219014: Timed out after waiting 30,000 ms
- c) Bridge is stopping, will not retry
- d) Local Member is not set at on ClusterConnection ClusterConnectionImp
Workaround:

For a) and b):
1. Verify that the VMware Cloud Director cells have network connectivity and can communicate with each other.
2. Restart the VMware Cloud Director cell that contains the error message.
For c) and d), restart the VMware Cloud Director cell that contains the error message.
New - The VMware Cloud Director appliance database disk resize script might fail if the backing SCSI disk identifier changes

The database disk resize script runs successfully only if the backing database SCSI disk ID remains the same. If the ID changes for any reason, the script might appear to run successfully but fails. The /opt/vmware/var/log/vcd/db_diskresize.log shows that the script fails with a No such file or directory error.
Workaround:
1. Log in directly or by using an SSH client to the primary cell as root.
2. Run the lsblk --output NAME,FSTYPE,HCTL command.
3. In the output, find the disk containing the database_vg-vpostgres partition and make note of its ID. The ID is under the HCTL column and has the following sample format 2:0:3:0.
4. In the db_diskresize.sh script, modify the partition ID with the ID from Step 3. For example, if the ID is 2:0:3:0, in line
```
echo 1 > /sys/class/scsi_device/2\:0\:2\:0/device/rescan
```
  you must change the ID to 2:0:3:0.
```
echo 1 > /sys/class/scsi_device/2\:0\:3\:0/device/rescan
```
5. Аfter saving the changes, manually re-invoke the resize script or reboot the appliance.
New - Deleting auto-discovered VMs from VMware Cloud Director moves the existing VMs in vApps to the StrandedItems folder and renames them

When you delete the auto-discovered VMs from VMware Cloud Director, the system moves the existing VMs that reside in vApps to the StrandedItems folder in vCenter Server and renames the vCenter Server managed VMs with a suffix before the VMs UUID, similar to vcentervm-1 (vm-uuid).

Workaround: None.
New - Upgrading to VMware Cloud Director 10.4.1 or later fails with a Fix postgres user home directory error
When you try to upgrade to VMware Cloud Director 10.4.1 or later, the upgrade fails. The update-postures-db.log contains the following error.
```
2023-05-15 16:38:01 | update-postgres-db.sh | Fix postgres user home directory
usermod: user postgres is currently used by process 17236
```
Other processes that are logged in as the postgres user on the VMware Cloud Director appliance might block the script that upgrades the PostgreSQL major version from 10 to 14.
Workaround:
1. Before starting the VMware Cloud Director upgrade, find any processes that are logged in as the postgres user on the VMware Cloud Director appliance by running ps -u postgres on the appliance.
2. Stop any process that the command returns by running kill -9 <PID>, where PID is the unique process identifier.
New - Users cannot log in to some organizations after migration to or from the system organization LDAP configuration

If you migrate a user from the shared system organization LDAP configuration to another IDP source, and the reverse, that user cannot log in to any organization other than the one doing the migration. For example, in a deployment where the system organization manages TenantA and TenantB and all organizations import User1 from the shared system organization LDAP configuration, if TenantA sets up a SAML configuration and migrates User1 from LDAP to SAML, then, User1 can log in to TenantA through SAML, but they cannot log in to the system organization or TenantB.

Workaround: None.
New - Fast cross vCenter Server instantiation of a vApp template with memory state fails

If you attempt an instantiation that is eligible for fast cross vCenter Server vApp template instantiation while preserving the memory state of the VMs within the vApp, the operation fails with an error message.

java.util.concurrent.ExecutionException: com.vmware.ssdc.util.LMException: Internal Server Error

Workaround: Move the template to a datastore that is not shared between the vCenter Server instances to avoid VMware Cloud Director performing fast vApp template instantiation.
New - Publishing a vRealize Orchestrator workflow to the VMware Cloud Director service library fails with an error message

When you attempt to publish a vRealize Orchestrator workflow, the operation fails with a 500 Server Error error message.

This happens because the API returns a large number of links for each individual tenant to which the workflow is published and causes an overflow in the HTTP headers.

Workaround: To publish the workflow, use CURL or POSTMAN to run an API request with increased HTTP header size limit.
New - VMware Cloud Director operations, such as powering a VM on and off takes longer time to complete

VMware Cloud Director operations, such as powering a VM on or off takes longer time to complete. The task displays a Starting virtual machine status and nothing happens.

The jms-expired-messages.logs log file displays an error.

RELIABLE:LargeServerMessage & expiration=

Workaround: None.
New - Creating an organization VDC template with NSX network provider type and provider gateways that uses IP spaces fails

When you attempt to create an organization VDC template with NSX network provider type and provider gatewaya that uses IP spaces, the operation fails with the following error. Error:Cannot support external Network that is utilizing IP Spaces. Only external networks with legacy IP blocks are supported.

Workaround: Create organization VDC templates with NSX network provider type and provider gateways that use legacy IP blocks.
New - Migrating VMs between organization VDCs might fail with an insufficient resource error

If VMware Cloud Director is running with vCenter Server 7.0 Update 3h or earlier, when relocating a VM to a different organization VDC, the VM migration might fail with an insufficient resource error even if the resources are available in the target organization VDC.

Workaround: Upgrade vCenter Server to version 7.0 Update 3i or later.
You lose the custom changes of the parameter settings from the postgresql.auto.conf file after upgrading to VMware Cloud Director 10.4.1

After upgrading to VMware Cloud Director 10.4.1, the system deletes all parameter settings from the postgresql.auto.conf file and you lose all custom changes previously applied to the file.

This happens because the VMware Cloud Director appliance version 10.4.1 uses and upgrades the embedded database to PostgreSQL 14 database.

Workaround: By following the steps in the Modify the PostgreSQL Configurations in the VMware Cloud Director Appliance procedure, reapply the changes of the parameter settings to the postgresql.auto.conf file. To help you, the system copies the pre-upgrade content from the postgresql.auto.conf file to the /var/vmware/vpostgres/current/pgdata/postgresql.auto.old file.

Important:
You must test any changes in a test environment. For example, if you include any deprecated parameters, such as wal_keep_segments, which is deprecated since PostgreSQL 12, PostgreSQL does not start and the upgrade fails.
You cannot select Tanzu Kubernetes version 2.0 or later when creating a vSphere with Tanzu Supervisor deployment cluster

As a tenant, when attempting to create a vSphere with Tanzu Supervisor deployment cluster, you cannot select a Tanzu Kubernetes cluster version 2.0 and later.

Workaround: To offer and use Tanzu Kubernetes 2.0 and later, use VMware Cloud Director Container Service Extension 4.0.
If you try to restore the VMware Cloud Director appliance with the console proxy certificates, the restore fails

In the VMware Cloud Director appliance management UI, if you want to restore the appliance and select the Console Proxy check box under Select the certificates to be restored on to this node from the selected backup, the restore fails.

Workaround: Starting with version 10.4, the console proxy and REST API use a single certificate. In version 10.4.1 and later, the legacy console proxy implementation is not supported and selecting the check box is not necessary. Repeat the restore procedure without selecting the Console Proxy check box.
When starting the VMware Cloud Director appliance, the message [FAILED] Failed to start Wait for Network to be Configured. See 'systemctl status systemd-networkd-wait-online.service' for details appears.

The message appears incorrectly and does not indicate an actual problem with the network. You can disregard the message and continue to use the VMware Cloud Director appliance as usual.

Workaround: None.
The VMware Cloud Director Tenant Portal UI does not display the IOPS limits and reservations for a vSAN storage policy

vSAN manages itself the IOPS limits on vSAN storage policies. As a result, the VMware Cloud Director Tenant Portal UI does not display the IOPS reservations and limits for a vSAN storage policy and you cannot modify their values.

Workaround: None.
VMware Cloud Director appliance upgrade fails with an invalid version error when FIPS mode is enabled

For VMware Cloud Director versions 10.3.x and later, when FIPS mode is enabled, VMware Cloud Director appliance upgrade fails with the following error.

Failure: Installation failed abnormally (program aborted), the current version may be invalid.
Workaround:
1. Before you upgrade the VMware Cloud Director appliance, deactivate FIPS Mode on the cells in the server group and the VMware Cloud Director appliance. See Activate or Deactivate FIPS Mode on the VMware Cloud Director Appliance.
2. Verify that the /etc/vmware/system_fips file does not exist on any appliance.
3. Upgrade the VMware Cloud Director appliance.
4. Enable FIPS mode again.
Restore from an appliance backup might fail with an Invalid command-line arguments. Missing argument for option: consoleproxy-cert error

If you run the clear-console-proxy-settingsCMT command before you take an appliance backup, then, if you choose to restore the console proxy certificate from the backup, the restore process fails with an Invalid command-line arguments. Missing argument for option: consoleproxy-cert error.

The issue occurs because the command to clear the console proxy settings removes the console proxy certificate, and the console proxy settings are missing for the backup. If the console proxy certificate is not in the backup, you cannot restore it.

If the console proxy settings were cleared, run the appliance restore without selecting to restore the console proxy certificate.
You can't view and edit the license type for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API

You can't view and edit the license for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API. This happens because in VMware Cloud Director 10.4, the Controller license type was replaced by a selection between a Standard and a Premium feature set at the Service Engine Group level to provide more flexibility.

Workaround: Use the supportedFeatureSet path for service engine groups and on edge gateways to activate and deactivate the available features.
You cannot create and use VMware Cloud Director VDC templates in VMware Cloud Director service environments that use VMware Cloud on AWS network pools

If you are using only a provider network pool that is backed by VMware Cloud on AWS for your provider VDC, you cannot create a VDC template and instantiate a VDC from a template. This happens because creating and instantiating VDC templates is supported only for provider VDCs backed by NSX-T Data Center and by NSX Data Center for vSphere. You can use VMware Cloud Director VDC templates with on-premises, Microsoft Azure VMware Solution, Oracle Cloud VMware Solution, or Google Cloud VMware Engine SDDCs.

Workaround: None.
Creating a new VM with encrypted vSAN storage policy fails with an Invalid storage policy for encryption operation error message

When creating a new VM, if you specify the storage policy of the VM as vSAN encrypted and the storage policy for the VM hard disk as both non-encrypted and non-vSAN, the operation fails with an error message.

Invalid storage policy for encryption operation
1. Specify the storage policies for the VM and the VM hard disk as vSAN encrypted.
2. After the VM deploys successfully, update the hard disk storage policy for the VM to non-encrypted and non-vSAN. For information, see Edit Virtual Machine Properties.
You cannot connect to VMware Cloud Director through VMware OVF Tool version 4.4.3 or earlier

When you attempt to connect to VMware Cloud Director through OVF Tool version 4.4.3 or earlier, this results in the following error. Error: No supported vCloud version was found. This happens because of an API behavior change in VMware Cloud Director 10.4 where the API does not return links to all the VDCs in an organization.

Workaround: Upgrade to OVF Tool 4.5.0. See VMware OVF Tool Release Notes.
You are unable to log in to VMware Cloud Director by using VMware PowerCLI 12.7.0 or earlier

When you attempt to log in to VMware Cloud Director by using VMware PowerCLI version 12.7.0 or earlier, this results in the following error. NOT_ACCEPTABLE: The request has invalid accept header: Invalid API version requested. This happens because VMware PowerCLI earlier than 13.0.0 do not support VMware Cloud Director API versions later than 33.0. See VMware Product Interoperability Matrix.

Workaround: Upgrade VMware PowerCLI to version 13.0.0.
VMware Cloud Director displays the old version for an upgraded vCenter Server instance

After you upgrade a vCenter Server instance to a newer version, in the list of vCenter Server instances, VMware Cloud Director still displays the old version for the upgraded instance.

Reset the connection between the vCenter Server instance and VMware Cloud Director. See Reconnect a vCenter Server Instance in VMware Cloud Director Service Provider Admin Portal Guide.
Refreshing the LDAP page in your browser does not take you back to the same page

In the Service Provider Admin Portal, refreshing the LDAP page in your browser takes you to the provider page instead of back to the LDAP page.

Workaround: None.
Mounting an NFS datastore from NetApp storage array fails with an error message during the initial VMware Cloud Director appliance configuration

During the initial VMware Cloud Director appliance configuration, if you configure an NFS datastore from NetApp storage array, the operation fails with an error message.

Backend validation of NFS failed with: is owned by an unknown user

Workaround: Configure the VMware Cloud Director appliance by using the VMware Cloud Director Appliance API.
The synchronization of a subscribed catalog times out while synchronizing large vApp templates

If an external catalog contains large vApp templates, synchronizing the subscribed catalog with the external catalog times out.Theissue occurs when the timeout setting is set to its default value of five minutes.

Workaround: Using the manage-config subcommand of the cell management tool, update the timeout configuration setting.

./cell-management-tool manage-config -n transfer.endpoint.socket.timeout -v [timeout-value]
In an IP prefix list, configuring any as the Network value results in an error message

When creating an IP prefix list, if you want to deny or accept any route and you configure the Network value as any, the dialog box displays an error message.

"any" is not a valid CIDR notation. A valid CIDR is a valid IP address followed by a slash and a number between 0 and 32 or 64, depending on the IP version.

Workaround: Leave the Network text box blank.
If you use vRealize Orchestrator 8.x, hidden input parameters in workflows are not populated automatically in the VMware Cloud Director UI

If you use vRealize Orchestrator 8.x, when you attempt to run a workflow through the VMware Cloud Director UI, hidden input parameters are not populated automatically in the VMware Cloud Director UI.
Workaround:To access the values of the workflow input parameters, you must create a vRealize Orchestrator action that has the same input parameter values as the workflow that you want to run.
1. Log in to the vRealize Orchestrator Client and navigate to Library>Workflows.
2. Select the Input Form tab and click Values on the right-hand side.
3. From the Value options drop-down menu, select External source, enter the Action inputs, and click Save. The possible input parameters are _vcd_orgName,_vcd_orgId, _vcd_userName, _vcd_isAdmin,_vcd_sessionToken.
4. Run the workflow in the VMware Cloud Director UI.
The vpostgres process in a standby appliance fails to start

The vpostgres process in a standby appliance fails to start and the PostgreSQL log shows an error similar to the following. FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the master server (its value was 16). This happens because PostgreSQL requires standby nodes to have the same max_worker_processes setting as the primary node. VMware Cloud Director automatically configures the max_worker_processes setting based on the number of vCPUs assigned to each appliance VM. If the standby appliance has fewer vCPUs than the primary appliance, this results in an error.

Workaround: Deploy the primary and standby appliances with the same number of vCPUs.
Upgrading from VMware Cloud Director 10.3.x to VMware Cloud Director 10.4.x results in an Connection to sfcbd lost error message

If you upgrade from VMware Cloud Director 10.3.x to VMware Cloud Director 10.4.x, the upgrade operation reports an error message.

Connection to sfcbd lost. Attempting to reconnect

Workaround: You can ignore the error message and continue with the upgrade.
When using FIPS mode, trying to upload OpenSSL-generated PKCS8 files fails with an error

OpenSSL cannot generate FIPS-complaint private keys. When VMware Cloud Director is in FIPS mode and you try to upload PKCS8 files generated using OpenSSL, the upload fails with a Bad request: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: ... not available: No such algorithm: ...error or salt must be at least 128 bits error.

Workaround: Deactivate the FIPS mode to upload the PKCS8 files.
Creation of Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in fails

When you create a Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in, you must select a Kubernetes version. Some of the versions in the drop-down menu are not compatible with the backing vSphere infrastructure. When you select an incompatible version, the cluster creation fails.

Workaround: Delete the failed cluster record and retry with a compatible Tanzu Kubernetes version. For information on the incompatibilities between Tanzu Kubernetes and vSphere, see Updating the vSphere with Tanzu Environment.
If you have any subscribed catalogs in your organization, when you upgrade VMware Cloud Director, the catalog synchronization fails

After upgrade, if you have subscribed catalogs in your organization, VMware Cloud Director does not trust the published endpoint certificates automatically. Without trusting the certificates, the content library fails to synchronize.

Workaround: Manually trust the certificates for each catalog subscription. When you edit the catalog subscription settings, a trust on first use (TOFU) dialog prompts you to trust the remote catalog certificate.

If you do not have the necessary rights to trust the certificate, contact your organization administrator.
After upgrading VMware Cloud Director and enabling the Tanzu Kubernetes cluster creation, no automatically generated policy is available and you cannot create or publish a policy

When you upgrade VMware Cloud Director to version 10.3.1 and vCenter Server to version 7.0.0d or later, and you create a provider VDC backed by a Supervisor Cluster, VMware Cloud Director displays a Kubernetes icon next to the VDC. However, there is no automatically generated Kubernetes policy in the new provider VDC. When you try to create or publish a Kubernetes policy to an organization VDC, no machine classes are available.

Workaround: Manually trust the corresponding Kubernetes endpoint certificates. See VMware knowledge base article 83583.
Entering a Kubernetes cluster name with non-Latin characters deactivates the Next button in the Create New Cluster wizard

The Kubernetes Container Clusters plug-in supports only Latin characters. If you enter non-Latin characters, the following error appears.

Name must start with a letter and only contain alphanumeric or hyphen (-) characters. (Max 128 characters).

Workaround: None.
NFS downtime can cause VMware Cloud Director appliance cluster functionalities to malfunction

If the NFS is unavailable due to the NFS share being full, becoming read only, and so on, can cause appliance cluster functionalities to malfunction. HTML5 UI is unresponsive while the NFS is down or cannot be reached. Other functionalities that might be affected are the fencing out of a failed primary cell, switchover, promoting a standby cell, and so on. For more information about setting up correctly the NFS shared storage, see Preparing the Transfer Server Storage for the VMware Cloud Director Appliance.
Workaround:
- Fix the NFS state so that it is not read-only.
- Clean up the NFS share if it is full.
Trying to encrypt named disks in vCenter Server version 6.5 or earlier fails with an error

For vCenter Server instances version 6.5 or earlier, if you try to associate new or existing named disks with an encryption enabled policy, the operation fails with a Named disk encryption is not supported in this version of vCenter Server. error.

Workaround: None.
A fast-provisioned virtual machine created on a VMware vSphere Storage APIs Array Integration (VAAI) enabled NFS array, or vSphere Virtual Volumes (VVols) cannot be consolidated

In-place consolidation of a fast provisioned virtual machine is not supported when a native snapshot is used. Native snapshots are always used by VAAI-enabled datastores, as well as by VVols. When a fast-provisioned virtual machine is deployed to one of these storage containers, that virtual machine cannot be consolidated .

Workaround: Do not enable fast provisioning for an organization VDC that uses VAAI-enabled NFS or VVols. To consolidate a virtual machine with a snapshot on a VAAI or a VVol datastore, relocate the virtual machine to a different storage container.
If you add an IPv6 NIC to a VM and then you add an IPv4 NIC to the same VM, the IPv4 north-south traffic breaks

Using the HTML5 UI, if you add an IPv6 NIC first or configure an IPv6 NIC as the primary NIC in a VM, and then you add an IPv4 NIC to the same VM, the IPv4 north-south communication breaks.

Workaround: First you must add the IPv4 NIC to the VM and then the IPv6 NIC.