VMware Cloud Director 10.4.2.1 | 22 JUN 2023 | Build 21954588 ( installed build 21954589)
Check for additions and updates to these release notes.
VMware Cloud Director 10.4.2.1 | 22 JUN 2023 | Build 21954588 ( installed build 21954589)
Check for additions and updates to these release notes.
VMware Cloud Director version 10.4.2.1 release includes the following:
Deactivation of the feedback button, when leaving the VMware Customer Experience Improvement Program (CEIP)
When you deactivate CEIP, the VMware Cloud Director Feedback button does not appear. For the feedback button to appear again, you must activate CEIP and log in again to VMware Cloud Director. For more information, see Join or Leave the VMware Customer Experience Improvement Program.
Bug fixes, updates the VMware Cloud Director appliance base OS and the VMware Cloud Director open-source components.
For information about system requirements and installation instructions, see VMware Cloud Director 10.4 Release Notes.
To access the full set of product documentation, go to VMware Cloud Director Documentation.
New - Importing a VM from vCenter Server by using the
move operation fails with an
Internal Server Error error message
If a VM has a configured
cloud-uuid property and you attempt to import this VM from vCenter Server by using the
move operation, the import fails with an
Internal Server Error error message.
New - Running an API call to query the user privileges on a set of entities results in an
Internal Server Error error message
When you use the VMware Cloud Director API to query the user privileges on a set of entities, the call fails with an
Internal Server Error error message.
New - Creating vApp from an OVF package fails with an
Invalid value '9' for attribute 'instanceId' on element 'BootOrderSection' error message
When you attempt to create a vApp from an OVF, the operation fails with an error message.
Invalid value 'ovf:/disk/vmdisk' for element 'HostResource'.Line 484: Invalid value '9' for attribute 'instanceId' on element 'BootOrderSection'..
This happens because when generating the OVF, VMware Cloud Director tends to overwrite the source input data and use its own instance IDs for the disk and CD ROM.
Deleting auto-discovered VMs from VMware Cloud Director moves the existing VMs in vApps to the
StrandedItems folder and renames them
When you delete the auto-discovered VMs from VMware Cloud Director, the system moves the existing VMs that reside in vApps to the
StrandedItems folder in vCenter Server and renames the vCenter Server managed VMs with a suffix before the VMs UUID, similar to
You cannot set the DNS suffix for an external network by using the VMware Cloud Director API
By using the VMware Cloud Director API, when you attempt to set the DNS information for an NSX-T segment backed external network, the suffix value is not applied.
Powering on a vApp fails with an
Unable to find the DVPortgroup corresponding to Network error message
In a multi-cluster environment, when you attempt to power on a vApp, the operation fails with an error message.
Unable to find the DVPortgroup corresponding to Network
This happens when the clusters in the multi-cluster environment are configured with dedicated hosts and DV switches and you configure a transport zone between the clusters. VMware Cloud Director deploys two distributed port groups on each cluster. During a vApp power on operation, VMware Cloud Director receives the distributed port group to host mapping for the host that belongs to different cluster than for the host where vApp resides.
An attempt to view the external networks on a provider VDC fails with a
NullPointerException error message
If your VMware Cloud Director environment contains an edge gateway that is not associated with an organization, an attempt to view the external networks on a provider VDC fails with a
NullPointerException error message.
Organization administrators can't add an NSX edge gateway to a data center group without the
Update Gateway right
Users with the
organization administrator role can't add an NSX edge gateway to a data center group unless they are assigned the
Update Gateway right. The
Update Gateway right provides tenants with additional rights which are not required for their role, such as
Enable DFW. This issue is resolved in this release. To add an edge gateway to a data center group, a user now needs to be assigned the
Configure VDC Group and
View Gateway rights.
New - API clients throw
Invalid mime type errors for responses from multisite VMware Cloud Director APIs
If the multisite field in the response header values specifies a list of organizations, the API client generates the following error.
org.springframework.util.InvalidMimeTypeException: Invalid mime type
The issue occurs because the VMware Cloud Director API returns an illegal @ character in the MIME (Multipurpose Internet Mail Extensions) type headers of the response. You can ignore the error because VMware Cloud Director continues to function properly.
New - The VMware Cloud Director appliance database disk resize script might fail if the backing SCSI disk identifier changes
The database disk resize script runs successfully only if the backing database SCSI disk ID remains the same. If the ID changes for any reason, the script might appear to run successfully but fails. The
/opt/vmware/var/log/vcd/db_diskresize.log shows that the script fails with a
No such file or directory error.
Log in directly or by using an SSH client to the primary cell as root.
lsblk --output NAME,FSTYPE,HCTL command.
In the output, find the disk containing the
database_vg-vpostgres partition and make note of its ID. The ID is under the HCTL column and has the following sample format
db_diskresize.sh script, modify the partition ID with the ID from Step 3. For example, if the ID is
2:0:3:0, in line
echo 1 > /sys/class/scsi_device/2\:0\:2\:0/device/rescan
you must change the ID to
echo 1 > /sys/class/scsi_device/2\:0\:3\:0/device/rescan
Аfter saving the changes, manually re-invoke the resize script or reboot the appliance.
Upgrading to VMware Cloud Director 10.4.1 or later fails with a
Fix postgres user home directory error
When you try to upgrade to VMware Cloud Director 10.4.1 or later, the upgrade fails. The
update-postures-db.log contains the following error.
2023-05-15 16:38:01 | update-postgres-db.sh | Fix postgres user home directory usermod: user postgres is currently used by process 17236
Other processes that are logged in as the
postgres user on the VMware Cloud Director appliance might block the script that upgrades the PostgreSQL major version from 10 to 14.
Before starting the VMware Cloud Director upgrade, find any processes that are logged in as the
postgres user on the VMware Cloud Director appliance by running
ps -u postgres on the appliance.
Stop any process that the command returns by running
kill -9 <PID>, where PID is the unique process identifier.
Users cannot log in to some organizations after migration to or from the system organization LDAP configuration
If you migrate a user from the shared system organization LDAP configuration to another IDP source, and the reverse, that user cannot log in to any organization other than the one doing the migration. For example, in a deployment where the system organization manages
TenantB and all organizations import
User1 from the shared system organization LDAP configuration, if
TenantA sets up a SAML configuration and migrates
User1 from LDAP to SAML, then,
User1 can log in to
TenantA through SAML, but they cannot log in to the system organization or
When starting the VMware Cloud Director appliance, the message
[FAILED] Failed to start Wait for Network to be Configured. See 'systemctl status systemd-networkd-wait-online.service' for details appears
The message appears incorrectly and does not indicate an actual problem with the network. You can disregard the message and continue to use the VMware Cloud Director appliance as usual.
You cannot create VMware Cloud Director VDC templates in VMware Cloud Director service environments
VMware Cloud Director service does not support Virtual Data Center (VDC) templates. You can use VDC templates on environments with provider VDCs with an NSX network provider type or an NSX Data Center for vSphere provider type. You cannot use VDC templates on VMware Cloud Director service environments because the provider VDCs have the VMC network provider type.
Creating an organization VDC template with NSX network provider type and provider gateways that uses IP spaces fails
When you attempt to create an organization VDC template with NSX network provider type and provider gateway that uses IP spaces, the operation fails with the following error.
Error:Cannot support external Network that is utilizing IP Spaces. Only external networks with legacy IP blocks are supported.
Workaround: Create organization VDC templates with NSX network provider type and provider gateways that use legacy IP blocks.
Changing the storage policy on a virtual disk of a VM fails with a
The operation failed because no suitable resource was found error message
If the virtual disk of a VM resides on a remote vSAN datastore, changing the storage policy of the virtual disk results in an error message.
The operation failed because no suitable resource was found
Workaround: To move the VM to a different storage policy, change the virtual disk storage policy to
VM default policy and then change the VM storage policy to the desired storage policy.
VMware Cloud Director shows an empty value for the IOPS limit for a VM disk with VC-IOPS enabled storage policy
If you apply a VC-IOPS enabled storage policy with custom reservation, limit, and shares, on a VM disk, VMware Cloud Director displays the values for IOPS reservations, but displays the IOPS limit as empty. This happens because vCenter Server 8U1 introduces a new mechanism for Storage I/O Control (SIOC) which no longer sets the IOPS limit as a VM disk property.
You cannot create a disabled organization using the legacy VMware Cloud Director API
Attempting to use the legacy VMware Cloud Director API organization creation endpoint
POST [vcd_public_endpoint]/api/admin/orgs to create a disabled organization results in a
400 BadRequestException containing the following snippet:
<Error ... stackTrace="com.vmware.vcloud.api.presentation.service.BadRequestException: Unexpected error. unexpected end of subtree
Workaround: Use the VMware Cloud Director OpenAPI endpoint to create a disabled organization. Alternatively, you can use the UI, OpenAPI, or legacy API to create an enabled organization and disable it after creation.
Creating an organization VDC Kubernetes policy with provider gateways that uses IP spaces fails
If you configure an IP space backed provider gateway and you create a VDC and an edge gateway based on the same IP space, an attempt to create a Kubernetes policy for this VDC fails with an error message.
com.vmware.ssdc.util.LMException: Index 0 out of bounds for length 0
This happens because the IP space backed edge gateways are not associated with a primary IP address, which is required for the creation of SNAT by the Kubernetes policy.
Workaround: Create VDC and edge gateways with NSX network provider type and provider gateways that use legacy IP blocks.
If you try to restore the VMware Cloud Director appliance with the console proxy certificates, the restore fails
In the VMware Cloud Director appliance management UI, if you want to restore the appliance and select the Console Proxy check box under Select the certificates to be restored on to this node from the selected backup, the restore fails.
Workaround: Starting with version 10.4, the console proxy and REST API use a single certificate. In version 10.4.1 and later, the legacy console proxy implementation is not supported and selecting the check box is not necessary. Repeat the restore procedure without selecting the Console Proxy check box.
You cannot select Tanzu Kubernetes version 2.0 or later when creating a TKGs cluster
As a tenant, when attempting to create a TKGs cluster, you cannot select a Tanzu Kubernetes cluster version 2.0 and later.
Workaround: To offer and use Tanzu Kubernetes 2.0 and later, use VMware Cloud Director Container Service Extension 4.0.
Migrating VMs between organization VDCs might fail with an insufficient resource error
If VMware Cloud Director is running with vCenter Server 7.0 Update 3h or earlier, when relocating a VM to a different organization VDC, the VM migration might fail with an insufficient resource error even if the resources are available in the target organization VDC.
Workaround: Upgrade vCenter Server to version 7.0 Update 3i or later.
VMs become non-compliant after converting a reservation pool VDC into a flex organization VDC
In an organization VDC with a reservation pool allocation model, if some of the VMs have nonzero reservation for CPU and Memory, non-unlimited configuration for CPU and Memory, or both, after converting into a flex organization VDC, these VMs become non-compliant. If you attempt to make the VMs compliant again, the system applies an incorrect policy for the reservation and limit and sets the CPU and Memory reservations to zero and the limits to Unlimited.
A system administrator must create a VM sizing policy with the correct configuration.
A system administrator must publish the new VM sizing policy to the converted flex organization VDC.
The tenants can use the VMware Cloud Director API or the VMware Cloud Director Tenant Portal to assign the VM sizing policy to the existing virtual machines in the flex organization VDC.
The VMware Cloud Director Tenant Portal UI does not display the IOPS limits and reservations for a vSAN storage policy
vSAN manages itself the IOPS limits on vSAN storage policies. As a result, the VMware Cloud Director Tenant Portal UI does not display the IOPS reservations and limits for a vSAN storage policy and you cannot modify their values.
VMware Cloud Director appliance upgrade fails with an invalid version error when FIPS mode is enabled
For VMware Cloud Director versions 10.3.x and later, when FIPS mode is enabled, VMware Cloud Director appliance upgrade fails with the following error.
Failure: Installation failed abnormally (program aborted), the current version may be invalid.
Before you upgrade the VMware Cloud Director appliance, deactivate FIPS Mode on the cells in the server group and the VMware Cloud Director appliance. See Activate or Deactivate FIPS Mode on the VMware Cloud Director Appliance.
Verify that the
/etc/vmware/system_fips file does not exist on any appliance.
Upgrade the VMware Cloud Director appliance.
Enable FIPS mode again.
Restore from an appliance backup might fail with an
Invalid command-line arguments. Missing argument for option: consoleproxy-cert error
If you run the
clear-console-proxy-settingsCMT command before you take an appliance backup, then, if you choose to restore the console proxy certificate from the backup, the restore process fails with an
Invalid command-line arguments. Missing argument for option: consoleproxy-cert error.
The issue occurs because the command to clear the console proxy settings removes the console proxy certificate, and the console proxy settings are missing for the backup. If the console proxy certificate is not in the backup, you cannot restore it.
If the console proxy settings were cleared, run the appliance restore without selecting to restore the console proxy certificate.
The VMware Cloud Director console proxy, uploading OVFs and media, and powering on a VM fail
VMware Cloud Director 10.4 enhances SSL connectivity to all vSphere infrastructure components, including ESXi, by incorporating the vSphere Certificate Authority (CA) into the VMware Cloud Director trust mechanisms. In certain cases, the vSphere endpoint and the vSphere CA use different trust anchors and VMware Cloud Director must trust more than one trust anchor from vSphere. If the vSphere CA is not trusted, some VMware Cloud Director features do not work.
To complete the vSphere integration, refer to KB 78885. You can also trust all the necessary certificates by running the
trust-infra-certs CMT command. See Import Endpoints Certificates from vSphere Resources.
You can't view and edit the license type for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API
You can't view and edit the license for your previously registered NSX Advanced Load Balancer Controller instances in the VMware Cloud Director API. This happens because in VMware Cloud Director 10.4, the Controller license type was replaced by a selection between a Standard and a Premium feature set at the Service Engine Group level to provide more flexibility.
Workaround: Use the
supportedFeatureSet path for service engine groups and on edge gateways to activate and deactivate the available features.
The Delete Stranded Item window becomes unresponsive
When you attempt to delete a stranded item in VMware Cloud Director by clicking OK on the Delete Stranded Item window, the window becomes unresponsive. This issue occurs when your network connection to the VMware Cloud Director instance is slow. Fetching a stranded item might take up to five minutes, during which the UI is unresponsive. If you click the Cancel button, the window closes, but the deletion of the item is not cancelled.
Workaround: Wait for the window to close on its own.
You cannot create a VDC template and instantiate a VDC from a template if you are using only a VMware Cloud on AWS network pool for your provider VDC
If you are using only a provider network pool that is backed by VMware Cloud on AWS for your provider VDC, you cannot create a VDC template and instantiate a VDC from a template.
This happens because creating and instantiating VDC templates is supported only for provider VDCs backed by NSX-T Data Center and by NSX Data Center for vSphere.
Creating a new VM with encrypted vSAN storage policy fails with an
Invalid storage policy for encryption operation error message
When creating a new VM, if you specify the storage policy of the VM as vSAN encrypted and the storage policy for the VM hard disk as both non-encrypted and non-vSAN, the operation fails with an error message.
Invalid storage policy for encryption operation
Specify the storage policies for the VM and the VM hard disk as vSAN encrypted.
After the VM deploys successfully, update the hard disk storage policy for the VM to non-encrypted and non-vSAN. For information, see Edit Virtual Machine Properties.
You cannot connect to VMware Cloud Director through VMware OVF Tool version 4.4.3 or earlier
When you attempt to connect to VMware Cloud Director through OVF Tool version 4.4.3 or earlier, this results in the following error.
Error: No supported vCloud version was found. This happens because of an API behavior change in VMware Cloud Director 10.4 where the API does not return links to all the VDCs in an organization.
Workaround: Upgrade to OVF Tool 4.5.0. See VMware OVF Tool Release Notes.
You are unable to log in to VMware Cloud Director by using VMware PowerCLI 12.7.0 or earlier
When you attempt to log in to VMware Cloud Director by using VMware PowerCLI version 12.7.0 or earlier, this results in the following error.
NOT_ACCEPTABLE: The request has invalid accept header: Invalid API version requested. This happens because VMware PowerCLI earlier than 13.0.0 do not support VMware Cloud Director API versions later than 33.0. See VMware Product Interoperability Matrix.
Workaround: Upgrade VMware PowerCLI to version 13.0.0.
VMware Cloud Director displays the old version for an upgraded vCenter Server instance
After you upgrade a vCenter Server instance to a newer version, in the list of vCenter Server instances, VMware Cloud Director still displays the old version for the upgraded instance.
Reset the connection between the vCenter Server instance and VMware Cloud Director. See Reconnect a vCenter Server Instance in VMware Cloud Director Service Provider Admin Portal Guide.
Refreshing the LDAP page in your browser does not take you back to the same page
In the Service Provider Admin Portal, refreshing the LDAP page in your browser takes you to the provider page instead of back to the LDAP page.
Mounting an NFS datastore from NetApp storage array fails with an error message during the initial VMware Cloud Director appliance configuration
During the initial VMware Cloud Director appliance configuration, if you configure an NFS datastore from NetApp storage array, the operation fails with an error message.
Backend validation of NFS failed with: is owned by an unknown user
Workaround: Configure the VMware Cloud Director appliance by using the VMware Cloud Director Appliance API.
The synchronization of a subscribed catalog times out while synchronizing large vApp templates
If an external catalog contains large vApp templates, synchronizing the subscribed catalog with the external catalog times out. The issue occurs when the timeout setting is set to its default value of five minutes.
Workaround: Using the
manage-config subcommand of the cell management tool, update the timeout configuration setting.
./cell-management-tool manage-config -n transfer.endpoint.socket.timeout -v [timeout-value]
In an IP prefix list, configuring
any as the Network value results in an error message
When creating an IP prefix list, if you want to deny or accept any route and you configure the Network value as
any, the dialog box displays an error message.
"any" is not a valid CIDR notation. A valid CIDR is a valid IP address followed by a slash and a number between 0 and 32 or 64, depending on the IP version.
Workaround: Leave the Network text box blank.
The vpostgres process in a standby appliance fails to start
vpostgres process in a standby appliance fails to start and the PostgreSQL log shows an error similar to the following.
FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the master server (its value was 16). This happens because PostgreSQL requires standby nodes to have the same
max_worker_processes setting as the primary node. VMware Cloud Director automatically configures the
max_worker_processes setting based on the number of vCPUs assigned to each appliance VM. If the standby appliance has fewer vCPUs than the primary appliance, this results in an error.
Workaround: Deploy the primary and standby appliances with the same number of vCPUs.
VMware Cloud Director API calls to retrieve vCenter Server information return a URL instead of a UUID
The issue occurs with vCenter Server instances that failed the initial registration with VMware Cloud Director version 10.2.1 and earlier. For those vCenter Server instances, when you make API calls to retrieve the vCenter Server information, the VMware Cloud Director API incorrectly returns a URL instead of the expected UUID.
Workaround: Reconnect to the vCenter Server instance to VMware Cloud Director.
Upgrading from VMware Cloud Director 10.3.x to VMware Cloud Director 10.4.x results in an
Connection to sfcbd lost error message
If you upgrade from VMware Cloud Director 10.3.x to VMware Cloud Director 10.4.x, the upgrade operation reports an error message.
Connection to sfcbd lost. Attempting to reconnect
Workaround: You can ignore the error message and continue with the upgrade.
When using FIPS mode, trying to upload OpenSSL-generated PKCS8 files fails with an error
OpenSSL cannot generate FIPS-complaint private keys. When VMware Cloud Director is in FIPS mode and you try to upload PKCS8 files generated using OpenSSL, the upload fails with a
Bad request: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: ... not available: No such algorithm: ...error or
salt must be at least 128 bits error.
Workaround: Deactivate the FIPS mode to upload the PKCS8 files.
Creation of Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in fails
When you create a Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in, you must select a Kubernetes version. Some of the versions in the drop-down menu are not compatible with the backing vSphere infrastructure. When you select an incompatible version, the cluster creation fails.
Workaround: Delete the failed cluster record and retry with a compatible Tanzu Kubernetes version. For information on the incompatibilities between Tanzu Kubernetes and vSphere, see Updating the vSphere with Tanzu Environment.
If you have any subscribed catalogs in your organization, when you upgrade VMware Cloud Director, the catalog synchronization fails
After upgrade, if you have subscribed catalogs in your organization, VMware Cloud Director does not trust the published endpoint certificates automatically. Without trusting the certificates, the content library fails to synchronize.
Workaround: Manually trust the certificates for each catalog subscription. When you edit the catalog subscription settings, a trust on first use (TOFU) dialog prompts you to trust the remote catalog certificate.
If you do not have the necessary rights to trust the certificate, contact your organization administrator.
After upgrading VMware Cloud Director and enabling the Tanzu Kubernetes cluster creation, no automatically generated policy is available and you cannot create or publish a policy
When you upgrade VMware Cloud Director to version 10.3.1 and vCenter Server to version 7.0.0d or later, and you create a provider VDC backed by a Supervisor Cluster, VMware Cloud Director displays a Kubernetes icon next to the VDC. However, there is no automatically generated Kubernetes policy in the new provider VDC. When you try to create or publish a Kubernetes policy to an organization VDC, no machine classes are available.
Workaround: Manually trust the corresponding Kubernetes endpoint certificates. See VMware knowledge base article 83583.
Entering a Kubernetes cluster name with non-Latin characters deactivates the Next button in the Create New Cluster wizard
The Kubernetes Container Clusters plug-in supports only Latin characters. If you enter non-Latin characters, the following error appears.
Name must start with a letter and only contain alphanumeric or hyphen (-) characters. (Max 128 characters).
NFS downtime can cause VMware Cloud Director appliance cluster functionalities to malfunction
If the NFS is unavailable due to the NFS share being full, becoming read only, and so on, can cause appliance cluster functionalities to malfunction. HTML5 UI is unresponsive while the NFS is down or cannot be reached. Other functionalities that might be affected are the fencing out of a failed primary cell, switchover, promoting a standby cell, and so on. For more information about setting up correctly the NFS shared storage, see Preparing the Transfer Server Storage for the VMware Cloud Director Appliance.
Fix the NFS state so that it is not
Clean up the NFS share if it is full.
Trying to encrypt named disks in vCenter Server version 6.5 or earlier fails with an error
For vCenter Server instances version 6.5 or earlier, if you try to associate new or existing named disks with an encryption enabled policy, the operation fails with a
Named disk encryption is not supported in this version of vCenter Server. error.
A fast-provisioned virtual machine created on a VMware vSphere Storage APIs Array Integration (VAAI) enabled NFS array, or vSphere Virtual Volumes cannot be consolidated
In-place consolidation of a fast provisioned virtual machine is not supported when a native snapshot is used. Native snapshots are always used by VAAI-enabled datastores, as well as by vSphere Virtual Volumes. When a fast-provisioned virtual machine is deployed to one of these storage containers, that virtual machine cannot be consolidated .
Workaround: Do not enable fast provisioning for an organization VDC that uses VAAI-enabled NFS or vSphere Virtual Volumes. To consolidate a virtual machine with a snapshot on a VAAI or a vSphere Virtual Volumes datastore, relocate the virtual machine to a different storage container.
If you add an IPv6 NIC to a VM and then you add an IPv4 NIC to the same VM, the IPv4 north-south traffic breaks
Using the HTML5 UI, if you add an IPv6 NIC first or configure an IPv6 NIC as the primary NIC in a VM, and then you add an IPv4 NIC to the same VM, the IPv4 north-south communication breaks.
Workaround: First you must add the IPv4 NIC to the VM and then the IPv6 NIC.