VMware Cloud Director 10.3.2a Release Notes

VMware Cloud Director 10.3.2a | 23 FEB 2022 | Build 19375095 (installed build 19375051) Check for additions and updates to these release notes.

What's in this Document

• What's New
• System Requirements and Installation
• Documentation
• Previous Releases of VMware Cloud Director 10.3.x
• Resolved Issues
• Known Issues

IMPORTANT: VMware removed VMware Cloud Director 10.3.2 from all sites on February 23, 2022 due to the snapshot revert operation leading to data loss. Build 19375095 (installed build 19375051) replaces build 19173640 (installed build 19173133). For more information, see Reverting a snapshot of a VM in a multi-VM vApp will revert snapshots for all VMs in the same vApp (87696).

What's New

• NSX-T Manager Segment Profile Templates. As a service provider, you can define segment profile templates to be applied to organization VDC networks and to vApp networks upon their creation or as an update. A segment profile template is a set of NSX-T Data Center segment profiles that are created in NSX-T Manager and used by VMware Cloud Director for configuring network properties. Depending on your environment needs, you can set segment profile templates that are applied at different levels. See Using NSX-T Manager Segment Profile Templates in VMware Cloud Director. 

• Non-Distributed Routing with NSX-T Data Center. VMware Cloud Director supports non-distributed routing for organization VDC networks backed by NSX-T Data Center. As a service provider, you can configure an NSX-T Data Center edge gateway to allow non-distributed routing and you can connect routed organization VDC networks directly to a tier-1 service router, forcing all VM traffic for a specific network through the service router.

  If you are migrating your networking infrastructure from NSX Data Center for vSphere to NSX-T Data Center and you were using your organization VDC network gateway address as a DNS server address, you can use non-distributed routing to configure your organization VDC network that is backed by NSX-T Data Center to also use its network gateway's IP address as a DNS server address.

  You can also use the non-distributed routing feature to create firewall rules and isolate east-west traffic between organization VDC networks that are connected to the same NSX-T Data Center edge gateway. See Using Non-Distributed Routing with NSX-T Data Center.

• QoS Rate Limits Configuration on an NSX-T Edge Gateway. You can configure QoS (Quality of Service) rate limits to ingress and egress traffic on NSX-T Data Center edge gateways. Ingress and egress rate limits control the inbound and outbound traffic from the edge gateway by determining whether the size of network packets meets predefined criteria, such as committed bandwidth and burst size. See Configure QoS Rate Limits on an NSX-T Edge Gateway.

• NVIDIA vGPU Support. ​To enable self-service for tenant deployment of workloads that require access to NVIDIA vGPU devices, cloud providers can configure and advertise to tenants vGPU policies. The vGPU policies define the placement and sizing settings of VMs that require vGPU resources. See Understanding VM Sizing, VM Placement, and vGPU Policies.

• Automatic OIDC Key Refresh. You can configure VMware Cloud Director to automatically refresh your OIDC key configurations from the JWKS endpoint you provide. See Configure Your System to Use an OpenID Connect Identity Provider.

• VMware Cloud Director Appliance Backup Delete Function. You can delete specific backup files using the VMware Cloud Director appliance management UI or the VMware Cloud Director appliance API. See Backup and Restore of VMware Cloud Director Appliance. 

  Note: The VMware Cloud Director 10.3.1 and earlier backups are incompatible with VMware Cloud Director appliance 10.3.2a and later. For VMware Cloud Director appliance 10.3.2a and later, the backup name is in the format backup-date-time-format.zip. If you do not expect to restore a system to version 10.3.1 or earlier, you can delete these backups and their directory. The directory is located at /opt/vmware/vcloud-director/data/transfer/backups.

• NSX Data Center for vSphere to NSX-T Data Center Мigration Enhancements. The MoveVApp API supports routed vApp network configuration.

System Requirements and Installation

For information about system requirements and installation instructions, see VMware Cloud Director 10.3 Release Notes.

Supported LDAP Servers

Note: VMware Cloud Director 10.3 and later supports Windows Server 2019 as a platform for the LDAP Service.

You can import users and groups to VMware Cloud Director from the following LDAP services.

Platform	LDAP Service	Authentication Methods
Windows Server 2012	Active Directory	Simple, Simple SSL
Windows Server 2016	Active Directory	Simple, Simple SSL
Windows Server 2019	Active Directory	Simple, Simple SSL
Linux	OpenLDAP	Simple, Simple SSL

Documentation

To access the full set of product documentation, go to VMware Cloud Director Documentation.

Previous Releases of VMware Cloud Director 10.3.x

VMware Cloud Director 10.3.1 Release Notes

VMware Cloud Director 10.3 Release Notes

Resolved Issues

• New VMware Cloud Director UI and tasks are slow to load and complete

  The Artemis message bus communication is not working and when you trigger operations from the UI, they can take up to 5 minutes to complete or might time out. The performance issues can affect operations such as powering on VMs and vApps, provider VDC creation, vApp deployment, and so on.

• New In a vApp with multiple VMs, reverting a VM to a snapshot causes all VMs in the vApp to revert to their snapshots

  If multiple VMs in a vApp have snapshots, reverting any of the VMs to a snapshot triggers the revert operation on all VMs in the vApp.

  This issue is resolved in VMware Cloud Director version 10.3.2a. If you are using VMware Cloud Director 10.3.2 build 19173640 (installed build 19173133), to resolve the issue, upgrade to VMware Cloud Director 10.3.2a build 19375095 (installed build 19375051).

  For a mitigation strategy option that you can use until you upgrade to VMware Cloud Director 10.3.2a, see Reverting a snapshot of a VM in a multi-VM vApp will revert snapshots for all VMs in the same vApp (87696).

• New The Container filter data grid does not display a VM when migrating VMs between resource pools

  When migrating VMs between resource pools on a provider VDC, if you use the Container filter to find a VM by using a name which is specific to the vApp name, the data grid displays an empty result.

• VMware Cloud Director does not display the values for the LDAP synchronization settings

  In the VMware Cloud Director Service Provider Admin Portal, after configuring the LDAP synchronization settings, the LDAP synchronization page does not display the settings you configured.

• The LDAP Synchronization Settings tab in the VMware Cloud Director Service Provider Admin Portal does not display the Edit button

  On the LDAP Synchronization Settings tab, the HTML5 UI does not display the Edit button and you cannot edit the LDAP settings for your organization.

• Importing Elliptic-curve cryptography (ECC) key to the certificate library fails with an error message

  When importing a certificate to the certificate library, if you import ECC-based key, the operation fails with an error message.
  Bad request: java.security.InvalidKeyException: No installed provider supports this key:sun.security.ec.ECPrivateKeyImpl

• Importing a private key in unsupported format fails with a misleading error message

  When importing a certificate to the certificate library, if the private key is in unsupported format, the operation fails with a misleading error message.
  Cannot parse certificate.

• The Edit button on the Hard Disks page is deactivated for a VM that you import from vCenter Server

  In vCenter Server, if you configure a VM with virtual hardware version 12, after importing the VM to VMware Cloud Director, the Edit button on the Hard Disks page for this VM is deactivated and you cannot change its hardware properties.
  This happens because VMware Cloud Director does not support virtual machine hardware version 12.

• Updating the Gold Master status of a vApp template in the original catalog does not change the Gold Master status for the same vApp template in the subscribed catalog

  After the first synchronization of a subscribed catalog with the original catalog, if you change the Gold Master status of a vApp template in the original catalog and synchronize the catalogs again, the Gold Master status of the vApp template does not change in the subscribed catalog.

• You cannot update the DNS settings of an organization VDC network

  When you attempt to edit the DNS settings of an organization VDC network backed by NSX Data Center for vSphere nothing happens and the DNS configurations remain unchanged. The vcloud-container-debug.log file displays the following error message.
  Failed to update NSX-V orgVdc network
java.lang.NullPointerException

• The VMware Cloud Director API returns an incorrect value for the IopsAllocated parameter in an IOPS enabled storage policy

  If you enable the IOPS setting for a storage policy and you assign VMs and named disks to this storage policy, running the GET /api/admin/vdcStorageProfile/{id} API request returns an incorrect value for the IopsAllocated parameter.

• After taking a VM snapshot, VMware Cloud Director displays an incorrect storage value for the same VM

  If you specify multiple storage policies for a VM, after taking a snapshot of this VM, the VMware Cloud Director UI displays an incorrect value for the VM storage usage.

• The Create vApp from Template wizard does not display the list of networks

  In a vApp template, if you enable the customization of a VM and configure only direct vApp network type, when you attempt to create new vApp by using this vApp template, the Configure Networking page of the Create vApp from Template wizard does not display the list of networks.

• When you use the VMware Cloud Director API to create a VM from a template and you don't specify a default storage policy, if there is no default storage policy set for the template, the newly created VM attempts to use the storage policy of the source template itself​

  When you use the VMware Cloud Director API to create a VM from a template and you don't specify a default storage policy, if there is no default storage policy set for the template, the newly created VM attempts to use the storage policy of the source template itself instead of using the storage policy of the organization VDC in which you are deploying it.

• After Add and Remove a VDC from a VDC group operations, the status of an edge gateway that is shared across all data centers in the VDC group is displayed as Busy

  If a VDC is configured with a provider VDC Kubernetes policy, if you add or remove the VDC from a VDC group, on the Edge Gateway page, the status of the edge gateway that is shared across all data centers in the VDC group is displayed as Busy and you cannot edit this edge gateway.

• Deleting a cell in failover mode results in a high CPU and memory usage

  During a failover process, if the entry for a failed cell is deleted from the failed_cells table before the failover process finishes, the attempts to complete the failover are repeatedly failing and restarting. As a result, you observe high CPU and memory usage.

• In the Upload Media dialog box, clicking OK does not deactivate the button

  In the Upload Media dialog box, after uploading a media file and clicking OK, the UI does not display information about the upload operation and the OK button is still active. As a result, you can upload the same media file multiple times.

•  While a vApp is transitioning its state, selecting multiple vApps automatically selects a wrong vApp

  While a vApp is transitioning its state, if you toggle on the Multiselect option and select multiple vApps before the list of vApps is automatically refreshed, the system adds a random vApp to the selection list.

• After viewing the services of an edge gateway, the Services dialog box keeps displaying the information from the initial edge gateway for all other edge gateways

  After opening the Services screen for an edge gateway, if you select another edge gateway and attempt to view its services, the UI displays the information from the initial edge gateway for all other edge gateways.
  This happens because the UI caches the information about the initial edge gateway until you refresh the browser.

• The Monitor tab for a scale group cannot display more than 50 entries in the list of tasks

  The Monitor tab for a scale group does not have pagination and displays a maximum of 50 entries. If the list of tasks contains more than 50 entries, the UI only displays the first 50 entries.

• A rule takes longer time than configured to trigger the growing or shrinking of scale groups​

  After you configure the duration period  for which a condition must be valid to trigger an auto scale rule, the growing or shrinking of scale groups happens with a huge delay.

• Adding a virtual machine hard disk results in transaction timeout expired error message

  If you configure a virtual machine with 32 virtual hard disks, adding the 33rd virtual hard disk fails with an error message.
  transaction timeout expired

• Virtual machine with shut down guest operating system still consumes compute resources

  If you shut down the guest operating system of a VM, the VM continues to consume compute resources. This happens because VMware Cloud Director recognizes the VM as partially powered off.

• Deleting a vApp template fails with a Could not find object error message

  If a VM resides in a vApp template and you delete the VM from vCenter Server, when you attempt to delete the vApp template from an organization catalog in VMware Cloud Director, the operation fails with an error message.
  Could not find object moref 'vm-xxxx' and VC ID 'XXXX' in inventory category 'VirtualMachine'

• In the vSphere Client, creating a subscribed content library originating from a published VMware Cloud Director catalog fails with an error message​

  In the vSphere Client, when you create a subscribed content library that originates from a published VMware Cloud Director catalog, the operation fails with an error message​.
  Operation failed! The object or item referred to could not be found.

  The log message in the vCenter Server log files displays error similar to Connect to VCSP server by URL https://{remove-vcd-host-name}/vcsp/lib/{lib-uuid}/ failed.

• Using the VMware Cloud Director Quick Search to find entities from the tenant portal results in Authentication Failed error message

  In the VMware Cloud Director Service Provider Admin Portal, if you use the quick search to find vApps and VMs in the tenant portal and you click on the search result, the operation fails with an error message.
  Authentication Failed.

• You cannot change the guest OS customization properties for a virtual machine configured with Other 4.x and Other 5.x operating systems

  If you configure the operating system of a VM as Other 4.x or Other 5.x, the Edit option on the Guest OS Customization screen appears dimmed and you cannot change the guest operating system settings for this VM.

• The Create vApp from vApp template wizard displays the customization drop-down fields as text fields

  If you instantiate new vApp from a vApp template with customized properties, the Custom Properties page of the Create vApp from Template wizard displays the predefined template customization fields as input fields, instead of as drop-down menus.

• Running the vmware-vcd-support script results in a warning message

  When you run the vmware-vcd-support script, during the file preparation phase, the operation results in a warning message like could not run lspci -intel_conf1 -M or could not write to /tmp/lspci1.9225.txt Do you have a full disk? Continuing...

  The warnings do not prevent the successful execution of the script but result in capturing fewer support log messages.

• Auto scaling fails with an Operation denied error message

  An auto scaling operation fails with an Operation denied error message.
  This happens because the API token that the auto scaling service uses to authenticate against the VMware Cloud Director API, expires and the rule that triggers the growing or shrinking of scale groups stops working.

• Changing the name and description of a global tenant role is not reflected in the Tenant Portal

  Changing the name and description of a published global tenant role in the VMware Cloud Director Service Provider Admin Portal does not update the name and description for the same role in the Tenant Portal.

• Moving a vApp that contains a VM configured with static IP address to a different VDC resets the vApp IP address

  If a vApp contains a VM, connected to the vApp network by a static IP pool,  the destination VM does not maintain the same IP address as the source VM after moving the vApp by running the vdc/action/moveVApp API operation.

• As an owner of a scale group, I do not have administrator rights over the VMs that are created in this scale group

  In a scale group, if a growing task triggers the creation of a new VM, the owner of the new VM is the system administrator instead of the owner of the scale group.

• The web console displays the user input as a mix of upper and lower cases

  In a VM web console, if you consecutively turn on and then turn off the Caps Lock key, the console displays the users input as a mix of upper and lower cases.

• Submitting a change for a NAT rule assigns the changes to a different NAT rule

  When you submit a change for a NAT rule, the system assigns the changes to a different NAT rule.

• Updating the resource allocation shares of a VM with CPU and Memory reservations of zero and CPU and Memory limits of Unlimited does not change the value of the VM's shares

  In a reservation pool VDC, attempting to set the resource allocation shares of a VM that is configured with CPU and Memory reservations to zero and CPU and Memory limits to Unlimited does not change the VM shares.

• Deleting a vApp from VMware Cloud Director fails with a Could not delete folder​ error message

  When you attempt to delete a vApp that contains no VMs, the operation fails with a Could not delete folder error message.

  This can happen when during the vApp creation by instantiation of a template or copying a VM into a vApp, the VM copy operation fails after
  creating the cloned VM and the system does not remove the failed VM from vCenter Server. As a result, the vApp folder in vCenter Server contains child VMs and the vApp delete operation fails.

• Copying a powered-on vApp with routed network fails with an error message

  In a powered-on vApp configured with a routed vApp network, if you add a new VM and connect it to the routed vApp network, attempting to copy this vApp to a different organization VDC results in an error message.
  Cannot use IP XX.XXX.XXX.XX as external IP for vApp router because it has already been allocated.

• After upgrade to VMware Cloud Director 10.3, the UI cannot display the list with external networks

  If you configure an external network to connect to an organization VDC network, after upgrading to VMware Cloud Director 10.3, the VMware Cloud Director UI cannot display the list of external networks.

• Publishing a provider VDC Kubernetes policy to an organization VDC fails with an error message​

  In an organization VDC, if one or more of the host clusters do not have Workload Management enabled, publishing a provider VDC Kubernetes policy fails with an error message.
  Before you can publish a VdcKubernetesPolicy you must refresh the cluster information by running reconnect on the clusters Virtual Center.

• VMware Cloud Director cell randomly restarts​

  A VMware Cloud Director cell restarts after an uncaught exception occurs in the MQTT service.

• Power on a VM that is connected to an external network fails with an A specified parameter was not correct error message

  If you configure a VM to use an external network backed by multiple distributed port groups, where each distributed port group resides on a different vSphere distributed switch, and all distributed switches are on the same vCenter Server system, powering on the VM fails with an error message.
  A specified parameter was not correct: spec.deviceChange.device.port.switchUuid Host esx-01a.corp.local is not a member of VDS DSwitch_C2.

• Performing an operation on an edge gateway in an organization VDC with a Kubernetes policy corrupts all firewall rules related to this edge gateway

  In an organization VDC with a Kubernetes policy or an organization VDC with a Kubernetes policy that is a member of a data center group, if you perform an add, delete, modify, or change membership operation on an edge gateway, all firewall rules related to this edge gateway for all other Workload Control Plane Tier-1 gateways in all organizations in the VDC or the data center group, get corrupted.

• Deleting a vApp networks from a powered-on vApp does not remove the logical networks in NSX

  Deleting a vApp network from a powered-on vApp by using the VMware CLoud Director API does not deletes the network from NSX-T Data Center and NSX Data Center for vSphere. As a result, the network is in inconsistent state.

Known Issues

• New VMware Cloud Director UI and tasks are slow to load and complete

  The Artemis message bus communication is not working and when you trigger operations from the UI, they can take up to 5 minutes to complete or might time out. The performance issues can affect operations such as powering on VMs and vApps, provider VDC creation, vApp deployment, and so on.

  The log files might contain an error message, such as:

  • a) Connection failure to <VCD Cell IP Address> has been detected: AMQ229014: Did not receive data from <something> within the 60,000ms

  • b) Connection failure to /<VCD Cell IP Address>:61616 has been detected: AMQ219014: Timed out after waiting 30,000 ms

  • c) Bridge is stopping, will not retry

  • d) Local Member is not set at on ClusterConnection ClusterConnectionImp

  Workaround:

  For a) and b):

  1. Verify that the VMware Cloud Director cells have network connectivity and can communicate with each other.

  2. Restart the VMware Cloud Director cell that contains the error message.

  For c) and d), restart the VMware Cloud Director cell that contains the error message.

• New The VMware Cloud Director appliance database disk resize script might fail if the backing SCSI disk identifier changes

  The database disk resize script runs successfully only if the backing database SCSI disk ID remains the same. If the ID changes for any reason, the script might appear to run successfully but fails. The /opt/vmware/var/log/vcd/db_diskresize.log shows that the script fails with a No such file or directory error.

  Workaround:

  1. Log in directly or by using an SSH client to the primary cell as root.

  2. Run the lsblk --output NAME,FSTYPE,HCTL command.

  3. In the output, find the disk containing the database_vg-vpostgres partition and make note of its ID. The ID is under the HCTL column and has the following sample format 2:0:3:0.

  4. In the db_diskresize.sh script, modify the partition ID with the ID from Step 3. For example, if the ID is 2:0:3:0, in line

     echo 1 > /sys/class/scsi_device/2\:0\:2\:0/device/rescan

     you must change the ID to 2:0:3:0.

     echo 1 > /sys/class/scsi_device/2\:0\:3\:0/device/rescan

  5. Аfter saving the changes, manually re-invoke the resize script or reboot the appliance.

• New Publishing a vRealize Orchestrator workflow to the VMware Cloud Director service library fails with an error message

  When you attempt to publish a vRealize Orchestrator workflow, the operation fails with a 500 Server Error error message.

  This happens because the API returns a large number of links for each individual tenant to which the workflow is published and causes an overflow in the HTTP headers.

  Workaround: To publish the workflow, use CURL or POSTMAN to run an API request with increased HTTP header size limit.

• New Configuring the IP mode for a VM NIC to Static - Manual with IPv4 address results in an error message

  For a new or existing VM, if you attempt to configure the IP mode to Static - Manual with an IPv4 address, the validation fails with an error message. <IP-address> is not a valid IPv6 address.

  Workaround: Set the VM's IP mode to Static - IP Pool, assign the IPv4 address, and then switch the IP mode back to Manual.

• New When you use the VMware Cloud Director UI to create a new VM with a placement policy, all virtual machines that are part of the VM group defined in the used placement policy might disappear

  When you use the VMware Cloud Director UI to create a new VM that uses a certain placement policy, all virtual machines listed in the VM group that's defined in the used placement policy might disappear from the VM group.

  Workaround: When the VMs get deleted from the group, they become non-compliant with the placement policy that you used to create the new VM. To restore the VMs to the group, manually make each of them compliant with the used placement policy.

• New VMware Cloud Director operations, such as powering a VM on and off takes longer time to complete

  VMware Cloud Director operations, such as powering a VM on or off takes longer time to complete. The task displays a Starting virtual machine status and nothing happens.

  The jms-expired-messages.logs log file displays an error.

  RELIABLE:LargeServerMessage & expiration=

  Workaround: None.

• New Migrating a VM that is connected to a vSphere-backed external network between resource pools fails

  If a VM is connected to an external network which is backed by multiple vSphere networks, and you attempt to migrate the VM between resource pools, the operation fails if the source and destination resource pools are backed by different host clusters and if the destination resource pool does not have access to the external network to which the VM was originally connected.

  Workaround: None.

• New A VM with IP mode set to DHCP might not be able to connect to an external network

  If a VM with IP mode set to DHCP is connected to a vApp network that uses port forwarding, the VM cannot connect to an external network. This happens because in NSX-backed organization VDCs, enabling IP masquerading for a vApp network does not create a corresponding SNAT rule on the vApp edge in NSX to allow outbound access for a VM without a static IP.

  Workaround: Add to the vApp network a second vApp with a static IP and an explicit DNAT rule that allows access to the external network to the vApp network.

• New You cannot create VMware Cloud Director VDC templates in VMware Cloud Director service environments

  VMware Cloud Director service does not support Virtual Data Center (VDC) templates. You can use VDC templates on environments with provider VDCs with an NSX network provider type or an NSX Data Center for vSphere provider type. You cannot use VDC templates on VMware Cloud Director service environments because the provider VDCs have the VMC network provider type.

  Workaround: None.

• New Migrating VMs between organization VDCs might fail with an insufficient resource error

  If VMware Cloud Director is running with vCenter Server 7.0 Update 3h or earlier, when relocating a VM to a different organization VDC, the VM migration might fail with an insufficient resource error even if the resources are available in the target organization VDC.

  Workaround: Upgrade vCenter Server to version 7.0 Update 3i or later.

• New VMs become non-compliant after converting a reservation pool VDC into a flex organization VDC

  In an organization VDC with a reservation pool allocation model, if some of the VMs have nonzero reservation for CPU and Memory, non-unlimited configuration for CPU and Memory, or both, after converting into a flex organization VDC, these VMs become non-compliant. If you attempt to make the VMs compliant again, the system applies an incorrect policy for the reservation and limit and sets the CPU and Memory reservations to zero and the limits to Unlimited.

  Workaround:

  1. A system administrator must create a VM sizing policy with the correct configuration.
  2. A system administrator must publish the new VM sizing policy to the converted flex organization VDC.
  3. The tenants can use the VMware Cloud Director API or the VMware Cloud Director Tenant Portal to assign the VM sizing policy to the existing virtual machines in the flex organization VDC.
• New Suspending a VM through the VMware Cloud Director UI results in a partially suspended state of the VM

  In the VMware Cloud Director Tenant Portal, when you suspend a VM, VMware Cloud Director does not undeploy the VM, and the VM becomes Partially Suspended instead of Suspended.

  Workaround: None.

• New Role name and description are localized in the VMware Cloud Director UI and can cause duplication of role names

  The problem occurs because the UI translation does not affect the back end and API. You might create roles with the same names as the translated names which results in perceived duplicate roles in the UI and conflicts with the API usage of role names when creating service accounts.

  Workaround: None.

• New The Customer Experience Improvement Program (CEIP) status is Enabled even after deactivating it during the installation of VMware Cloud Director

  During the installation of VMware Cloud Director, if you deactivate the option to join the CEIP, after the installation completes, the CEIP status is active.

  Workaround: Deactivate the CEIP by following the steps in the Join or Leave the VMware Customer Experience Improvement Program procedure.

• New VMware Cloud Director appliance upgrade fails with an invalid version error when FIPS mode is enabled

  For VMware Cloud Director versions 10.3.x and later, when FIPS mode is enabled, VMware Cloud Director appliance upgrade fails with the following error.

  Failure: Installation failed abnormally (program aborted), the current version may be invalid.

  Workaround:

  1. Before you upgrade the VMware Cloud Director appliance, deactivate FIPS Mode on the cells in the server group and the VMware Cloud Director appliance. See Activate or Deactivate FIPS Mode on the VMware Cloud Director Appliance.

  2. Verify that the /etc/vmware/system_fips file does not exist on any appliance.

  3. Upgrade the VMware Cloud Director appliance.

  4. Enable FIPS mode again.

• New If you enable the non-blocking AMQP notifications, AMQP based API extensibility requests to VMware Cloud Director time out

  After upgrading to VMware Cloud Director 10.3.2, AMQP API reply messages to VMware Cloud Director are not handled and time out if non-blocking AMQP notifications are enabled in VMware Cloud Director.

  Workaround: Deactivate the non-blocking AMQP notifications.

• New When performing a database upgrade for VMware Cloud Director, the upgrade fails with insert or update on table error

  The issue occurs due to stale information in tables associated with a foreign key constraint. Missing data in one of the tables causes a conflict with the foreign key constraint.

  Workaround: See VMware knowledge base article 88010.

• New Refreshing the LDAP page in your browser does not take you back to the same page

  In the Service Provider Admin Portal, refreshing the LDAP page in your browser takes you to the provider page instead of back to the LDAP page.

  Workaround: None.

• New Mounting an NFS datastore from NetApp storage array fails with an error message during the initial VMware Cloud Director appliance configuration

  During the initial VMware Cloud Director appliance configuration, if you configure an NFS datastore from NetApp storage array, the operation fails with an error message.
  Backend validation of NFS failed with: is owned by an unknown user

  Workaround: Configure the VMware Cloud Director appliance by using the VMware Cloud Director Appliance API.

• New The synchronization of a subscribed catalog times out while synchronizing large vApp templates

  If an external catalog contains large vApp templates, synchronizing the subscribed catalog with the external catalog times out.
  This happens when the timeout setting is set to its default value of five minutes.

  Workaround: Using the manage-config subcommand of the cell management tool, update the timeout configuration setting.
  ./cell-management-tool manage-config -n transfer.endpoint.socket.timeout -v [timeout-value]

• New If you migrate a VM, vApp, or independent disk to a vCenter Server instance that uses well-signed certificates, the migration fails

  When trying to migrate a VM, vApp, or independent disk from one vCenter Server instance to another that uses well-signed certificates, the migration fails. The problem occurs when using the VMware Cloud Director UI and API requests such as recompose, migrateVms, moveVApp, and so on.

  Workaround: None.

• When you enable FIPS mode, the vRealize Orchestrator integration fails with an error related to invalid parameters.

  When you enable FIPS mode, the integration between VMware Cloud Director and vRealize Orchestrator does not work. The VMware Cloud Director UI returns an Invalid VRO request params error. The API calls return the following error: 

  Caused by: java.lang.IllegalArgumentException: 'param' arg cannot be null at org.bouncycastle.jcajce.provider.ProvJKS$JKSKeyStoreSpi.engineLoad(Unknown Source) at java.base/java.security.KeyStore.load(KeyStore.java:1513) at com.vmware.vim.install.impl.CertificateGetter.createKeyStore(CertificateGetter.java:128) at com.vmware.vim.install.impl.AdminServiceAccess. (AdminServiceAccess.java:157) at com.vmware.vim.install.impl.AdminServiceAccess.createDiscover(AdminServiceAccess.java:238) at com.vmware.vim.install.impl.RegistrationProviderImpl. (RegistrationProviderImpl.java:56) at com.vmware.vim.install.RegistrationProviderFactory.getRegistrationProvider(RegistrationProviderFactory.java:143) at com.vmware.vcloud.vro.client.connection.STSClient.getRegistrationProvider(STSClient.java:126) ... 136 more

  Workaround: None.

• After upgrade to VMware Cloud Director 10.3.2a, opening the list of external networks results in a warning message

  When trying to open the list of external networks, the VMware Cloud Director UI displays a warning message.
  One or more external networks or T0 Gateways have been disconnected from its IP address data.
  
  This happens because the external network gets disconnected from the Classless Inter-Domain Routing (CIDR) configuration before the upgrade to VMware Cloud Director 10.3.2a.

  Workaround: Contact VMware Global Support Services (GSS) for assistance with the workaround for this issue.

• In an IP prefix list, configuring any as the Network value results in an error message

  When creating an IP prefix list, if want to deny or accept any route and you configure the Network value as any, the dialog box displays an error message.
  "any" is not a valid CIDR notation. A valid CIDR is a valid IP address followed by a slash and a number between 0 and 32 or 64, depending on the IP version.

  Workaround: Leave the Network text box blank.

• If you use vRealize Orchestrator 8.x, hidden input parameters in workflows are not populated automatically in the VMware Cloud Director UI

  If you use vRealize Orchestrator 8.x, when you attempt to run a workflow through the VMware Cloud Director UI, hidden input parameters are not populated automatically in the VMware Cloud Director UI.

  Workaround:
  To access the values of the workflow input parameters, you must create a vRealize Orchestrator action that has the same input parameter values as the workflow that you want to run. 
  1. Log in to the vRealize Orchestrator Client and navigate to Library>Workflows.
  2. Select the Input Form tab and click Values on the right-hand side.
  3. From the Value options drop-down menu, select External source, enter the Action inputs and click Save.
  4. Run the workflow in the VMware Cloud Director UI.

• The vpostgres process in a standby appliance fails to start

  The vpostgres process in a standby appliance fails to start and the PostgreSQL log shows an error similar to the following. FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the master server (its value was 16). This happens because PostgreSQL requires standby nodes to have the same max_worker_processes setting as the primary node. VMware Cloud Director automatically configures the max_worker_processes setting based on the number of vCPUs assigned to each appliance VM. If the standby appliance has fewer vCPUs than the primary appliance, this results in an error.

  Workaround: Deploy the primary and standby appliances with the same number of vCPUs.

• VMware Cloud Director API calls to retrieve vCenter Server information return a URL instead of a UUID

  The issue occurs with vCenter Server instances that failed the initial registration with VMware Cloud Director version 10.2.1 and earlier. For those vCenter Server instances, when you make API calls to retrieve the vCenter Server information, the VMware Cloud Director API incorrectly returns a URL instead of the expected UUID.

  Workaround: Reconnect to the vCenter Server instance to VMware Cloud Director.

• Upgrading from VMware Cloud Director 10.2.x to VMware Cloud Director 10.3 results in an Connection to sfcbd lost error message

  If you upgrade from VMware Cloud Director 10.2.x to VMware Cloud Director 10.3, the upgrade operation reports an error message.
  Connection to sfcbd lost. Attempting to reconnect

  Workaround: You can ignore the error message and continue with the upgrade.

• When using FIPS mode, trying to upload OpenSSL-generated PKCS8 files fails with an error

  OpenSSL cannot generate FIPS-complaint private keys. When VMware Cloud Director is in FIPS mode and you try to upload PKCS8 files generated using OpenSSL, the upload fails with a Bad request: org.bouncycastle.pkcs.PKCSException: unable to read encrypted data: ... not available: No such algorithm: ... error or salt must be at least 128 bits error.

  Workaround: Deactivate the FIPS mode to upload the PKCS8 files.

• Creation of Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in fails

  When you create a Tanzu Kubernetes cluster by using the Kubernetes Container Clusters plug-in, you must select a Kubernetes version. Some of the versions in the drop-down menu are not compatible with the backing vSphere infrastructure. When you select an incompatible version, the cluster creation fails.

  Workaround: Delete the failed cluster record and retry with a compatible Tanzu Kubernetes version. For information on the incompatibilities between Tanzu Kubernetes and vSphere, see Updating the vSphere with Tanzu Environment.

• If you have any subscribed catalogs in your organization, when you upgrade VMware Cloud Director, the catalog synchronization fails

  After upgrade, if you have subscribed catalogs in your organization, VMware Cloud Director does not trust the published endpoint certificates automatically. Without trusting the certificates, the content library fails to synchronize.

  Workaround: Manually trust the certificates for each catalog subscription. When you edit the catalog subscription settings, a trust on first use (TOFU) dialog prompts you to trust the remote catalog certificate.
  If you do not have the necessary rights to trust the certificate, contact your organization administrator.

• After upgrading VMware Cloud Director and enabling the Tanzu Kubernetes cluster creation, no automatically generated policy is available and you cannot create or publish a policy

  When you upgrade VMware Cloud Director to version 10.3.1 and vCenter Server to version 7.0.0d or later, and you create a provider VDC backed by a Supervisor Cluster, VMware Cloud Director displays a Kubernetes icon next to the VDC. However, there is no automatically generated Kubernetes policy in the new provider VDC. When you try to create or publish a Kubernetes policy to an organization VDC, no machine classes are available.

  Workaround: Manually trust the corresponding Kubernetes endpoint certificates. See VMware knowledge base article 83583.

• Entering a Kubernetes cluster name with non-Latin characters deactivates the Next button in the Create New Cluster wizard

  The Kubernetes Container Clusters plug-in supports only Latin characters. If you enter non-Latin characters, the following error appears. Name must start with a letter and only contain alphanumeric or hyphen (-) characters. (Max 128 characters).

  Workaround: None.

• NFS downtime can cause VMware Cloud Director appliance cluster functionalities to malfunction

  If the NFS is unavailable due to the NFS share being full, becoming read only, and so on, can cause appliance cluster functionalities to malfunction. HTML5 UI is unresponsive while the NFS is down or cannot be reached. Other functionalities that might be affected are the fencing out of a failed primary cell, switchover, promoting a standby cell, and so on. For more information about setting up correctly the NFS shared storage, see Preparing the Transfer Server Storage for the VMware Cloud Director Appliance.

  Workaround: 

  • Fix the NFS state so that it is not read-only.
  • Clean up the NFS share if it is full.
• Trying to encrypt named disks in vCenter Server version 6.5 or earlier fails with an error

  For vCenter Server instances version 6.5 or earlier, if you try to associate new or existing named disks with an encryption enabled policy, the operation fails with a Named disk encryption is not supported in this version of vCenter Server. error.

  Workaround: None.

• A fast-provisioned virtual machine created on a VMware vSphere Storage APIs Array Integration (VAAI) enabled NFS array, or vSphere Virtual Volumes (VVols) cannot be consolidated

  In-place consolidation of a fast provisioned virtual machine is not supported when a native snapshot is used. Native snapshots are always used by VAAI-enabled datastores, as well as by VVols. When a fast-provisioned virtual machine is deployed to one of these storage containers, that virtual machine cannot be consolidated .

  Workaround: Do not enable fast provisioning for an organization VDC that uses VAAI-enabled NFS or VVols. To consolidate a virtual machine with a snapshot on a VAAI or a VVol datastore, relocate the virtual machine to a different storage container.

• If you add an IPv6 NIC to a VM and then you add an IPv4 NIC to the same VM, the IPv4 north-south traffic breaks

  Using the HTML5 UI, if you add an IPv6 NIC first or configure an IPv6 NIC as the primary NIC in a VM, and then you add an IPv4 NIC to the same VM, the IPv4 north-south communication breaks.

  Workaround: First you must add the IPv4 NIC to the VM and then the IPv6 NIC.

• Guest OS customizations like hostname and network do not work for the AlmaLinux OS

  If you deploy an AlmaLinux template, VMware Cloud Director ignores the hostname and network configurations even when you force the guest customizations.

  Workaround: Edit the /etc/redhat-release file to replace AlmaLinux with CentOS Linux.