What's in the Release Notes

The release notes cover the following topics:

About VMware Integrated OpenStack
Compatibility
Upgrading to Version 5.1.0.3
Deprecation Notices
Internationalization
Open Source Components for VMware Integrated OpenStack
Resolved Issues
Known Issues

About VMware Integrated OpenStack

VMware Integrated OpenStack greatly simplifies deploying an OpenStack cloud infrastructure by streamlining the integration process. VMware Integrated OpenStack delivers out-of-the-box OpenStack functionality and an easy configuration workflow through a deployment manager vApp that runs directly in vCenter Server.

Compatibility

See the VMware Product Interoperability Matrices for details about the compatibility of VMware Integrated OpenStack with other VMware products, including vSphere components.

Upgrading to Version 5.1.0.3

The upgrade to VMware Integrated OpenStack 5.1.0.3 is a patch process. If you are running VMware Integrated OpenStack 5.1, 5.1.0.1, or 5.1.0.2, see the Patch Process section. If you are running VMware Integrated OpenStack 5.0 or an earlier version, see the Earlier Versions section.

Patch Process

If you are running VMware Integrated OpenStack 5.1, 5.1.0.1, or 5.1.0.2, you can apply the patch directly to your existing deployment. To do so, perform the following steps:

Verify that the VMware Integrated OpenStack service is either running or not yet deployed. If the VMware Integrated OpenStack service is in any other state, the upgrade will fail.
If you have already deployed OpenStack, take snapshots of the OpenStack Management Server and the deployment.
Do not perform this step if you have not yet deployed OpenStack on the current environment.
1. In the vSphere Client, take a snapshot of the OpenStack Management Server virtual machine.
2. Log in to the OpenStack Management Server virtual machine and take a snapshot by running the following command:
  sudo viopatch snapshot take
  This command stops OpenStack services. Services will be started again when the patch is installed.
  NOTE: If the command fails, see "For deployments using a remote vCenter Server, the viopatch command fails to take snapshots" in the Known Issues section.
Download the patch file to the OpenStack Management Server virtual machine.
Add the patch file by running the following command:
sudo viopatch add -l path/vio-patch-5.1.0.3_5.1.0.14459038_all.deb
Install the patch file by running the following command:
sudo viopatch install -p vio-patch-5.1.0.3 -v 5.1.0.14459038
During the patch installation, the API endpoint is automatically turned off. As a result, any API calls during installation will return a 503 error.
Log out of the vSphere Client and log back in. Any error messages encountered during login can be ignored.

NOTE: The viopatch uninstall action is deprecated and cannot be used to revert to the previous version. The snapshots created in this process are therefore necessary for reversion. Do not remove these snapshots until all validation tasks have been completed and you are certain that you will not need to revert to the previous version.

After you have validated that the patched version is operating correctly, you can run sudo viopatch snapshot remove to delete the snapshot. This action is destructive and cannot be reversed.

If you need to revert to the previous version after installing the patch, perform the following steps.

In the vSphere Client, revert the OpenStack Management Server virtual machine to the previous snapshot.
On the OpenStack Management Server virtual machine, restart the OpenStack service by running the following command:
sudo service oms restart
Revert to the previous snapshot by running the following command:
sudo viopatch snapshot revert
On the vCenter Server virtual machine, stop the vSphere Client service, delete residual files, and restart the service:
service-control --stop vsphere-ui cd /etc/vmware/vsphere-ui/vc-packages/vsphere-client-serenity/ rm -rf * cd /usr/lib/vmware-vsphere-client/server/work rm -rf * service-control --start vsphere-ui
Log out of the vSphere Client and log back in.

Earlier Versions

To upgrade from VMware Integrated OpenStack 5.0 to VMware Integrated OpenStack 5.1.0.3, perform the following steps:

Upgrade to VMware Integrated OpenStack 5.1 as described in Patch VMware Integrated OpenStack.
Apply the VMware Integrated OpenStack 5.1.0.3 patch as described above in the Patch Process section.

To upgrade from VMware Integrated OpenStack 4.x to VMware Integrated OpenStack 5.1.0.3, perform the following steps:

Deploy the VMware Integrated OpenStack 5.1 OVA as described in Install the New Version.
On the new 5.1 deployment, apply the VMware Integrated OpenStack 5.1.0.3 patch as described above in the Patch Process section.
Migrate to the patched deployment as described in Migrate to the New VMware Integrated OpenStack Deployment.

Deprecation Notices

New versions of vRealize Automation will no longer be certified with VMware Integrated OpenStack.

Some OpenStack Management Server lifecycle management APIs available in VMware Integrated OpenStack 5.1.0.3 will be changed or deprecated in a future major release of VMware Integrated OpenStack.

The SDDC provider in VMware Integrated OpenStack with Kubernetes, intended for deployments without VMware Integrated OpenStack, will be deprecated in a future major release. For new VMware Integrated OpenStack with Kubernetes deployments, use the OpenStack provider.

Internationalization

VMware Integrated OpenStack 5.1.0.3 is available in English and seven additional languages: Simplified Chinese, Traditional Chinese, Japanese, Korean, French, German, and Spanish.

The following items must contain only ASCII characters:

Names of OpenStack resources (such as projects, users, and images)
Names of infrastructure components (such as ESXi hosts, port groups, data centers, and datastores)
LDAP and Active Directory attributes

Open Source Components for VMware Integrated OpenStack

The copyright statements and licenses applicable to the open source software components distributed in VMware Integrated OpenStack 5.1.0.3 are available on the Open Source tab of the product download page. You can also download the disclosure packages for the components of VMware Integrated OpenStack that are governed by the GPL, LGPL, or other similar licenses that require the source code or modifications to source code to be made available.

Resolved Issues

Additional parameters have been added to the custom.yml file.
The following items can now be configured in custom.yml:
- mysql_innodb_deadlock_detect - enables or disables the deadlock mechanism.
- mysql_innodb_lock_wait_timeout - sets the timeout period for transactions waiting for a lock.
If separate vSphere domains are used for the management and compute vCenter Server instances, certificate operations may fail.
When the management vCenter Server instance is not used for Nova compute and the user name is located in a different vSphere domain, certificate-related operations, such as the viocli certificate command, may be unable to authenticate.

This issue has been resolved in this release.
Attempting to extend an attached volume puts the volume in an error state.
VMware Integrated OpenStack does not support extending attached volumes.

The extend_attached_volume API has been disabled so that attached volumes cannot be accidentally extended.
Neutron operations may fail with connection errors.
Under heavy load, some requests may fail due to connection errors.

This issue has been resolved in this release.
After an interface is removed from a virtual machine in vSphere, its logical ports cannot be deleted in OpenStack.
If you use vSphere to remove an interface from a virtual machine corresponding to an OpenStack instance, OpenStack becomes unable to detach the Neutron ports on that interface.

This issue has been resolved in this release.
Configuring a single LDAP server to multiple Keystone domains fails.
If you configure multiple Keystone domains to be backed by a single LDAP server, Keystone cannot connect to the LDAP server.

This issue has been resolved in this release.
A disk resize failure on an instance may prevent subsequent disk resize operations from succeeding on the instance.
After a disk resize operation fails, it is possible that a stale disk may remain on the instance. This causes later resize operations to fail.

This issue has been resolved in this release.
A REST API transport error occurs when upgrading from VMware Integrated OpenStack 4.1.
When you click Finish to complete the upgrade process, the error message "Internal error: REST API transport layer error." is displayed in a pop-up window.

This issue has been resolved in this release.
For NSX-T Data Center deployments, user names cannot include the underscore (_) character.
The underscore character is not supported in user names that interface with NSX-T Data Center.

This issue has been resolved in this release.
After enabling multi-tenancy on Designate with an Infoblox back end, zones and record sets enter the error state.
If you configure Designate with an Infoblox back-end server and set the designate_infoblox_multi_tenant parameter to true, the status of your Designate zones and record sets is displayed as "ERROR".

This issue has been resolved in this release.
Concurrent operations on a virtual machine may fail.
If an OpenStack operation and a vSphere operation are performed at the same time on the same virtual machine, the OpenStack operation may fail.

This issue has been resolved in this release.
When upgrading compact VMware Integrated OpenStack deployments, an extra IP address is required on the management network.
In previous versions, the upgrade process for all deployment types required that a minimum of three IP addresses be available on the management network.

For compact deployments, this has been corrected to a minimum of two IP addresses.
Glance may fail to import VMDK files.
If the vmware_adapter_type metadata of a VMDK file does not match the introspected value, Glance may fail to import the file.

This issue has been resolved in this release.
For NSX Data Center for vSphere deployments, LBaaS listeners are not assigned a pool.
When you create a listener, the value of its defaultPool parameter is set to null.

This issue has been resolved in this release.
The URL for an LBaaS health monitor cannot contain a query string.
Because the question mark (?) character is not allowed in health monitor URLs, a URL containing a query string, such as .../test.html?action=test, is considered invalid.

This issue has been resolved in this release.
A DRS conflict may prevent OpenStack instances from booting.
If a DRS conflict occurs while an instance is being launched, ports might not be cleaned up and the launch process may time out.

This issue has been resolved in this release.
Datastores with the same MOID in different vCenter Server instances cannot be added to a deployment.
If a new datastore has the same managed object identifier (MOID) as a datastore that is already in your deployment, the VMware Integrated OpenStack user interface does not display the new datastore or allow you to add it to your deployment. This could occur when the two datastores are located in separate compute vCenter Server instances.

This issue has been resolved in this release.
Controller nodes may enter an out of memory state.
Due to a memory leak in an upstream Horizon memoization function, the VMware Integrated OpenStack dashboard may consume excessive memory on controller nodes.

This issue has been resolved in this release.

Known Issues

The known issues are grouped as follows.

VMware Integrated OpenStack
VMware Integrated Openstack with Kubernetes

VMware Integrated OpenStack

In an environment with multiple vCenter Server instances, Nova live migration cannot move an instance to a host in the remote vCenter Server.
Nova live migration across vCenter Server instances is not supported.

Workaround: None.
Deleting an instance associated with a floating IP address does not delete associated DNS records.
If you delete an instance that has a floating IP address associated, Designate will not remove the DNS records for that instance.

Workaround: Disassociate the floating IP address from the instance before deleting the instance.
Compute nodes might not start after being removed and re-added.
If there is a tenant virtual data center created for a compute node, that node will fail to start if it is removed and added again.

Workaround: None.
Local storage may be incorrectly calculated on the VMware Integrated OpenStack dashboard.
If multiple compute nodes use the same datastore, the Hypervisors page on the VMware Integrated OpenStack dashboard will incorrectly display that the total disk space available is the size of the single datastore multiplied by the number of compute nodes using it. In addition, the entry in the Local Storage (used) column for each compute node will display the total used space on the datastore, not the used space for a single compute node.

Workaround: None.
In an environment with multiple vCenter Server instances, instances cannot find templates and fail to boot.
If an image is deleted manually from the compute vCenter Server, instances might fail to boot with the error "Unable to find template at location". This issue can also occur when re-adding compute nodes.

Workaround: Determine the location of the remote vCenter Server from the image by running the glance image-show image-uuid command. Then delete the location from Glance by running the glance location-delete --url image-location command.
For deployments using a remote vCenter Server, the viopatch command fails to take snapshots.
In a deployment where all control virtual machines are deployed in a management vCenter Server instance and use a Nova compute node deployed in a remote vCenter Server instance, the viopatch snapshot take command cannot obtain information about the management vCenter Server instance. The command fails with the error "AttributeError: 'NoneType' object has no attribute 'snapshot'."

Workaround: On the OpenStack Management Server virtual machine, manually set the IP address, username, and password of the management vCenter Server by running the following commands:
```
export VCENTER_HOSTNAME = mgmt-vc-ip-address
export VCENTER_USERNAME = mgmt-vc-username
export VCENTER_PASSWORD = mgmt-vc-password
```
VMware Integrated OpenStack cannot connect to NSX-T after the NSX-T password is changed.
If you change the NSX-T password while the Neutron server is running, VMware Integrated OpenStack might fail to connect to NSX-T.

Workaround: Before changing the NSX-T password, log in to the active controller node and run the systemctl stop neutron-server command to stop the Neutron server service. The service will be restarted after you update the NSX-T password in VMware Integrated OpenStack.
The Nova compute service fails to start after an upgrade from version 4.x.
If a Nova compute node was deleted in version 4.x and a new Nova compute node using the same vCenter Server and same cluster was added later, the Nova compute service will fail to start after you upgrade to version 5.x. "ERROR nova ResourceProviderCreationFailed" is written to /var/log/nova/nova-compute.log.

Workaround: Perform the following steps to remove the Nova compute node from the database:
1. Find the MOID of the deleted compute node.
2. Log in to the active database node and open the nova_api database:
  mysql use nova_api
3. In the "resource_providers" table, remove the resource_provider record with the MOID of the deleted compute node.
4. In the "host_mappings" table, remove the host record for the deleted compute node.
A datastore failure might render the OpenStack deployment inaccessible.
If all nodes in an HA deployment use the same datastore, the failure of that datastore will cause the entire deployment to be inaccessible.

Workaround: Try to fix the failed datastore and recover its data. After the virtual machine for each node is shown in vCenter Server, restart the OpenStack deployment. If the datastore is not recoverable, use the viocli recover command to restore the failed nodes.
The Keystone endpoint is in the error state.
After the internal endpoint in-flight encryption setting is changed, the Keystone endpoint fails to reconnect. This issue occurs when you set the internal_api_protocol parameter to http for an HA deployment or https for a compact or tiny deployment.

Workaround: Modify the Keystone endpoint URL.
1. In the vSphere Web Client, select Administration > OpenStack.
2. Select the KEYSTONE endpoint and click the Edit (pencil) icon.
3. In the Update Endpoint section displayed, change the URL to begin with http or https depending on your configuration.
4. Enter the administrator password and click Update.
The VMware Integrated OpenStack OVA cannot be deployed in the HTML5 vSphere Client in vCenter Server 6.7.
After you deploy the VMware Integrated OpenStack OVA using the HTML5 vSphere Client in vCenter Server 6.7, the VMware Integrated OpenStack vApp fails to power on and the UI displays the error: "The virtual machine has a required vService dependency 'vCenter Extension Installation' which is not bound to a provider."

Workaround: Deploy the VMware Integrated OpenStack OVA using the Flex-based vSphere Web Client or using the OVF Tool.

For more information, see the vSphere 6.7 Release Notes and KB 55027.
Host names that start with a number cause a "java.io.IOException" error.
The OpenStack Management Server does not support host names that start with a number. The error "java.io.IOException: DNSName components must begin with a letter" appears if the host name starts with a number.

Workaround: Use a host name that does not start with a number. For more information, see the JDK upstream issue: https://bugs.openjdk.java.net/browse/JDK-8054380
East-west traffic does not travel between virtual machines booted on a virtual wire provider network.
If you create a provider network using virtual wire and do not create a SpoofGuard policy, east-west traffic will not travel between the virtual machines booted on this provider network.

Workaround: Create a SpoofGuard policy and add virtual wire to the policy before creating the virtual wire provider network.
Certificate verification may fail on the OpenStack Management Server.
When you use the viocli or viopatch command-line utility, the following error may occur:
```
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
```
Workaround: On the OpenStack Management Server, disable verification of the vCenter Server certificate by running the following command:
```
export VCENTER_INSECURE=True
```
Deleting a router interface times out.
When concurrent Heat stacks are deployed with shared NSX routers, router interface deletion can time out. The following might be displayed: neutron_client_socket_timeout, haproxy_neutron_client_timeout, or haproxy_neutron_server_timeout.

Workaround: Do not use shared routers in environments where network resources frequently change. If NAT or floating IP addresses are required, use an exclusive router. Otherwise, use a distributed router.

VMware Integrated Openstack with Kubernetes

For NSX-T deployments, the vkube cluster heal command might fail.
If you use the vkube cluster heal command on a cluster whose k8s-master-0 node is in the Error state, the following error message may be displayed:
```
fatal: [k8s-master-0-0ffeac43-78ea-4eab]: FAILED! => {"changed": false, "msg": "Unable to start service etcd: Job for etcd.service failed because a timeout was exceeded. See \"systemctl status etcd.service\" and \"journalctl -xe\" for details.\n"}
```
Workaround: Using SSH, log in to the k8s-master-1 node for the affected cluster and manually delete the unreachable member from the etcd cluster. Then close the SSH connection and run the vkube cluster heal command again.
A cluster in the ERROR state cannot be deleted.
If the infrastructure is out of resources, cluster creation, healing, and scaling will fail and the cluster will enter the ERROR state.

Workaround: Perform the following steps:
1. Log in to the toolbox container.
2. Use the OpenStack client to delete hosts in the ERROR state.
3. Run the vkube cluster delete command again to delete the cluster.
If a username or domain contains a backslash (\), authentication fails.
The Keystone authentication plugin uses the backslash character as a separator to encode the domain name and username in a single string . If there is an additional backslash in either the domain name or username, the Keystone authentication plugin will not decode the domain name and username correctly.

Workaround: Use domain names or usernames that do not include the backslash character.