VMware Integrated OpenStack 7.1 | 13 MAY 2021 | Build OVA 17987092, Patch 17987093

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

About VMware Integrated OpenStack

VMware Integrated OpenStack greatly simplifies deploying an OpenStack cloud infrastructure by streamlining the integration process. VMware Integrated OpenStack delivers out-of-the-box OpenStack functionality and an easy configuration workflow through a deployment manager that runs as a virtual appliance in vCenter Server.

What's New

  • Support for the latest versions of VMware products: 
    • VMware Integrated OpenStack 7.1 is fully compatible with VMware vSphere 7.0 U2, NSX-T 3.1.1, and NSX-V 6.4.10
  • New features and enhancement:
    • Management Plane:
      • Support of Disaster Recovery for VIO Management Plane. Provides new viocli command to support the Management Plane recovery from a disaster. The full Disaster Recovery procedure is validated with SRM, vSphere Replication and NSX-T multisite feature, which covers Nova Instances, Cinder Volumes and Neutron Networks. After the deployment is recovered in target DR site, users could use VIO to manage the recovered Nova/Cinder/Neutron objects.
      • Support of Neutron NSX-T Management Plugin to Policy Plugin migration. 
      • Support multiple license keys: Allow admin to input multiple VIO license keys and assign the keys for usage.
    • OpenStack Driver:
      • Support Octavia Flavor. The feature allows users using the OpenStack Octavia flavors capability on load balancers created by VIO on NSX-T. This feature is supported by Neutron NSX-P plugin, and with NSX-T 3.1 or later.
    • Scalability:
      • Support of larger scalability. One VIO deployment now could support up to 128 Nova Compute Nodes, and support up to 10k Tenant Networks with Neutron NSX-T Policy Plugin, up to 8 VIO deployments federation with vIDM as IdP.

Upgrade to Version 7.1

  • Upgrade from 7.0 / 7.0.1, use viocli patch command. Please find the detailed instructions in product installation guide.
  • Upgrade from 6.0, please use blue/green upgrade procedure described in product installation guide.
  • VIO 7.1 does not support direct upgrade from 5.x, please upgrade to 7.0.1 firstly.

Compatibility

Deprecation Notices 

  • The following networking features have been deprecated and will be removed in VIO next release:
    • The NSX Data Center for vSphere driver for Neutron.
    • The NSX-T Management Plugin for Neutron will be replaced by the NSX-T Policy Plugin.
    • The TVD plugin, which allows a single VMware Integrated OpenStack deployment to use an NSX Data Center for vSphere back end and an NSX-T Data Center back end.
  • Neutron FWaaSv2 will be deprecated in a future version.

Resolved Issues

The resolved issues are grouped as follows.

Resolved VIO Management Issues
  • 2750794 fixed: The backup data were deleted when VIO admin deletes the VIO backup schedule job.

    In VIO 7.0, when admin deletes the VIO backup schedule job, the backup data stored on vCenter Content Library were also deleted.
    In VIO 7.1, the backup data will not be deleted when backup schedule job deleted.

  • 2738659 fixed: The VIO backup restore procedure could not work correctly when vCenter SSL is enabled.

    The VIO backup restore procedure could not work correctly when vCenter SSL is enabled. It will failed to upload the backup data to vCenter's content library and reports "x509: certificate unknown authority" error.

  • 2680755 fixed: Cannot progress VIO deploy wizard with an error "The vCenter resources loading timeout".

    VIO 7 uses govmomi to retrieve vCenter inventory information. The govmomi will run into panic if DistributedVirtualPortgroup has "MAC" or "System Traffic" traffic qualifier for traffic filtering and marking in the target vSphere environment.

  • 2677748 fixed: The VIO manager web UI is unable to list OpenStack services status

    The VIO manager web UI is unable to list services spinning wheel and only with "no resources found" message.

  • 2591794 fixed: Metadata service is unreachable in the NSX-V environment

    The metadata service is unreachable if the metadata proxy edge is not routable from the VMware Integrated OpenStack API network. VIO7.1 will reach to metadata proxy via management API automatically in the case.

  • 2688209 fixed: The VIO support bundle contains log information only for approximately two days.

    In VIO 6.0 and 7.0, the support bundle can contain the log information for about two days only. This log information is not sufficient for troubleshooting purposes. To resolve this issue, VIO 7.1 enables auto-generated log backups to increase log retention up to 7 days.

  • 2707205 fixed: VIO 6/7 didn't set rp_filter to loose mode on controller nodes

    VIO 6/7 transitioned to PhotonOS and didn't explicitly set rp_filter on controller nodes, which means they're set to "1" (RFC3704 strict mode) by default. Applications on the management network may be unable to connect to the public endpoints as the strict mode setting causes packets to be dropped.

  • 2631412 fixed: VIO GUI uses certificate with name 'Kubernetes Ingress Controller Fake Certificate'

    The certificate name is the default one created by Nginx Ingress Controller. When you upgrade to VIO 7.1, it will be replaced with the one with proper name. 

Resolved OpenStack Issues
  • 2594923 fixed: The vmware_cpu_affinity property of glance image does not work.

    The vmware_cpu_affinity property of glance image does not work. There will be an error when using such a Glance image to create Nova Instances "Error: A list is required in field vmware_cpu_affinity, not a str (HTTP 400)"

    In VIO 7.1, the property could be set as below example:

    openstack image set --property vmware_cpu_affinity="[0,1]" image_name
  • 2753879 fixed: Heat stack was not able to update due to fwaas v1 extension.

    The upstream heat still uses fwaas v1, and doesn't support fwaas v2. However, VIO7 only supports fwaas v2.

  • 2713308 fixed: The unused base glance images is accumulating in nova cache folder.

    The unused base glance images is accumulating in nova cache folder. The backend process for cleaning unused image does not work correctly.

  • 2710808 fixed: VIO7 only support one ns_record in designate pool

    VIO7 only support one ns_record in designate pool, this is improved in VIO7.1 to support multiple ns_records in designate.

  • 2707581 fixed: if a QoS policy is configured as default, external network also uses this QoS policy.

    if a certain QoS policy is configured as default, external network also uses this QoS policy. This is not correct for external network.

  • 2705010 fixed: The loadbalancer size with Octavia could not be changed in VIO7.

    Octavia LBaaS does not provide a way for user to change the loadbalancer size.
    VIO 7.1 provides options for the size configuration.

    default_edge_size = <purpose>:<edge size>[,...]

    Supported purpose are router, dhcp, lb.
    Supported sizes are compact, large, xlarge, quadlarge.
    For example: default_edge_size = lb:xlarge

    Please use "viocli update neutron" to add below parameters under [nsxv] section to configure:

    conf:
  neutron:
  plugins:
    nsx:
      nsxv:
        default_edge_size: lb:xlarge
  • 2701437 fixed: The glance image option "vmware_create_template=false" could not work correctly.

    Glance image could not create and boot Nova instances under below configurations:

    • When the option vmware_create_template is false in glance configuration
    • When user create the glance image using openstack cli with the property "vmware_create_template=false"
      For example:​
    openstack image create --disk-format vmdk --file vmdk --property vmware_create_template=false imageName
  • 2699470 fixed: Can not run nova-manage command in the nova compute pod.

    When run nova-manage command from nova compute pod, there would be DBNonExistentTable exception, because there is no [database] section in nova-compute.conf.

  • 2688655 fixed: Openstack cli authentication via openid does not work on VIO 7.0.1

    Openstack cli authentication via openid does not work on VIO 7.0.1 with Unauthorized (HTTP 401) error.

  • 2674517 fixed: The swap disk defined in flavor not properly mounted in the Nova instance.

    The swap disk defined in flavor not properly mounted in the Nova instance. This is fixed in VIO 7.1 with limitation of not supporting resize VM with swap disk.

  • 2672946 fixed: Nova Compute AZ setup pod runs into CrashLoopBackOff state when host aggregation names has some specific conditions.

    If There is a host aggregate named "nova" in AZ nova and another host aggregate named "xxxxx-nova" which matches "^.*nova$" in the same zone (for example, 5-nova), the nova compute az-setup pod will run into CrashLoopBackOff state.

  • 2656225 fixed: User cannot open VM console with an error "Error: Console is currently unavailable" via Horizon UI

    Horizon could not show VM consoles with an error "Error: Console is currently unavailable" under some situations. This is caused when vmx config path becomes longer, nova-compute cannot insert a row containing the information of a MKS ticket and vmx config path to DB. For example, SvMotion for a VIO instance causes the change of vmx config path to be longer string.

  • 2652286 fixed: LB health monitor deletion fails with "Server-side error: "'NoneType' object has no attribute 'load_balancer_id'"

    This is a bug in the Octavia service with affects the VMware NSX plugin. In some cases the Octavia service fail to retrieve the pool corresponding to a health monitor thus triggering this error. The bug is currently open and tracked at https://storyboard.openstack.org/#!/story/2008231

  • 2678067 fixed: The mouse cursor not works consistently for Windows Nova VM via the Horizon Console

    When access Horizon console on any browser, Cursor for Windows VM will not respond unless tab or control + click is used. Eventually it will stop responding to left click again.

  • 2755304 fixed: Octavia LB status change transaction fails and stuck in PENDING_DELETE status.

    Octavia uses a UNIX socket to communicate internally within the driver agent. Occasionally there is a timeout while writing into this socket, which fails the status change transaction. Retry mechanism is added for the status change transaction.

  • 2643797​ fixed: When configure trusted_dashboard Horizon FQDN is saved as IP address in kystone.conf, not the FQDN name.

    When configure federation settings for keystone, the provided HorizonFQDN are saved as IP address in keystone config file, not as the provided FQDN name.

    viocli update keystone
conf:
  keystone:
    federation:
      trusted_dashboard: https://HorizonFQDN/auth/websso/

Known Issues

  • Public API rate limiting is not available.

    In VMware Integrated OpenStack 7.1, it is not possible to enforce rate limiting on public APIs.

    Workaround: None. This feature will be offered in a later version.

  • Creating a load balancer with a private subnet that is not attached to a router results in an ERROR state

    With the neutron NSX-T plugins such as MP and policy plugins, creating a load balancer with a private subnet that is not attached to a router results in a load balancer that is in an ERROR state and the error will not be reported to the user.

    Workaround: Create the load balancer with a subnet that is attached to a router.

  • If incorrect credentials are entered when deploying OpenStack, the wizard may fail to recognize correct credentials.

    During the OpenStack deployment process, if vCenter Server or NSX Manager credentials are entered incorrectly, the wizard may fail to recognize correct credentials. Even if you remove the incorrect information and enter the correct credentials, the wizard may fail to validate them.

    Workaround: Close the deployment wizard and open it again.

  • OpenStack Port Security cannot be enforced on direct ports in NSX-V Neutron Plugin

    Enabling port-security for ports with vnic-type direct can be ineffective. Security features are not available for direct ports.

    Workaround: None.

  • Cannot log in to VIO if vCenter and NSX password contains $$

    If the VIO account configured for the underlying vCenter and NSX use the password that contains "$$", VIO cannot complete the authentication for vCenter and NSX due to "$$" used in the password. The OpenStack pods can run into CrashLoopBackOff.

    Workaround: Use other passwords that do not contain "$$".

  • Load balancers are stuck in PENDING_XXX state and cannot be operated upon.

    This stuck issue happens for every load balancer which is created, modified, or deleted when the octavia-da in the octavia-api pod crashed.

    Workaround: These loadbalancers cannot be used anymore in octavia. They should be removed from the octavia DB manually.

  • User could not download glance image from openstack cli client

    When download image from openstack cli, there is error: "[Errno 32] Corrupt image download." This is because VIO store the image as VM template in vSphere datastore by default. The md5sum value is not save between VMDK and VM template.

    Workaround: The glance image could be downloaded with below configurations:

    • the option vmware_create_template is false in glance configuration
    • user create the glance image using openstack cli with the property "vmware_create_template=false"
  • After setting a firewall group administratively DOWN (state=DOWN), firewall group operational status is always DOWN, even after the firewall group admin state is brought back UP

    The neutron-fwaas service will ignore changing operational status on transitions that do not involve adding a port or removing a port from the firewall group.

    Workaround: Add or remove a port, or you can add and remove a port that is already bound to the firewall group.

  • Cannot choose not to ignore certificate validation if vCenter and NSX certificate is signed by an intermediate CA

    When the vCenter and NSX certificate is signed by an intermediate CA, some VIO services cannot be configured properly to do certificate validation. Failure can be seen in various formats. For example, cannot unselect "ignore certificate validation" when adding or editing vCenter or NSX.

    Workaround: Choose "ignore certificate validation" from UI and edit vCenter and NSX CR and set spec.insecure to true.

  • Clicking edit and save on a Neutron segment accidentally enables multicast

    In NSX-T policy UI, if any unrelated changes are made in the multicast routing, multicast routing will be enabled on the segment.

    Workaround: Explicitly disable multicast in UI when editing the segment.

  • Add member operations fail with "Provider 'vmwareedge' reports error: Could not retrieve certificate::" (HTTP status 500)

    Cannot add or remove members from HTTPS _TERMINATED Octavia load balancers.

    Workaround: Use OpenStack CLI to add or remove members.

    1. Fetch the tls_container_ref for all the impacted users

    2. Find container, secret, and certificate URIs

    3. Retrieve Octavia service user id

    4. Add URIs retrieved in step 2 to ACLs for user id retrieved in step 3

  • Tier1 gateways could not rollback completely during large-scale MP2P migration

    Some tier1 gateways could not rollback completely, and the deletion status remained in progress during large-scale MP2P migration. Unsuccessful rollback might have caused due to an error during migration.

    Workaround: Restore UA and re-migrate.

  • Duplicate entry error in Keystone Federation

    After deleting the OIDC in Keystone Federation, if the same user tries to log in with OIDC, authentication fails with a 409 message.

    Workaround: Delete the user either through Horizon or OpenStack CLI.

    For example:

    1. In Horizon, log in with an admin account.

    2. Set the domain context with the federated domain.

    3. In the user page, delete the user with User Name column is None.

    In OpenStack CLI

    openstack user list --domain <federated domain name>

    openstack user delete <user id> --domain <federated domain name>

  • Upon successful migration, migrator pod logs are not available in the VIO support bundle
     

    Once the migration succeeds, the VIO control plane gets reconfigured, and the migrator pod gets deleted. Therefore its log is not captured on the support bundle.

    Workaround: The logs for the migrator pod are available on the controller node, and they are executed and stored in /var/log/vmware/mp2p_migration.log. You can retrieve the log files by accessing the controller nodes via viossh. The log files are available only on the controller where the job execution takes place, and therefore it might be necessary to iterate over several controllers until you find them.

  • Fail to enable ceilometer when there are 10k neutron tenant networks.

    When there are large amount resources such as networks created in vSphere, VIO will generate many customer resources for those objects. If CRs number are too big, VIO Manager Web UI will be failed on the backend API because the response data are too large for http request.

    Workaround: In VIO manager, manually delete the discoveries Customer Resources.

    The CRs could be listed by using below command:

    kubectl -n openstack get discoveries.vio.vmware.com

    The CRs could be deleted with below command. For example:

    kubectl -n openstack delete discoveries.vio.vmware.com vcenter-vcenter2-networks-2
  • Certificate needs to be CA signed and re-applied after restoration

    The certs secret which contains vio private key and certificate does not in backup scope currently. After not-in-place restoration, the cert imported previously will not exist in new deployment.

    Workaround:

    1. Save the certs secret from original deployment
    osctl get secret certs -oyaml > certs.yaml
    2. after restoration, replace the "private_key" and "vio_certificate" value in certs secret with the data from step1.
    3. stop/start services.

  • Cannot create instances on a specific nova compute node and the nova compute log stuck.

    When creating an instance, it is in BUILD state and never succeeds. Check in nova-compute log, there is only a few logs and without more information.

    Workaround: Restart the novacompute pod manually.

  • There is no response when saving the firewall rules changes from Horizon UI.

    If any of required options which are marked with "*" is not updated when editing firewall rules, there is no UI response when saving changes.

    Workaround: Please update all required options when editing firewall rules.

  • Some Day2 operations fail to work after changing vCenter username and password from VIO Manager Web UI.

    When user updates vCenter credential in VIO Manager Web UI, OpenStack services could work. But VIO's control plane can not communicate to vCenter since vCenter secret in k8s cloud provider is not updated.

    Workaround: use "kubectl patch secret" command to update the vCenter credential in VIO manager.

    Check the current vc-credential secret info:

    kubectl -n kube-system get secret viocluster1-vc-credentials -o yaml

    Update the vc-credentials secret with new username/password (in base64 format):

    kubectl -n kube-system patch secret viocluster1-vc-credentials --patch \
'{"data": {"your_vcenter.password": "password_in_base64", "your_vcenter.username": "username_in_base64"}}'
  • Horizon UI shows "xmltooling::IOException" when login with SAML Federation IdP.

    When VIO is configured with external SAML IdP, there is "xmltooling::IOException" error when user tries to login with SAML Federation.

    Workaround: Click Refresh button in Browser, user will be proceed to IdP login page.

check-circle-line exclamation-circle-line close-line
Scroll to top icon