check-circle-line exclamation-circle-line close-line

VMware Integrated OpenStack 4.1 Release Notes

VMware Integrated OpenStack 4.1 | 18 JAN 2018 | Build 7538136
VMware Integrated OpenStack with Kubernetes 4.1 | 18 JAN 2018 | Build 7549988

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

About VMware Integrated OpenStack

VMware Integrated OpenStack greatly simplifies deploying an OpenStack cloud infrastructure by streamlining the integration process. VMware Integrated OpenStack delivers out-of-the-box OpenStack functionality and an easy configuration workflow through a deployment manager vApp that runs directly in vCenter Server.


VMware Integrated OpenStack version 4.1 is available in English and seven additional languages: Simplified Chinese, Traditional Chinese, Japanese, Korean, French, German, and Spanish. ASCII characters must be used for all input and naming conventions of OpenStack resources (such as project names, user names, image names, and so on) and for the underlying infrastructure components (such as ESXi hostnames, virtual switch port group names, data center names, datastore names, and so on). 

What's New

This release is based on the latest OpenStack Ocata release and provides the following new features and enhancements:

  • Support for the latest versions of VMware products. VMware Integrated OpenStack 4.1 supports and is fully compatible with VMware vSphere 6.5 Update 1, VMware NSX for vSphere 6.3.5, VMware NSX-T 2.1.and vSAN 6.6.1
  • Support for HTML5 client.  VMware Integrated OpenStack using the HTML5 client supports vCenter versions: 6.5.0U1, 6.5.0U1b, 6.5.0U1c, 6.5.0f.
  • Multiple domain LDAP backend. Keystone can now be backed by multiple LDAP domains, enabling support of more complex structures and environments. 
  • Native NSX-T LBaaS. VMware Integrated OpenStack fully supports the new NSX-T native loadbalancer for both VM and container based workloads. 
  • HA Proxy limiting. HA proxy has been updated with additional configuration settings to prevent API overload due to malicious or inadvertent mistakes.
  • VIO-in-a-box. A new deployment option has been introduced to allow for deployment to a single server in an appliance model. 
  • Public OMS APIs.  Management server APIs that can be used directly to automate deployment and lifecycle management of VMware Integrated OpenStack are now documented and available for general consumption. 

VMware Integrated OpenStack with Kubernetes

A container orchestration platform built on Kubernetes is now included with VMware Integrated OpenStack, enabling the privisioning of full infrastructure stacks for application developers. The platform provides the following new features:

  • Support for the latest version of Kubernetes. VMware Integrated OpenStack 4.1 includes and fully supports Kubernetes version 1.8.1.
  • Logging enhancements. Logs can now be forwarded to any syslog compliant destination, enabling integration with existing log management tools.
  • Additional components. The container platform has added support for Helm and Heapster to help manage and monitor deployed applications. 
  • Control plane backup and recovery. This release includes additional tools and best practices to assist with container platform control plane backups. 


The VMware Product Interoperability Matrix provides details about the compatibility of the current version of VMware Integrated OpenStack with VMware vSphere components, including ESXi, VMware vCenter Server, the vSphere Web Client, and optional VMware products. Check the VMware Product Interoperability Matrix also for information about supported management and backup agents before you install VMware Integrated OpenStack or other VMware products.

Upgrading to Version 4.1

Complete multi-step upgrade procedures for both VMware Integrated OpenStack and VMware Integrated OpenStack with Kubernetes upgrades are provided in the product documentation.

Upgrading VMware Integrated OpenStack

You can upgrade directly to VMware Integrated OpenStack 4.1 from a VMware Integrated OpenStack 3.x deployment. The upgrade from VMware Integrated OpenStack 4.0 to VMware Integrated OpenStack 4.1 is a patch process. Both upgrades are documented in the VMware Integrated OpenStack Administration Guide

Upgrading VMware Integrated OpenStack with Kubernetes

To upgrade from VMware Integrated OpenStack with Kubernetes 4.0 to VMware Integrated OpenStack with Kubernetes 4.1, see Upgrade to VMware Integrated OpenStack with Kubernetes 4.1 in the VMware Integrated OpenStack with Kubernetes Getting Started Guide.

Open Source Components for VMware Integrated OpenStack 4.1

The copyright statements and licenses applicable to the open source software components distributed in VMware Integrated OpenStack 4.1 are available on the Open Source tab of the product download page. You can also download the source files for any GPL, LGPL, or other similar licenses that require the source code or modifications to source code to be made available for the most recent available release of VMware Integrated OpenStack.

Resolved Issues

The resolved issues are grouped as follows.

VMware Integrated OpenStack
  • VMware Integrated OpenStack plugin is not displayed in vCenter HTML5 client

    After deploying the VMware Integrated OpenStack vApp, the vCenter HTML5 client occasionally fails to load VMware Integrated OpenStack plugin.

    Workaround: Try to log in again. If that fails to fix the problem, perform the following steps:

    1. Log in to the vSphere web client using the Flash UI.
    2. Click the Home icon and select Administration.
    3. On the Navigator tree, select System Configuration and click Services to display a list of services.
    4. Select VMware vSphere Client.
    5. Click the Actions icon and select Restart.
VMware Integrated OpenStack with Kubernetes
  • No vSphere clusters appear in the UI for an SDDC cloud provider

    If you create an SDDC cloud provider in the UI and choose to upload a Root CA file, no clusters appear on the vSphere Clusters page. This is due to a bug in the certificate verification process. This problem does not occur if you secure your vCenter server with a certificate approved by a trusted certificate authority or if you choose to ignore the vCenter Server certificate validation.

    This issue is resolved in this release.

  • After the admin password is changed, a user cannot list the user and group.

    The cached admin password is out of sync with the new admin password.

    Workaround: To update the cached admin password, obtain the update_admin_pwd script from VMware GSS. Then run update_admin_pwd and run vkube cluster update on each cluster.

  • Error message "Only {0} files are supported" appears when uploading files in the UI

    This error message appears in the UI when an incorrect file type is uploaded. This can occur in either of the following cases:

    • When creating a cloud provider or cluster, you attempt to upload a payload file that previously downloaded during the creation process.
    • When creating a SDDC cloud provider or OpenStack provider, you attempt to upload a CA certificate file.

    Workaround: Apply the workaround that pertains to your case:

    • If attempting to upload a payload file, verify that the payload file has the .json extension and has a valid JSON content.
    • If attempting to upload a CA certificate, verify that the certificate file has the .crt extension.
  • After entering provider name, UI still prompts for provider name

    When creating an SDDC or OpenStack Cloud Provider, the UI displays the error "Provider name is required." even if the provider name is specified in the Add a Provider wizard.

    Workaround: This problem occurs the first time the UI is loaded with browsers of a certain versions. To work around the problem, close the browser tab. Then open a new tab and connect to the virtual appliance again.

Known Issues

The known issues are grouped as follows.

VMware Integrated OpenStack
  • Provider network creation through horizon fails if no UUID of a transport zone is entered

    When you create VLAN type networks, it is mandatory that you provide the UUID value for the transport zone in the provider_network text box in horizon. If no value is entered, network creation fails.

    Workaround: See the UUID of the transport zone in your VMware NSX interface and enter that value in the provider_network text box.

  • Metadata is not accessible for a DHCP disabled subnet on a distributed logical router

    Instances on DHCP disabled subnets cannot access metadata through the interface of the router if a distributed logical router is used. This behavior is not observed for shared and exclusive routers. This might be an expected behavior since same logical networks, for example metadata networks, cannot be attached to multiple distributed logical routers.

    Workaround: None.

  • When you boot from a glance image created using the Ubuntu Xenial OVA, the OS fails to boot

    The OS fails to boot with the following errors:

    error: file `/boot/grub/i386-pc/efi_gop.mod' not found
    error: file `/boot/grub/i386-pc/efi_uga.mod' not found

    This is an issue with the Xenial cloud OVA that is tracked by a bug in the Ubuntu cloud-images project, for more inforamtion see

    Workaround: Until the Ubuntu bug is resolved and new OVA is published, use Xenial ISO images.

  • LBaaS v2 fails when you add members to a pool that is created with the --loadbalancer option

    OpenStack LBaaS v2 provides two options to configure a loadbalancer pool: --loadbalancer and --listener. At least one of the two option must be specified to create the pool.
    If you create the pool for the OpenStack LBaaS v2 with the --loadbalancer option, addition of members fails and the loadbalancer goes to an ERROR state.

    Workaround: Create the pool with the --listener option.

  • Renamed OpenStack instances appears under the old name in vCenter Server

    If you rename your OpenStack instance by using the nova rename command, changes appear only in the OpenStack database. Your vCenter Server instance shows the old name.

    Workaround: None

  • Availability zone configuration might not be successfully applied

    After you modify the configuration of an availability zone, the new configuration might not be applied until the backup edges are deleted and recreated.

    For example, the following configuration in the nsx.ini file features an availability zone that has backup edges:
    If you change the resource pool of that zone and restart Neutron, the backup edges won't be updated. If you deploy new routers or networks, they will use the out-of-date backup edges that leads to inconsistent availability zone configuration.

    Call the admin utilities after you change the configuration of an availability zone and before you start Neutron:

    1. Modify the availability zone configuration in the nsx.ini file.
    2. Delete all backup edges in succession.
      nsxadmin -r backup-edges -o clean --property edge-id=edge-XX
    3. Restart Neutron.
    4. Verify the new configuration.
  • Certificate is not in CA store error might appear when you deploy new OpenStack instance

    When you deploy a new VMware Integrated OpenStack instance with an IP address that was used by another instance that has been connected to vRealize Automation, you might get certificate errors:
    Cannot execute the request: ; Certificate is not in CA store.Certificate is not in CA store. (Workflow:Invoke a REST operation / REST call (item0)#35)

    Workaround: Delete the certificate of the old VMware Integrated OpenStack instance and import the new one by running the respective workflows in vRealize Orchestrator.

    1. Log in to vRealize Orchestrator.
    2. Go to Library > Configuration > SSL Trust Manager.
    3. Run the workflow to delete the trusted certificates of the old VMware Integrated OpenStack instance.
    4. Run the workflow to import the certificate of the new instance from URL.
  • Unable to modify syslog setting post deployment in VMware Integrated OpenStack Manager interface

    After deploying VIO, you cannot modify the syslog server configuration using the setting in the VIO Manager interface (VMware Integrated OpenStack > Management Server > Edit Settings > vApp Options).

    Workaround: Modify the configuration here: VMware Integrated OpenStack > OpenStack Cluster > Manage > Syslog Server.

  • Dashboard might show a Volume as attached even if it failed to attach
    This is a known OpenStack issue, first reported in the Icehouse release.

  • Long image upload times cause NotAuthenticated failure
    This is a known OpenStack issue (, first reported in the Icehouse release.

  • Special characters in datastore names not supported by Glance (Image Service)
    If a datastore name has non-alphanumeric characters like colons, ampersands, or commas, the datastore cannot be added to the Glance service. Specifically, the following characters are not permitted in Glance datastore names because their use is reserved for other purposes and therefore can interfere with the configuration: : , / $ (colon, comma, forward slash, dollar).

    Workaround: Do not use these symbols.

  • If either controller VM reboots, high availability might be compromised

    When a controller fails, the other controller continues to provide services. However, when the initial controller reboots, it might no longer provides services, and thus is not available if the other controller also fails.

    Workaround: If a controller fails and HA is invoked, review your deployment to ensure that both controllers are providing services after the failed controller reboots. For more information about how to start and stop VMware Integrated OpenStack deployments, see VMware knowledge base article 2148892.

  • Metadata service is not accessible on subnets created with the no-gateway option

    Deployments using NSX 6.2.2 or earlier do not support no-gateway networks; Edges are used for edge-routed networks and DHCP is used for VDR networks. Deployments using NSX 6.2.3 or later do not support no-gateway or no-dhcp networks; DHCP is used for any DHCP network and Edges are used for non-DHCP networks. In 2.x, autoconfiguration is turned off for Edge VMs. When applicable, DHCP sets the gateway and metadata is served through this gateway Edge. As a result, when a subnet is created with the the no-gateway option, there is no router Edge to capture the metadata traffic.

    Workaround: For networks with the no-gateway option, configure a route for to forward traffic to DHCP Edge IP.

  • Problem uploading patch file in Firefox Browser

    If you are using Firefox to update the patch for VMware Integrated OpenStack, the upload will fail if Firefox is using version 19 of the Adobe Flash plug-in. For more information, see the Adobe bug base (

    Workaround: Obtain the patch using the CLI. You can also work around this issue by using an alternative browser or restoring the Flash plugin in your Firefox browser to an earlier version (15, 16, 17 or 18.)

  • OpenStack management service does not automatically restart
    Under certain conditions, the OpenStack management service does not automatically restart. For example, after a failover event, all OpenStack services successfully restart but the management service remains unreachable.

    Workaround: Manually restart the VMware Integrated OpenStack vApp in the vSphere Web Client. Right-click the icon in the Inventory page and select Shut Down. After all the services shut down, power on the vApp. Check the OpenStack manager logs to confirm that the restart was successful.

    NOTE: Restarting interrupts services.

  • Network creation might fail when running Heat templates

    Observed in VMware Integrated OpenStack deployments using NSX 6.2.2. When running multiple Heat templates, an iteration of a network creation sometimes fails at the backend.

    Resolved in NSX 6.2.3 and greater.

  • Recovery operation returns "Nodes already exist" error

    Under certain conditions, running the viocli recovery - <DB name> command fails if the ansible operation is interrupted. As a result, the database nodes remain and causes the error.

    Workaround: Manually remove the nodes and run the viocli recovery command again.

  • LBaaS v2 migration: health monitors not associated to a pool do not migrate
    In LBaaS v2, health-monitors are required to specify and be attached to a pool. In LBaaS v1, health monitors can be created without pool association, and associated with a pool in a separate procedure.

    As a result, when migrating to LBaaS v2, unassociated health monitors are excluded.

    Workaround: Before migrating to LBaaS v1, associate all health monitors with a pool to ensure their successful migration. The migration process is optional after installing or upgrading to VMware Integrated OpenStack 3.0. See the VMware Integrated OpenStack Administration Guide.

  • NSX LBaaS v2.0 tenant limitation

    NSX for vSphere load balancers support only one tenant per subnet. Under normal operation, this is not an issue because tenants create their own load balancers. If a user attempts to create and attach a load balancer to a subnet, the load balancer will be created in an ERROR state.

    Workaround: Allow tenants to create their own load balancers. Do not create and attach a load balancer to an existing subnet.

  • Heat stack deletion fails with "Failed to publish configuration on NSX Edge" error

    Observed in deployments using NSX v6.2.2. Under stressful conditions, the Heat stack or OpenStack API might fail at the backend.

    Workaround: Retry the failed operation.

  • Images must be VMX version 10 or greater
    This issue affects streamOptimized images and OVAs. For example, if an image is not VMX-10 or greater, it might import without difficulty but OpenStack instances created from the image will not function. This is typically experienced when OpenStack compute nodes are deployed on older ESXi versions, such as 5.5. You also cannot correct such an image by modifying the image metadata (vmware_hw_version) or flavor metadata (vmware:hw_version).

  • Recovery after vSphere HA event shows synchronization and process start-up failures

    If vSphere experiences and HA event, it can affect your VMware Integrated OpenStack deployment. After the recovery, in VMware Integrated OpenStack, run the viocli deployment status -v command. If the resulting report shows any synchronization or process start-up failures, use the workaround below.

    Workaround: Use the viocli services stop command to stop all OpenStack services. Use the viocli services start command to restart all OpenStack services. After restarting, run the viocli deployment status -v command again. There should be no errors.

  • OpenStack recovery sometimes fails when starting RabbitMQ

    In rare occurrences, VMware Integrated OpenStack recovery fails at the point where RabbitMQ initiates.

    Workaround: Repeat the recovery process. Recovery should succeed the second time.

  • Heat stack deletion fails to delete associated Cinder volumes

    Under heavy loads, Cinder volumes sometimes fail to be deleted after their Heat stacks are deleted, resulting in database deadlock warnings and slower Cinder performance.

    Workaround: There is no workaround.

  • SQL-configured users cannot be modified in dashboard
    If your VMware Integrated OpenStack deployment is configured to use LDAP for user authentication, you cannot modify user definitions in the OpenStack dashboard (Horizon) even those that are sourced from a SQL database.

  • OpenStack dashboard: router-size drop-down menu is missing

    In the OpenStack dashboard (Horizon), you can specify the size when you create an exclusive router. However when you modify a router from shared to exclusive, the router-size drop-down menu does not appear, preventing you from specifying the router size.

    Workaround: Restore the default value, modify type to exclusive again. The drop-down menu should appear.

  • When you attach a firewall to a router without a gateway, the rules are not added to the NSX router

    Firewall as a Service rules are added only to a router with a gateway because those rules have no affect on routers with no gateway, as there in no relevant traffic to protect from.

    Workaround: To add the rules, set a gateway to the router.

  • For VMware NSX-T deployments, when you attach a firewall to a router without a gateway, the rules are added to the NSX router

    Firewall as a Service rules are added to a router without a gateway, even though there is no relevant traffic to match those rules.

    Workaround: To activate the rules, set a gateway to the router.

  • For deployments with NSX for vSphere, updating the admin_state parameter has no impact

    Updating the admin_state parameter to False for a Nova port does nothing, as the parameter is not supported with NSX for vSphere .

    Workaround: There is no workaround.

  • For VMware NSX for vSphere deployments, after you attach a gateway to a MDProxy router, the NSX E dge (MDProxy edge) vnic0 index changes from VM Network to a Gateway network portgroup 

    This configuration might cause implications during fetching of a md proxy server from an OpenStack instance. Admin tenant can attach an mdproxy router to a gateway that results in a block of metadata server access from the OpenStack instance.

    Workaround: Do not attach a gateway to a MDProxy router.

  • Upgrade to 4.0 process might fail with "ssh_exchange_identification: Connection closed by remote host" or "Can’t connect to MySQL server" errors that appear in ansible.log

    Upgrade progress might fail due to iglitches in the infrastructure, such as network latency or resource contention.

    Workaround: Click continue upgrade in the vSphere Web Client to retry the upgrade.

  • BGP tenant networks are lost on service GW edges

    After the BGP peering between the BGPSpeaker and the Service GW is established , running the neutron command neutron bgp-speaker-network-remove to disassociate BGPSpeaker with the external/provider network may cause that the tenant routes on the Service gateways are lost even after the external/provider network to BGPSpeaker is restored using neutron bgp-speaker-network-add. The issue occurs as result of toggling ECMP flags and configuring the BGP at the same time triggered by the Neutron command set.

    Workaround: The default value of 2 seconds for the ecmp_wait_time parameter might not be long enough and must be increased to 5 seconds.This change remediates the missing routes, after performing the same set of operations bgp-speaker-network-remove followed by bgp-speaker-add. Change the property value in the nsxv.ini file.

  • Metadata Agent HTTP communication with Nova Server experiences security risk

    The metadata agent that lives on the edge appliance serves as a reverse proxy and reaches out to an upstream Nova server to gather metadata information about the NSX environment on OpenStack. The nginx reverse proxy configuration also supports plaintext com- munication. The lack of TLS encryption exposes sensitive data to disclosure and attackers on the network can also modify data from the site in transit.

    Workaround: Use HTTPS with CA support instead of HTTP for a secure communication between mdproxy and Nova Server. Perform the following steps.

    1. Enable Nova Metadata HTTPS support by adding the following parameters to nova.conf.

      enabled_ssl_apis = metadata
      ssl_cert_file = <nova metadata https server certificate file path>
      ssl_key_file = <nova metadata https server private key file path>

    2. Log in to the NSX manager, go to System Trust CERTIFICATES to import a CA certificate or chain of certificates as needed and record the UUIDs of the certificates.
    3. Create a HTTPS mdproxy service using REST call.
      1. Prepare a https_mdproxy.json file by using the following format: 

        https_mdproxy.json json format is as bellow:
        "display_name" : "https_md_proxy",
        "resource_type" : "MetadataProxy",
        "metadata_server_url" : "",
        "metadata_server_ca_ids": ["4574853e-e312-4b02-a1c1-002b570c2aa8","940251c9-3500-4bcd-9761-b49d0c2a95d1"],
        "secret": "123",
        "edge_cluster_id" : "00aaf00d-a8ba-42b6-a3bf-9914c8567401"

      2. Deploy one https mdproxy service with CA by calling REST and wait for the NSX Manager to return a

        201 CREATE success status. curl -i -k -u <nsx-mgr-admin:nsx-mgr-passwd>
        -H "content-type: application/json"
        -H "Accept: application/json"
        -X POST https://<nsx-mgr-ip>/api/v1/md-proxies
        -d "`cat ./https_mdproxy.json`


    4.  Record the mdproxy service's uuid and use this mdproxy service as VMware Integrated OpenStack metadata proxy service. The communication between mdproxy and Nova Server is now secured by https with certificates authentication.

  • For NSX-T deployments, newly created tier-0 router does not connect to tier-1 router during router-gateway-set 

    If you have a Tier-0 router configured, when you create a new one, the UUID of the new router does not auto-populate in the nsxv3.ini file. New Tier-1 routers that you create do not connect to your new Tier-0 router. You must manually update the nsxv3.ini file and recreate your external network to fix the issue.

    Workaround: Perform the following steps to resolve the issue.

    1. Get the UUID of your new Tier-0 router.
    2. Open the  /etc/neutron/plugin/vmware/nsxv3.ini file and update the UUID for the Tier-0 router.
    3. Restart the Neutron server.
    4. Delete your your existing external network and create a new one.
  • When you remove a gateway of a BGP enabled shared router, you might observe a brief network outage on other shared BGP enabled routers

    In a shared router case, multiple routers can get hosted on the same edge. The gateway IP of one of those routers is used as router id in the BGP case. When a gateway of the router is cleared, th plugin picks the gateway IP of some other BGP enabled router as router id and modifies that field. During the process, a disruption in the peering occurs, as the advertised routes for any other BGP routers hosted on that edge are lost. Peering recoveres after the peering is reestablished.

    Workaround: There is no workaround. Use an exclusive router to avoid the issue.

  • Tenants traffic might get blocked after enabling NSX policies in Neutron

    After enabling security-group-policy in the Neutron plug-in, the NSX firewall sections order might be wrong. The correct order must be:

    1. NSX policies
    2. Tenants security groups
    3. Default sections

    Workaround: Create the first NSX policy before configuring VMware Integrated OpenStack. If you have already made configurations, go to the NSX Firewall page in the vSphere Web Client and move the policies sections up.

  • Public load balancer IP conflicts with OpenStack API network settings

    For settings configured outside of the UI, the Public Load Balancer IP address and OpenStack API network IP addresses might overlap. When the configuration is exported and re-applied to the OpenStack or VMware Integrated OpenStack setup, the IP overlap will not be allowed.

    Workaround:When providing or configuring IP addresses, ensure that the Public load balancer IP and OpenStack API networks do not overlap.

  • OpenStack UI only exports the original input value for the public Virtual IP

    If the public Virtual IP is changed to a different value from the initial configuration, then the VMware Integrated OpenStack or OpenStack configuration is exported and re-loaded on setup, the exported configuration will contain the public Virtual IP value of the original configuration, not the updated value.

    Workaround: Update the public Virtual IP value in the exported and saved configuration file before re-loading the OpenStack configuration. Or update the public Virtual IP in the UI when confirming the re-deployment.

  • Error occurs when trying to deploy VMware Integrated OpenStack using HTML5 UI

    If you are in using the HTML5 UI and you are trying to deploy VMware Integrated OpenStack using an old template without selecting a deployment type (either HA or compact), the UI throws an internal REST API error. The error occurs in the last step of the deployment wizard.

    Workaround:To work around the problem, select one of the following options:

    • Use the Flex-based UI where this problem does not occur as frequently.
    • When using the HTML5 UI, do not use a template to deploy VMware Integrated OpenStack or use a template that includes deployment_type
    • When using an older template, ensure the deployment type is manually selected.
  • Load balancer goes into an ERROR state

    If a load balancer is created using a subnet that is not connected to a Tier 1 network router, then the load balancer is not created and will go into an ERROR state.

    Workaround: Attach a Tier 1 network router to the subnet before creating a load balancer.

  • Instance boot fails

    If a user tries to deploy VMware Integrated OpenStack instances while the system is under a heavy load, for example, if Keystone is inundated with API requests, Keystone will fail to serve with the error "Service Unavailable".

    Workaround: Retry operations when the load on the system is lower.

  • Service disruption occurs during refresh of Nova or Neutron services

    When VMware Integrated OpenStack detects an OpenStack setting that does not conform to license mandates, it tries to correct the setting by restarting Nova or Neutron services.

    workaround: None. However to verify that OpenStack settings do conform to license mandates, a best practice is to apply the license before deploying OpenStack.

  • Using iBGP peering between the DLR tenant edge (PLR) and provider gateway edge fails to properly advertise the tenant network and breaks external communication

    When iBGP peering is used, advertised routes are installed in the peers without any modifications in the nexthop. As a result, the provider gateway edge installs routing between tenant networks with the nexthop IP in the 'transit network' range instead of the tenant's PLR edge uplink. Since the gateway edge cannot resolve the route lookup for transit network, communication breaks.

    Workaround: Use eBGP peering when working with distributed routers.

  • Cloud Services Router contains IP address only and not the FQDN

    During VMware Integrated OpenStack deployment, the public hostname was not specified in the Load balancer configuration or did not conform to requirements for public access. The public hostname is used for external access to Horizon or APIs.

    Workaround: To change or edit the public hostname after deployment, see

  • Customization of policy files does not sync to Horizon

    The UI does not honor changes to the policy specified in a custom playbook.

    Workaround. None. If a custom playbook is used to edit policy files, make the same changes in the Horizon policy files to ensure consistency.

  • Boot Nova instance fails with error "no valid host found"

    Booting an instance using the tenant_vdc property under stress conditions may fail.

    Workaround. Retry.

VMware Integrated Openstack with Kubernetes
  • After cycling power on a VMware Integrated OpenStack with Kubernetes VM with an SDDC provider, OpenStack service containers stop working and do not restart automatically.

    If a VMware Integrated OpenStack with Kubernetes VM has one SDDC provider deployed, and the VM is powered off and on to address an issue with an ESXi host for example, the VM is migrated to another host. Subsequent operations on the provider such as kubernetes cluster creation and scaleout will fail.

    Workaround: To refresh the provider, perform the following steps:

    1. On the VMware Integrated OpenStack with Kubernetes VM, login as root with the password set during OVA deployment:
      vkube login --insecure
    2. Obtain the SDDC provider ID:
      vkube provider list --insecure
    3. Use the provider ID to refresh the SDDC provider:
      vkube provider refresh sddc <provider_id> --insecure 


  • SDDC Cloud Provider creation fails with "dpkg: unrecoverable fatal error, aborting:"

    Creating an SDDC Cloud Provider fails with an error written to the logs of the column-api container on the Virtual Appliance. The message that appears is similar to the following.

    - docker logs column-api -fTASK [bootstrap-os : Bootstrap | Install python 2.x and pip] ******************* - - [06/Sep/2017 05:47:32] "GET /runs/46a74449-7123-4574-90c2-3404dfac6641 HTTP/1.1" 200 -fatal: [k8s-node-1-2393e79d-ec6a-4e63-8f63-c6308d72496e]: FAILED! => {"changed": true, "failed": true, "rc": 100, "stderr": "Shared connection to closed.", "stdout": "........"dpkg: unrecoverable fatal error, aborting:", " files list file for package 'python-libxml2' is missing final newline", "E: Sub-process /usr/bin/dpkg returned an error code (2)"]}"

    Workaround: The error is a result of file corruption during SDDC cloud provider creation. Delete the SDDC cloud provider and re-create it.

  • After guest OS of Kubernetes cluster node is restarted, flannel pod does not start up correctly

    When the guest OS of Kubernetes cluster node is restarted, all IP tables rules are cleaned up. As a result, the flannel pod does start up correctly.

    Workaround: Restart the kube-proxy. If you kill the kube-proxy process directly, the hyper-kube will start a new kube-proxy automatically.

  • When performing operations on the cluster, user sees "No policy assigned"

    A user that is a member of a group assigned to either an exclusive or shared cluster sees "No policy assigned" when performing operations on the cluster, such as running the kubectl. This occurs because the group information of the authenticated user is not stored correctly during the user session.

    Workaround:Assign an individual user to the cluster instead of a group.

  • Cannot restore cluster if cluster is deleted

    Once the Delete Cluster and Delete Provider commands have been run, networks, routers, and loadbalancers that have been deleted cannot be recovered.

    workaround: None. Re-create the environment.

  • Cannot access Kubernetes API server using virtual IP address

    When deploying multiple Kubernetes clusters with an OpenStack provider in an NSX-T network environment, sometimes you cannot access the Kubernetes API server using virtual IP address.

    Workaround: Log in to the NSX-T backend server and update the load balancer virtual server with the floating IP address.

  • No external routing after recovery. Cluster using an SDDC provider with VDS backend appears 'ACTIVE'.

    If the pod nginx-ingress-controller-xxxx is in an error state after recovery, no external routing occurs.

    Workaround: Perform the following steps to clear the error state.

    1. Run a test command on the cluster:
      kubectl delete serviceaccount default -n kube-system
      kubectl delete pod nginx-ingress-controller-xxxx -n kube-system
    2. Run vkube cluster update on the OVA.