check-circle-line exclamation-circle-line close-line

Check this site regularly for additions and updates to these release notes.

VMware Enterprise PKS is used to create and manage on-demand Kubernetes clusters.



Release Date: November 14, 2019


This section describes new features and changes in this release.

PKS Control Plane and API

Enterprise PKS v1.6.0 updates include:

Kubernetes Control Plane

Enterprise PKS v1.6.0 updates include:

  • Increases the Worker VM Max in Flight default value from 1 to 4 in the PKS API configuration pane, which accelerates cluster creation by allowing up to four new nodes to be provisioned simultaneously. The updated default value is only applied during new Enterprise PKS installation and is not applied during an Enterprise PKS upgrade. If you are upgrading Enterprise PKS from a previous version and want to accelerate multi-cluster provisioning, you can increase the value of Worker VM Max in Flight manually.

PKS Monitoring and Logging

Enterprise PKS v1.6.0 updates include:

  • Redesigns the Logging and Monitoring panes of the Enterprise PKS tile and renames them to Host Monitoring and In-Cluster Monitoring. For information about configuring these panes, see the Installing Enterprise PKS topic for your IaaS.
  • Adds the Max Message Size field in the Host Monitoring pane. This allows you to configure the maximum number of characters of a log message that is forwarded to a syslog endpoint. This feature helps ensure that log messages are not truncated at the syslog endpoint. By default, the Max Message Size field is 10,000 characters. For more information, see Host Monitoring in the Installing Enterprise PKS topic for your IaaS.
  • Adds the Include kubelet metrics setting. This enables operators to collect workload metrics across all Kubernetes clusters. For more information, see Host Monitoring in the Installing Enterprise PKS topic for your IaaS.
  • Adds support for Fluent Bit output plugins to log sinks. For information about configuring Fluent Bit output plugins, see Create a ClusterLogSink or LogSink Resource with a Fluent Bit Output Plugin in Creating Sink Resources.
  • Adds support for filtering logs and events from a ClusterLogSink or LogSink resource. For more information, see Filter Sinks in Creating Sink Resources.

Windows on PKS

Enterprise PKS v1.6.0 updates include:

  • Adds support for floating Windows stemcells on vSphere. For information about Kubernetes clusters with Windows workers in Enterprise PKS, see Configuring Windows Worker-Based Kubernetes Clusters (Beta).
  • Enables operators to configure the location of the Windows pause image. For information about configuring Kubelet customization - Windows pause image location, see Plans in Configuring Windows Worker-Based Kubernetes Clusters (Beta).

PKS with NSX-T Networking

Enterprise PKS v1.6.0 updates include:

  • NSX Error CRD lets cluster managers and users view NSX errors in Kubernetes resource annotations, and use the command kubectl get nsxerror to view the health status of NSX-T cluster networking objects (NCP v2.5.0+). For more information, see Viewing the Health Status of Cluster Networking Objects (NSX-T only).
  • DFW log control for dropped traffic lets cluster administrators define network profile to turn on logging and log any dropped or rejected packet by NSX-T distributed firewall rules (NCP v2.5.0+). For more information, see Defining Network Profiles for NCP Logging.
  • Load balancer and ingress resource capacity observability using the NSXLoadBalancerMonitor CRD lets cluster managers and users use the command kubectl get nsxLoadBalancerMonitors to view a health score that reflects the current performance of the NSX-T load balancer service, including usage, traffic, and current status (NCP v2.5.1+). For more information, see Ingress Scaling (NSX-T only).
  • Ingress scale out using the LoadBalancer CRD lets cluster managers scale out the NSX-T load balancer for ingress routing (NCP v2.5.1+). For more information, see Ingress Scaling (NSX-T only).
  • Support for Ingress URL Rewrite. For more information, see Using Ingress URL Rewrite.
  • Support for Active–Active Tier-0 router configuration when using a Shared-Tier-1 topology.
  • Ability to place the load balancer and Tier-1 Active/Standby routers on different failure domains. See Multisite Deployment of NSX-T Data Center for more information.

PKS on AWS Networking

Enterprise PKS v1.6.0 updates include:

Enterprise PKS Management Console

VMware Enterprise PKS Management Console v1.6.0-rev.2 updates include:

Customer Experience Improvement Program

Enterprise PKS v1.6.0 updates include:

  • Administrators can name Enterprise PKS installations so they are more easily recognizable in reports. For more information, see Sample Reports.

Component Updates

Enterprise PKS v1.6.0 updates include:

  • Bumps Kubernetes to v1.15.5.
  • Bumps UAA to v73.4.8.
  • Bumps Jackson dependencies in the PKS API.

Bug Fixes

Enterprise PKS v1.6.0 includes the following bug fixes:

  • Fixes an issue where enabling the Availability Sets mode at the BOSH Director > Azure Config resulted in the kubelet failing to start on provisioning of a Kubernetes cluster.
  • Fixes an issue where persistent volume attachment failed on vSphere in a scenario where an AZ defined in Ops Manager does not contain a resource pool.
  • Increases network_profile column size.
  • Fixes a Telemetry event generation issue where the upgrade_cluster_end event is not sent for completed cluster upgrades.
  • Fixes an issue where networking changes did not propagate when upgrading from Enterprise PKS v1.5 or later.
  • Fixes an issue where the Ingress IP address was excluded from the Enterprise PKS floating IP pool.
  • Fixes an issue where the PKS OSB Proxy start was delayed by scanning all NSX-T firewall rules.
  • Fixes an issue with the PKS clusters upgrade errand not pushing the latest NSX-T certificate to Kubernetes Master nodes.
  • Fixes an issue with the PKS OSB Proxy taking a long time to start due to scanning all NSX-T firewall rules.
  • Fixes an issue with PKS releasing floating IP addresses incompletely while deleting clusters under active/active mode.
  • Fixes an issue with the DNS Lookup Feature: INGRESS IP not kept out of PKS Floating IP pool.
  • Fixes an issue with the command pks cluster details does not display NS Group ID of master VMs.
  • Checks the high availability mode of the Tier-0 router before creating PKS a cluster.

Product Snapshot

Component Details
PKS version v1.6.0
Release Date November 14, 2019
Compatible Ops Manager versions *

v2.6.13 and later or v2.7.3 and later

Xenial Stemcell version v456.51
Windows Stemcell version v2019.7
Kubernetes version v1.15.5
On-Demand Broker version v0.29.0
NSX-T versions

v2.5.0, v2.4.3

NCP version v2.5.1
Docker version v18.09.8
Backup and Restore SDK version v1.17.0
UAA version 73.4.8
vSphere versions Refer to the VMware Product Interoperability Matrices

VMware Enterprise PKS Management Console Product Snapshot

Note: Enterprise PKS Management Console provides an opinionated installation of Enterprise PKS. The supported versions may differ from or be more limited than what is generally supported by Enterprise PKS.

Element Details
Version v1.6.0-rev.2
Release date November 26, 2019
Installed Enterprise PKS version v1.6.0
Installed Ops Manager version v2.7.3
Installed Kubernetes version v1.15.5
Supported NSX-T versions v2.5.0, v2.4.3
Installed Harbor Registry version v1.9.3

Upgrade Path

The supported upgrade paths to Enterprise PKS v1.6.0 are from Enterprise PKS v1.5.0 and later.

Note: Windows worker support is BETA. There is no upgrade support from PKS v1.5 to v1.6 for Windows worker nodes.

Breaking Changes

Enterprise PKS v1.6.0 has the following breaking changes:

Enterprise PKS Removes Sink Commands in the PKS CLI

Enterprise PKS removes the following Enterprise PKS Command Line Interface (PKS CLI) commands:

  • pks create-sink
  • pks sinks
  • pks delete-sink

You can use the following Kubernetes CLI commands instead:

  • kubectl apply -f YOUR-SINK.yml
  • kubectl get clusterlogsinks
  • kubectl delete clusterlogsink YOUR-SINK

For more information about defining and managing sink resources, see Creating Sink Resources.

Changes to PKS API Endpoints

This release moves the clusters, compute-profiles, quotas, and usages PKS API endpoints from v1beta1 to v1. v1beta1 is no longer supported for these endpoints. You must use v1. For example, instead of https://YOUR-PKS-API-FQDN:9021/v1beta1/quotas, use https://YOUR-PKS-API-FQDN:9021/v1/quotas.

Known Issues

Enterprise PKS v1.6.0 has the following known issues.

Network profile for “pks update-cluster” does not use the defaults from the original cluster manifest


The Network profile for pks update-cluster uses contents that are being updated and not using the defaults from the original cluster manifest.


The pks update-cluster operation sets the subnet_prefix to 0 in the ncp.ini file when the network-profile has pod_ip_block_ids set but it does not have pod_subnet_prefix.


When creating the network profile to be used for update, include all the below fields. Then update-cluster with the network profile should work.

{ "name": "np", "parameters": { "t0_router_id": "c501f114-870b-4eda-99ac-966adf464452", "fip_pool_ids": ["b7acbda8-46de-4195-add2-5fb11ca46cbf"], "pod_ip_block_ids": ["b03bff60-854b-4ccb-9b2b-016867b319c9","234c3652-69e7-4365-9627-8e0d8d4a6b86"], "pod_subnet_prefix": 24, "single_tier_topology": false } } 

Azure Default Security Group Is Not Automatically Assigned to Cluster VMs


You experience issues when configuring a load balancer for a multi-master Kubernetes cluster or creating a service of type LoadBalancer. Additionally, in the Azure portal, the VM > Networking page does not display any inbound and outbound traffic rules for your cluster VMs.


As part of configuring the Enterprise PKS tile for Azure, you enter Default Security Group in the Kubernetes Cloud Provider pane. When you create a Kubernetes cluster, Enterprise PKS automatically assigns this security group to each VM in the cluster. However, on Azure the automatic assignment may not occur.

As a result, your inbound and outbound traffic rules defined in the security group are not applied to the cluster VMs.


If you experience this issue, manually assign the default security group to each VM NIC in your cluster.

Cluster Creation Fails When First AZ Runs Out of Resources


If the first availability zone (AZ) used by a plan with multiple AZs runs out of resources, cluster creation fails with an error like the following:

L Error: CPI error 'Bosh::Clouds::CloudError' with message 'No valid placement found for requested memory: 4096


BOSH creates VMs for your Enterprise PKS deployment using a round-robin algorithm, creating the first VM in the first AZ that your plan uses. If the AZ runs out of resources, cluster creation fails because BOSH cannot create the cluster VM.

For example, if you have three AZs and you create two clusters with four worker VMs each, BOSH deploys VMs in the following AZs:

  AZ1 AZ2 AZ3
Cluster 1 Worker VM 1 Worker VM 2 Worker VM 3
  Worker VM 4    
Cluster 2 Worker VM 1 Worker VM 2 Worker VM 3
  Worker VM 4    

In this scenario, AZ1 has twice as many VMs as AZ2 or AZ3.

Azure Worker Node Communication Fails after Upgrade


Outbound communication from a worker node VM fails after upgrading Enterprise PKS.


Enterprise PKS uses Azure Availability Sets to improve the uptime of workloads and worker nodes in the event of Azure platform failures. Worker node VMs are distributed evenly across Availability Sets.

Azure Standard SKU Load Balancers are recommended for the Kubernetes control plane and Kubernetes ingress and egress. This load balancer type provides an IP address for outbound communication using SNAT.

During an upgrade, when BOSH rebuilds a given worker instance in an Availability Set, Azure can time out while re-attaching the worker node network interface to the back-end pool of the Standard SKU Load Balancer.

For more information, see Outbound connections in Azure in the Azure documentation.


You can manually re-attach the worker instance to the back-end pool of the Azure Standard SKU Load Balancer in your Azure console.

Error During Individual Cluster Upgrades


While submitting a large number of cluster upgrade requests using the pks upgrade-cluster command, some of your Kubernetes clusters are marked as failed.


BOSH upgrades Kubernetes clusters in parallel with a limit of up to four concurrent cluster upgrades by default. If you schedule more than four cluster upgrades, Enterprise PKS queues the upgrades and waits for BOSH to finish the last upgrade. When BOSH finishes the last upgrade, it starts working on the next upgrade request.

If you submit too many cluster upgrades to BOSH, an error may occur, where some of your clusters are marked as FAILED because BOSH can start the upgrade only within the specified timeout. The timeout is set to 168 hours by default. However, BOSH does not remove the task from the queue or stop working on the upgrade if it has been picked up.


If you expect that upgrading all of your Kubernetes clusters takes more than 168 hours, do not use a script that submits upgrade requests for all of your clusters at once. For information about upgrading Kubernetes clusters provisioned by Enterprise PKS, see Upgrading Clusters.

Kubectl CLI Commands Do Not Work after Changing an Existing Plan to a Different AZ


After you update the AZ of an existing plan, kubectl CLI commands do not work for your clusters associated with the plan.


This issue occurs in IaaS environments that do not support attaching a disk across multiple AZs.

When the plan of an existing cluster changes to a different AZ, BOSH migrates the cluster by creating VMs for the cluster in the new AZ and removing your cluster VMs from the original AZ.

On an IaaS that does not support attaching VM disks across AZs, the disks BOSH attaches to the new VMs do not have the original content.


If you cannot run kubectl CLI commands after reconfiguring the AZ of an existing cluster, contact Support for assistance.

One Plan ID Longer than Other Plan IDs


One of your plan IDs is one character longer than your other plan IDs.


In Enterprise PKS, each plan has a unique plan ID. A plan ID is normally a UUID consisting of 32 alphanumeric characters and 4 hyphens. However, the Plan 4 ID consists of 33 alphanumeric characters and 4 hyphens.


You can safely configure and use Plan 4. The length of the Plan 4 ID does not affect the functionality of Plan 4 clusters.

If you require all plan IDs to have identical length, do not activate or use Plan 4.

Kubernetes Cluster Name Limitation for Tanzu Mission Control Integration

Tanzu Mission Control integration cannot attach Tanzu Mission Control to Kubernetes clusters that have uppercase letters in their names.


Clusters that you create with pks create-cluster do not appear in the Tanzu Mission Control, even though you configured Tanzu Mission Control integration as described in Integrate Tanzu Mission Control.


The regex pattern that parses cluster names in Tanzu Mission Control integration fails with names that contain uppercase letters.


When running pks create-cluster to create clusters that you want to track in Tanzu Mission Control, pass in names that contain only lowercase letters and numbers.

Enterprise PKS Management Console Known Issues

The following known issues are specific to the Enterprise PKS Management Console v1.6.0-rev.2 appliance and user interface.

YAML Validation Errors Not Cleared


If you attempt to upload a YAML configuration file and the deployment fails because of an invalid manifest, Enterprise PKS Management Console displays an error notification with the validation error. If subsequent attempts also fail because of validation issues, the validation errors are appended to each other.


The validation errors are not cleared when you resubmit the YAML configuration file.



Enterprise PKS Management Console Notifications Persist


In the Enterprise PKS view of Enterprise PKS Management Console, error notifications sometimes persist in memory on the Clusters and Nodes pages after you clear those notifications.


After clicking the X button to clear a notification it is removed, but when you navigate back to those pages the notification might show again.


Use shift+refresh to reload the page.

Cannot Delete Enterprise PKS Deployment from Management Console


In the Enterprise PKS view of Enterprise PKS Management Console, you cannot use the Delete Enterprise PKS Deployment option even after you have removed all clusters.


The option to delete the deployment is only activated in the management console a short period after the clusters are deleted.


After removing clusters, wait for a few minutes before attempting to use the Delete Enterprise PKS Deployment option again.

Configuring Enterprise PKS Management Console Integration with VMware vRealize Log Insight


Enterprise PKS Management Console appliance sends logs to VMware vRealize Log Insight over HTTP, not HTTPS.


When you deploy the Enterprise PKS Management Console appliance from the OVA, if you require log forwarding to vRealize Log Insight, you must provide the port on the vRealize Log Insight server on which it listens for HTTP traffic. Do not provide the HTTPS port.


Set the vRealize Log Insight port to the HTTP port. This is typically port 9000.

Deploying Enterprise PKS to an Unprepared NSX-T Data Center Environment Results in Flannel Error


When using the management console to deploy Enterprise PKS in NSX-T Data Center (Not prepared for PKS) mode, if an error occurs during the network configuration, the message Unable to set flannel environment is displayed in the deployment progress page.


The network configuration has failed, but the error message is incorrect.


To see the correct reason for the failure, see the server logs. For instructions about how to obtain the server logs, see Troubleshooting Enterprise PKS Management Console.

Using BOSH CLI from Operations Manager VM


The BOSH CLI client bash command that you obtain from the Deployment Metadata view does not work when logged in to the Operations Manager VM.


The BOSH CLI client bash command from the Deployment Metadata view is intended to be used from within the Enterprise PKS Management Console appliance.


To use the BOSH CLI from within the Operations Manager VM, see Connect to Operations Manager.

From the Ops Manager VM, use the BOSH CLI client bash command from the Deployment Metadata page, with the following modifications:

  • Remove the clause BOSH_ALL_PROXY=xxx.
  • Replace the BOSH_CA_CERT section with BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate.

Run pks Commands against the PKS API Server


The PKS CLI is available in the Enterprise PKS Management Console appliance.


To be able to run pks commands against the PKS API Server, you must first log to PKS using the following command syntax pks login -a fqdn_of_pks ....

To do this, you must ensure either of the following:

  • The FQDN configured for the PKS Server is resolvable by the DNS server configured for the Enterprise PKS Management Console appliance, or
  • An entry that maps the Floating IP assigned to the PKS Server to the FQDN exists on /etc/hosts in the appliance. For example: api.pks.local.