VMware Tanzu Kubernetes Grid 1.4.1 | 06 JAN 2022

Check for additions and updates to these release notes.

What's New in v1.4.1

Here are the key new features and capabilities specific to Tanzu Kubernetes Grid v1.4.1. See Tanzu Kubernetes Grid v1.4.0 Release Notes for new features and capabilities that apply to all v1.4.x versions.

  • You can register deployed management clusters in Tanzu Mission Control. For a list of supported infrastructure providers, see Requirements for Registering a Tanzu Kubernetes Cluster with Tanzu Mission Control in the Tanzu Mission Control documentation.
    • The Register TMC pane is removed from the Tanzu Kubernetes Grid installer interface.
    • The tanzu management-cluster register command is removed from the Tanzu CLI.
    • The cluster configuration variable TMC_REGISTRATION_URL is ignored.
  • On vSphere with NSX Advanced Load Balancer configured as the control plane endpoint, Dex and Pinniped services for identity management deploy as service type LoadBalancer. Previously, they deployed as ServiceType: NodePort and you had to integrate NSX Advanced Load Balancer with your identity provider manually.
  • Management clusters created with the prod plan have three worker nodes by default. Previously, the default count was one.
  • Management clusters can be deployed with more control plane and worker nodes than the dev and prod plans define by default, by setting the CONTROL_PLANE_MACHINE_COUNT and WORKER_MACHINE_COUNT variables.
  • In proxied environments, configuration variable TKG_PROXY_CA_CERT lets the proxy server use a different self-signed certificate than is used by the private image registry.
  • (vSphere) Clusters that have NSX Advanced Load Balancer (ALB) as their control plane API endpoint server can use an external identity provider for login authentication, via Pinniped.

Product Snapshot for Tanzu Kubernetes Grid v1.4.1

Tanzu Kubernetes Grid v1.4 supports the following infrastructure platforms and operating systems (OSs), as well as cluster creation and management, networking, storage, authentication, backup and migration, and observability components. The component versions listed in parentheses are included in Tanzu Kubernetes Grid v1.4. For more information, see Component Versions.

vSphere Amazon EC2 Azure
Infrastructure platform vSphere 6.7U3 and later, vSphere 7, VMware Cloud on AWS****, Azure VMware Solution Native AWS* Native Azure*
Cluster creation and management Core Cluster API (v0.3.22), Cluster API Provider vSphere (v0.7.10) Core Cluster API (v0.3.22), Cluster API Provider AWS (v0.6.6) Core Cluster API (v0.3.22), Cluster API Provider Azure (v0.4.15)
Kubernetes node OS distributed with TKG Photon OS 3, Ubuntu 20.04 Amazon Linux 2, Ubuntu 20.04 Ubuntu 18.04, Ubuntu 20.04
Build your own image Photon OS 3, Red Hat Enterprise Linux 7, Ubuntu 18.04, Ubuntu 20.04 Amazon Linux 2, Ubuntu 18.04, Ubuntu 20.04 Ubuntu 18.04, Ubuntu 20.04
Container runtime Containerd (v1.4.6) Containerd (v1.4.6) Containerd (v1.4.6)
Container networking Antrea (v0.13.3), Calico (v3.11.3) Antrea (v0.13.3), Calico (v3.11.3) Antrea (v0.13.3), Calico (v3.11.3)
Container registry Harbor (v2.2.3) Harbor (v2.2.3) Harbor (v2.2.3)
Ingress NSX Advanced Load Balancer Essentials (v20.1.3)**, Contour (v1.17.2),Avi Kubernetes Operator (AKO) (v1.4.3_vmware.1),Avi Controller (v20.1.3 - v20.1.6) Contour (v1.17.2) Contour (v1.17.2)
Storage vSphere Container Storage Interface (v2.3.0***) and vSphere Cloud Native Storage In-tree cloud providers only In-tree cloud providers only
Authentication OIDC via Pinniped (v0.4.4), LDAP via Pinniped (v0.4.4) and Dex OIDC via Pinniped (v0.4.4), LDAP via Pinniped (v0.4.4) and Dex OIDC via Pinniped (v0.4.4), LDAP via Pinniped (v0.4.4) and Dex
Observability Fluent Bit (v1.7.5), Prometheus (v2.27.0), Grafana (v7.5.7) Fluent Bit (v1.7.5), Prometheus (v2.27.0), Grafana (v7.5.7) Fluent Bit (v1.7.5), Prometheus (v2.27.0), Grafana (v7.5.7)
Backup and migration Velero (v1.6.2) Velero (v1.6.2) Velero (v1.6.2)


  • * See Supported AWS and Azure Regions below.
  • ** NSX Advanced Load Balancer Essentials is supported on vSphere 6.7U3, vSphere 7, and VMware Cloud on AWS.
  • *** Version of vsphere_csi_driver. For a full list of vSphere Container Storage Interface components included in this release, see Component Versions.

Supported Kubernetes Versions in Tanzu Kubernetes Grid v1.4.1

Each version of Tanzu Kubernetes Grid adds support for new Kubernetes versions. This version also supports versions of Kubernetes from previous versions of Tanzu Kubernetes Grid.

Tanzu Kubernetes Grid Version Provided Kubernetes Versions Supported in v1.4?
1.4.x 1.21.2, 1.20.8, 1.19.12 YES, YES, YES
1.3.1 1.20.5, 1.19.9, 1.18.17 YES, YES, NO
1.3.0 1.20.4, 1.19.8, 1.18.16, 1.17.16 YES, YES, NO, NO
1.2.1 1.19.3, 1.18.10, 1.17.13 YES, NO, NO
1.2 1.19.1, 1.18.8, 1.17.11 YES, NO, NO

Component Version Updates

  • Contour v1.17.2
  • Envoy v1.18.4

Component Versions

The Tanzu Kubernetes Grid v1.4.1 release includes the following software component versions:

  • ako-operator: v1.4.0+vmware.1
  • alertmanager: v0.22.2+vmware.1
  • antrea: v0.13.3+vmware.1
  • cadvisor: v0.39.1+vmware.1
  • calico_all: v3.11.3+vmware.1
  • cloud-provider-azure: v0.7.4+vmware.1
  • cloud_provider_vsphere: v1.21.0+vmware.1
  • cluster-api-provider-azure: v0.4.15+vmware.1
  • cluster_api: v0.3.22+vmware.1
  • cluster_api_aws: v0.6.6+vmware.1
  • cluster_api_vsphere: v0.7.11+vmware.1
  • configmap-reload: v0.5.0+vmware.1
  • contour: v1.17.2+vmware.1
  • crash-diagnostics: v0.3.3+vmware.1
  • csi_attacher: v3.2.0+vmware.1
  • csi_livenessprobe: v2.2.0+vmware.1
  • csi_node_driver_registrar: v2.1.0+vmware.1
  • csi_provisioner: v2.2.0+vmware.1
  • dex: v2.27.0+vmware.1
  • envoy: v1.18.4+vmware.1
  • external-dns: v0.8.0+vmware.1
  • fluent-bit: v1.7.5+vmware.1
  • gangway: v3.2.0+vmware.2
  • grafana: v7.5.7+vmware.1
  • harbor: v2.2.3+vmware.1
  • imgpkg: v0.10.0+vmware.1
  • jetstack_cert-manager: v1.1.0+vmware.2
  • k8s-sidecar: v1.12.1+vmware.1
  • k14s_kapp: v0.37.0+vmware.1
  • k14s_ytt: v0.34.0+vmware.1
  • kapp-controller: v0.25.0+vmware.1
  • kbld: v0.30.0+vmware.1
  • kube-state-metrics: v1.9.8+vmware.1
  • kube-vip: v0.3.3+vmware.1
  • kube_rbac_proxy: v0.8.0+vmware.1
  • kubernetes-csi_external-resizer: v1.1.0+vmware.1
  • kubernetes-sigs_kind: v1.21.2+vmware.1-v0.8.1
  • kubernetes_autoscaler: v1.21.0+vmware.1, v1.20.0+vmware.1, v1.19.1+vmware.1
  • load-balancer-and-ingress-service: v1.4.3+vmware.1
  • metrics-server: v0.4.0_vmware.1
  • multus-cni: v3.7.1_vmware.2
  • pinniped: v0.4.4+vmware.1
  • prometheus: v2.27.0+vmware.1
  • prometheus_node_exporter: v1.1.2+vmware.1
  • pushgateway: v1.4.0+vmware.1
  • sonobuoy: v0.20.0+vmware.1
  • tanzu-framework: v1.4.1
  • tanzu-framework-addons: v1.4.1
  • tkg-bom: v1.4.1
  • tkg_telemetry: v1.4.0+vmware.1
  • velero: v1.6.2+vmware.1
  • velero-plugin-for-aws: v1.2.1+vmware.1
  • velero-plugin-for-microsoft-azure: v1.2.1+vmware.1
  • velero-plugin-for-vsphere: v1.1.1+vmware.1
  • vsphere_csi_driver: v2.3.0+vmware.1

Supported Upgrade Paths

You can only upgrade to Tanzu Kubernetes Grid v1.4.1 from v1.3.x and v1.4.0. If you want to upgrade to Tanzu Kubernetes Grid v1.4.1 from a version earlier than v1.3.x, you must upgrade to v1.3.x first before upgrading to v1.4.1.

Warning: On vSphere, if you have multiple datacenters running within a single vCenter, do not upgrade to Tanzu Kubernetes Grid v1.4.1.

When upgrading Kubernetes versions on Tanzu Kubernetes clusters, you cannot skip minor versions. For example, you cannot upgrade a Tanzu Kubernetes cluster directly from v1.19.x to v1.21.x. You must upgrade a v1.19.x cluster to v1.20.x before upgrading the cluster to v1.21.x.

Breaking Changes

You must apply the following changes to your Tanzu Kubernetes Grid v1.4.1 environment before upgrading from v1.4.0 and earlier.

  • (AWS only) If you want to use Tanzu Mission Control, you must add permissions to the nodes.tkg.cloud.vmware.com policy. This enables the Tanzu Mission Control resource retriever on AWS. See Tanzu Mission Control for the full list of required permissions.

Resolved Issues

The following issues from Tanzu Kubernetes Grid v1.4.0 have been resolved in v1.4.1.

  • Tanzu Kubernetes Grid management clusters can be registered with Tanzu Mission Control

    You can register Tanzu Kubernetes Grid management clusters with Tanzu Mission Control from the Tanzu Mission Control UI.

    The cluster configuration variable TMC_REGISTRATION_URL and the Tanzu CLI command tanzu management-cluster register are no longer used.

  • System clock is synchronized with NTP server

    Cluster VMs now pick up your NTP configuration provided via DHCP Option 42. This issue previously affected deployments on vSphere.

  • On vSphere, creating or upgrading a workload cluster with multiple control plane nodes does not stall.

    Previously, workload cluster creation or upgrade stalled after the first control plane node that was created or updated dropped off of the VIP network and became intermittently unresponsive.

  • Management cluster and infrastructure can be behind different proxies

    In proxied, internet-restricted environments in which the management cluster and infrastructure (such as vCenter) are on different networks and behind a different proxies, the local bootstrap kind cluster can access and pull container images.

    Resolves issue that prevented access from bootstrap cluster in such environments.

  • Production plan management cluster on AWS has three workers

    Default number of worker nodes deployed in production plan management cluster on AWS has been increased from one to three.

    Previous behavior: After selecting an Amazon EC2 instance type under AZ1 Worker Node Instance Type, AZ2 Worker Node Instance Type, and AZ3 Worker Node Instance Type in the Production view, the installer deployed the management cluster with only one worker node, AZ1 Worker Node Instance Type, instead of three worker nodes.

  • Disconnecting or powering off vCenter host does not cause workload cluster upgrade or deletion to hang.

Known Issues

The following known issues apply specifically to Tanzu Kubernetes Grid v1.4.1. See the Tanzu Kubernetes Grid v1.4.0 Release Notes for known issues that apply to all v1.4.x versions.

  • CAPV controller does not parse datacenter correctly in multi-datacenter vSphere environment.

    Warning: On vSphere, if you have multiple datacenters running within a single vCenter, do not upgrade to Tanzu Kubernetes Grid v1.4.1.

    During upgrade to v1.4.1 on vSphere, if you have multiple datacenters running within a single vCenter, the CAPV controller may fail to find datacenter contents, causing upgrade failure and loss of data.

  • Host network pods and node use the wrong IP in IPv6 clusters.

    When you deploy IPv6 clusters with multiple control plane nodes on vSphere and the clusters use Kubernetes 1.20.x or 1.21.x, one of your nodes as well as the etc, kube-apiserver, and kube-proxy pods may take on the IP you set for the VSPHERE_CONTROL_PLANE_ENDPOINT instead of an IP of their own. You might not see an error, but this could cause networking problems for these pods and prevent the control plane nodes from proper failover. To confirm this is your issue:

    1. Connect to the cluster and run kubectl get pods -A -o wide.
    2. Note the IPs for the etc, kube-apiserver, and kube-proxy pods.
    3. Run kubectl get nodes -o wide.
    4. Note the IP for the first node in the output. Compare the IPs for the pods and node to see if they match the VSPHERE_CONTROL_PLANE_ENDPOINT you set in the cluster configuration file.

    Workaround: None

  • Management cluster installation and upgrade fail in airgapped environment

    In an airgapped environment, running tanzu management-cluster create or tanzu management-cluster upgrade fails when the kind process attempts to retrieve a pause v3.5 image from k8s.gcr.io.

    Workaround: None

  • Management cluster upgrade fails on AWS with Ubuntu v20.04

    On Amazon EC2, with a management cluster based on Ubuntu v20.04 nodes, running tanzu management-cluster upgrade fails after the kind process retrieves an incompatible pause version (v3.6) image from k8s.gcr.io.

    Workaround: None

  • Installer ignores test names entered for LDAP check

    The Tanzu Kubernetes Grid installer interface ignores what you enter in the Test User Name (Optional) and Test Group Name (Optional) fields when verifying LDAP configuration. Instead, it uses cn for the test user name and ou for the test group name when running its LDAP check.

    Workaround: None

  • Management cluster create fails or performance slow with older NSX-T versions and Photon 3 or Ubuntu with Linux kernel 5.8 VMs

    Deploying a management cluster with the following infrastructure and configuration may fail or result in restricted traffic between pods:

    • vSphere with any of the following versions of NSX-T:
      • NSX-T v3.1.3 with Enhanced Datapath enabled
      • NSX-T v3.1.x lower than v3.1.3
      • NSX-T v3.0.x lower than v3.0.2 hot patch
      • NSX-T v2.x. This includes Azure VMware Solution (AVS) v2.0, which uses NSX-T v2.5
    • Base image: Photon 3 or Ubuntu with Linux kernel 5.8

    This combination exposes a checksum issue between older versions of NSX-T and Antrea CNI.

    TMC: If the management cluster is registered with Tanzu Mission Control (TMC) there is no workaround to this issue. Otherwise, see the workarounds below.


    • Upgrade to NSX-T v3.0.2 Hot Patch, v3.1.3, or later, without Enhanced Datapath enabled
    • Use an Ubuntu base image with Linux kernel 5.9 or later.
    • If the management cluster deploys successfully, run the following on all of its nodes:
      • ethtool -K eth0 tx-udp_tnl-segmentation off && ethtool -K eth0 tx-udp_tnl-csum-segmentation off

    This issue has easier workarounds in v1.4.2 and v1.5.

  • "Unable to retrieve complete list of server APIs" error when updating resources

     If you attempt to add or remove a resource for a workload cluster on Amazon EC2 that uses the Calico CNI, you may see an error similar to the following:

    unable to retrieve the complete list of server APIs: data.packaging.carvel.dev/v1alpha1: the server is currently unable to handle the request

    Add a CNI ingress role to the management cluster's resources by following the procedure in Add CNI Ingress Role for Calico in Deploy Tanzu Kubernetes Clusters to Amazon EC2.

    This known issue is resolved in Tanzu Kubernetes Grid v1.4.2 and later.

  • Upgrade failure when vSphere password contains single-quote (') character

    On vSphere, if your vSphere account password contains the single-quote (') character, management cluster upgrade may fail with error: cluster specific secret is not present [...] unable to retrieve vSphere credentials.

  • CLI output from management cluster upgrade on vSphere lists target version as v1.4.0.

    Output from tanzu management-cluster upgrade command upgrading to v1.4.1 on vSphere erroneously lists target version as v1.4.0.

    Workaround: The error can be ignored.

User Documentation

The Tanzu Kubernetes Grid 1.4 documentation applies to all of the 1.4.x releases.

check-circle-line exclamation-circle-line close-line
Scroll to top icon