VMware vSphere 8.0 | 11 OCT 2022

ESXi 8.0 | 11 OCT 2022 | Build 20513097

vCenter Server 8.0 | 11 OCT 2022 | Build 20519528

VMware NSX Advanced Load Balancer 21.1.4 | 07 APR 2022

Check for additions and updates to these release notes.

What's in the Release Notes

The release notes cover the following topics:

What's New

vSphere with Tanzu 8.0 introduces the following new features and enhancements:

  • Workload Management Configuration Monitoring

    You can now track the status of activation, deactivation, and upgrade of the Supervisor in greater detail. Once you initiate a Supervisor activation, deactivation, or upgrade, the Supervisor tries to reach the desired state by reaching various conditions associated with different components of the Supervisor, such as Control Plane VMs. You can track the status of each condition, view the associated warnings, retry the status, view which conditions are reached, and their time stamps.

  • vSphere Zones

    A vSphere Zone is a new construct that lets you assign availability zones to vSphere clusters. Deploying a Supervisor across vSphere Zones lets you provision Tanzu Kubernetes Grid clusters in specific availability zones and failure domains. This allows for workloads to span across multiple clusters, which increases the resiliency to hardware and software failures.

  • Multi-Cluster Supervisor

    By using vSphere Zones, you can deploy a Supervisor across multiple vSphere clusters to provide high availability and failure domains for Tanzu Kubernetes Grid clusters. This release is not intended for metro cluster deployments. The use case for a multi-cluster Supervisor is to have vSphere clusters placed in separate racks in a single physical datacenter. Then you add each vSphere cluster to a separate vSphere zone. This provides failover and resiliency to localized hardware and software failures. When a zone, rack, or vSphere cluster goes offline, the Supervisor detects the failure and restarts workloads on another vSphere zone.

  • Supervisor Supports Kubernetes 1.23

    This release adds support for Kubernetes 1.23 and drops the support for Kubernetes 1.20. The supported versions of Kubernetes in this release are 1.23, 1.22, and 1.21. Supervisors running on Kubernetes version 1.20 will be auto-upgraded to version 1.21 to ensure that all your Supervisors are running on the supported versions of Kubernetes.

  • Provide consistent network policies with SecurityPolicy CRDs

    SecurityPolicy custom resource definition provides the ability to configure network security controls for VMs and vSphere Pods in a Supervisor namespace.

  • Tanzu Kubernetes Grid 2.0

    Tanzu Kubernetes Grid now has moved to version 2.0. Tanzu Kubernetes Grid 2.0 is the culmination of tremendous innovation in the Tanzu and Kubernetes community, and provides the foundation for a common set of interfaces across private and public clouds with Tanzu Kubernetes Grid.  New in this release are the following two major features:

    • ClusterClass - Tanzu Kubernetes Grid 2.0 now supports ClusterClass.  ClusterClass provides an upstream-aligned interface, superseding the Tanzu Kubernetes Cluster, which brings customization capabilities to our Tanzu Kubernetes Grid platform.  ClusterClass enables administrators to define templates that will work with their enterprise environment requirements while reducing boilerplate and enabling delegated customization.  

    • Carvel - Carvel provides a set of reliable, single-purpose, composable tools that aid in application building, configuration, and deployment to Kubernetes.  In particular, kapp and kapp-controller provide package management, compatibility, and lifecycle through a set of declarative, upstream-aligned tooling.  Coupled with ytt for templating, this results in a flexible yet manageable package management capability.  

  • New documentation structure and landing page in vSphere 8

    The vSphere with Tanzu documentation now has improved structure that better reflects the workflows with the product and allows you to have more focused experience with the content. You can also access all the available technical documentation for vSphere with Tanzu from the new documentation landing page.

Installation and Upgrades for This Release

Read the Installing and Configuring vSphere with Tanzu documentation for guidance about installing and configuring vSphere with Tanzu. For information about updating vSphere with Tanzu, see the Maintaining vSphere with Tanzu documentation.

When you perform your upgrades, keep in mind the following:

  • Before you update to vCenter Server 8.0, make sure that the Kubernetes version of all Supervisors is of minimum 1.21, preferably the latest supported. The Tanzu Kubernetes release version of the Tanzu Kubernetes Grid clusters must be of 1.21, preferably the latest supported.

  • Upgrades from legacy TKGS TKr to TKG 2 TKr are blocked in vSphere with Tanzu 8.0. You will be able to upgrade to the zshippable TKr starting with the vSphere with Tanzu 8.0 MP1 release. Refer to the Tanzu Kuberentes releases Release Notes for details. See also Using Tanzu Kubernetes Grid 2 with vSphere with Tanzu.

Known Issues

Supervisor Known Issues

  • If a DevOps user starts volume operations or stateful application deployments while vCenter Server reboots, the operations might fail

    The problem occurs because the workload storage management user account gets locked, and the vSphere CSI plug-in that runs on the Supervisor fails to authenticate. As a result, the volume operations fail with InvalidLogin errors.

    The log file /var/log/vmware/vpxd/vpxd.log displays the following message:

    Authentication failed: The account of the user trying to authenticate is locked. :: The account of the user trying to authenticate is locked. :: User account locked: {Name: workload_storage_management-<id>, Domain: <domain>})


    1. Obtain the account unlock time.

      1. In the vSphere Client, navigate to Administration, and click Configuration under Single Sign On.

      2. Click the elect Accounts tab.

      3. Under Lockout Policy, get the Unlock time in seconds.

    2. Authenticate with the Supervisor using the vSphere Plugin for kubectl.

      kubectl vsphere login –server IP-ADDRESS --vsphere-username USERNAME

    3. Note down the original replica count for vsphere-csi-controller deployment in vmware-system-csi- namespace.

      kubectl get deployment vsphere-csi-controller -n vmware-system-csi -o=jsonpath='{.spec.replicas}'


    4. Scale down the vsphere-csi-controller deployment replica count to zero.

      kubectl scale --replicas=0 deployment vsphere-csi-controller -n vmware-system-csi

    5. Wait for the number of seconds indicated under Unlock time.

    6. Scale up the vsphere-csi-controller deployment replica count to the original value.

      kubectl scale --replicas=original-replica-count deployment vsphere-csi-controller -n vmware-system-csi

  • When deleting multiple FCDs and volumes from shared datastores such as vSAN, you might notice changes in performance

    The performance changes can be caused by a fixed issue. While unfixed, the issue caused stale FCDs and volumes to remain in the datastore after an unsuccessful FCD delete operation.

    Workaround: None. The delete operation works as usual despite the change in the performance.

  • When you delete a Supervisor namespace that contains a Tanzu Kubernetes Grid cluster, persistent volume claims present in the Supervisor might remain in terminating state

    You can observe this issue when a resource conflict occurs while the system deletes the namespace and detaches volumes from the pods in the Tanzu Kubernetes Grid cluster.

    The deletion of the Supervisor namespace remains incomplete, and the vSphere Client shows the namespace state as terminating. Persistent volume claims that were attached to pods in the Tanzu Kubernetes Grid cluster also remain in terminating state.

    If you run the following commands, you can see the Operation cannot be fulfilled on persistentvolumeclaims error:

    kubectl vsphere login --server=IP-ADDRESS --vsphere-username USERNAME

    kubectl logs vsphere-csi-controller-pod-name -n vmware-system-csi -c vsphere-syncer

    failed to update PersistentVolumeClaim: \\\"<pvc-name>\\\" on namespace: \\\"<supervisor-namespace>\\\". Error: Operation cannot be fulfilled on persistentvolumeclaims \\\"<pvc-name>\\\": the object has been modified; please apply your changes to the latest version and try again\


    Use the following commands to fix the issue:

    kubectl vsphere login --server=IP-ADDRESS --vsphere-username USERNAME

    kubectl patch pvc <pvc-name> -n <supervisor-namespace> -p '{"metadata":{"finalizers":[]}}' --type=merge

  • If a vSphere admistrator configures a self-service namespace template with resource limits that exceed the cluster capacity, an alert is not triggered.

    When vSphere admistrators configure resource limits that exceed the cluster capacity, DevOps Engineers might use the template to deploy vSphere Pods with high resources. As a result, the workloads might fail.

    Workaround: None

  • After an upgrade of vCenter Server and vSphere with Tanzu, a Tanzu Kubernetes Grid cluster cannot complete its upgrade because of volumes that appear as attached to the cluster’s nodes

    When the Tanzu Kubernetes Grid cluster fails to upgrade, you can notice a volume that shows up as attached to the cluster’s nodes and does not get cleared. This problem might be caused by an issue in the upstream Kubernetes.


    1. Obtain information about the TKG cluster node that has scheduling disabled by using the following command:

      kubectl get node tkc_node_name -o yaml


      # kubectl get node gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95 -o yaml
      apiVersion: v1
      kind: Node
          cluster.x-k8s.io/cluster-name: gcm-cluster-antrea-1
          cluster.x-k8s.io/cluster-namespace: c1-gcm-ns
          cluster.x-k8s.io/machine: gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95
          cluster.x-k8s.io/owner-kind: MachineSet
          cluster.x-k8s.io/owner-name: gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7
          csi.volume.kubernetes.io/nodeid: '{"csi.vsphere.vmware.com":"gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95"}'
          kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
          node.alpha.kubernetes.io/ttl: "0"
          volumes.kubernetes.io/controller-managed-attach-detach: "true"
        - devicePath: ""
          name: kubernetes.io/csi/csi.vsphere.vmware.com^943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd

      Check if this node has any vSphere CSI volumes in the status.volumeAttached property. If there are any volumes, proceed to the next step.

    2. Verify that no pods are running on the node identified in Step 1. Use this command:

      kubectl get pod -o wide | grep tkc_node_name


      kubectl get pod -o wide | grep gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95

      The empty output of this command indicates that there are no pods. Proceed to the next step because the output in Step 1 shows a volume attached to the node that does not have any pods.

    3. Obtain information about all node objects to make sure that the same volume is attached to another node:

      kubectl get node -o yaml


      The same volume name shows up in two TKG cluster node objects.

      On old node - "gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95"
        - devicePath: ""
          name: kubernetes.io/csi/csi.vsphere.vmware.com^943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd
      On new node "gcm-cluster-antrea-1-workers-pvx5v-84486fc97-ndh88"
        - devicePath: ""
          name: kubernetes.io/csi/csi.vsphere.vmware.com^943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd
        - kubernetes.io/csi/csi.vsphere.vmware.com^943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd
    4. Search for the PV based on the volume handle in the volume name.

      In the above example volume name is kubernetes.io/csi/csi.vsphere.vmware.com^943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd and the volume handle is 943bd457-a6cb-4f3d-b1a5-e301e6fa6095-59c73cea-4b75-407d-8207-21a9a25f72fd.

      Get all the PVs and search for the above volume handle by using this command:

      kubectl get pv -o yaml

      In the above example, the PV with that volume handle is pvc-59c73cea-4b75-407d-8207-21a9a25f72fd.

    5. Use the volumeattachment command to search for the PV found in previous step:

      kubectl get volumeattachment | grep pv_name


      # kubectl get volumeattachment | grep pvc-59c73cea-4b75-407d-8207-21a9a25f72fd
      NAME                                                                   ATTACHER                 PV                                         NODE                                                 ATTACHED   AGE
      csi-2ae4c02588d9112d551713e31dba4aab4885c124663ae4fcbb68c632f0f46d3e   csi.vsphere.vmware.com   pvc-59c73cea-4b75-407d-8207-21a9a25f72fd   gcm-cluster-antrea-1-workers-pvx5v-84486fc97-ndh88   true       3d20h

      You can observe a volume attachment attached to node gcm-cluster-antrea-1-workers-pvx5v-84486fc97-ndh88 instead of node gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95. This indicates that the status.volumeAttached in gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95 in incorrect.

    6. Delete the stale volumeAttached entry from the node object:

      kubectl edit node tkc_node_name


      kubectl edit node gcm-cluster-antrea-1-workers-pvx5v-5b9c4dbcc7-28p95

      Remove the stale volume entry from status.volumeAttached.

    7. Repeat the above steps for all stale volumes in the status.volumeAttached property.

    8. If WCPMachines still exists, manually delete it and wait for few minutes to reconcile the cluster.

      # kubectl get wcpmachines -n c1-gcm-ns
      NAMESPACE   NAME                                       ZONE   PROVIDERID                                       IPADDR
      c1-gcm-ns   gcm-cluster-antrea-1-workers-jrc58-zn6wl          vsphere://423c2281-d1bd-9f91-0e87-b155a9d291a1
      # kubectl delete wcpmachine gcm-cluster-antrea-1-workers-jrc58-zn6wl -n c1-gcm-ns
      wcpmachine.infrastructure.cluster.vmware.com "gcm-cluster-antrea-1-workers-jrc58-zn6wl" deleted

  • If you change rules in a security policy custom resource, stale rules might not be deleted

    This problem might occur when you update the security policy. For example, you create a security policy custom resource that contains rules A and B and then update the policy changing the rules to B and C. As a result, rule A is not deleted. The NSX Management Plane continues to display rule A in addition to B and C.

    Workaround: Delete the security policy and then create the same one.

  • Upon one-click upgrade of vCenter Server, the Supervisor will not be auto-upgraded

    If the Kubernetes version of the Supervisor is of version earlier than 1.22, upon one-click upgrade of vCenter Server to 8.0, the Supervisor is not able auto-upgrade to the minimum supported version Kubernetes version for 8.0, which is 1.23.

    Workaround: Before upgrading vCenter Server for 8.0, upgrade the Kubernetes version of the Supervisor to 1.22. If you have already upgrade vCenter Server to 8.0, manually upgrade the Kubernetes version of the Supervisor to 1.23.

  • Upgrading ESXi hosts from 7.0 Update 3 to 8.0 without Supervisor upgrade results in ESXi hosts showing as Not Ready and workloads going offline

    When you upgrade the ESXi hosts that are part of a Supervisor from vSphere 7.0 Update 3 to vSphere 8.0 and you do not upgrade the Kubernetes version of the Supervisor, the ESXi hosts are showing as Not Ready and workloads running on the hosts go offline.

    Workaround: Upgrade the Kubernetes version of the Supervisor to at least 1.21, 1.22, or 1.23.

  • Tanzu 7 license keys are supported in vSphere 8 as opposed to Tanzu 8 license keys

    There is a defect in vCenter Server 8.0 that it supports the Tanzu 7 license keys as opposed to the Tanzu 8 license keys. This defect does not impact your ability to fully use Tanzu features on vCenter Server 8.0. For more details, see this KB Article before modifying Tanzu licenses in your vCenter 8.0 deployment.

    Workaround: None.

Tanzu Kubernetes Grid 2.0 on Supervisor Known Issues

  • Existing Tanzu Kubernetes Grid clusters configured with a proxy server cannot be upgraded to a vSphere 8 Supervisor

    If you have configured an existing Tanzu Kubernetes Grid cluster with a proxy server, you cannot upgrade that cluster from a vSphere 7 Supervisor to Tanzu Kubernetes Grid 2.0 on vSphere 8 Supervisor.

    Workaround: The contents of the noProxy field has conflicts with upgrade checks. Because this field is required if the proxy stanza is added to the cluster spec, you must remove the proxy configuration in its entirety before upgrading to vSphere 8.

  • The antrea-resource-init pod hangs in Pending state.

    After upgrading the Tanzu Kubernetes release version of a Tanzu Kubernetes Grid cluster, the antrea-resource-init pod might be in Pending state.

    Workaround: Restart the pod on the Supervisor.

  • Tanzu Kubernetes Grid clusters v1.21.6 might enter a into FALSE state and some machines might not migrate

    After upgrading to vCenter Server 8 and updating the Supervisor, v1.21.6 Tanzu Kubernetes Grid clusters might enter into a FALSE state and some wcpmachines might not migrate to vspheremachines.

    Workaround: None

  • Tanzu Kubernetes Grid 2.0 clusters on aSupervisor deployed on three vSphere Zones do not support VMs with vGPU and instance storage.

    Tanzu Kubernetes Grid 2.0 clusters provisioned on a Supervisor instance deployed across three vSphere Zones do not support VMs with vGPU and instance storage.

    Workaround: None

  • Tanzu Kubernetes Grid 2.0 clusters provisioned with the v1beta1 API must be based on the default ClusterClass

    If you are creating a Tanzu Kubernetes Grid 2.0 cluster on Supervisor by using the v1beta1 API, the Cluster must be based on the default "tanzukubernetescluster" ClusterClass. The system does not reconcile a Cluster based on a different ClusterClass.

    Workaround: None

  • Node drain timeout is not propogated correctly for v1beta1 Clusters

    The nodedraintimeout is not propagated correctly because the Tanzu Kubernetes release resolver is mutating a Cluster based on Cluster type without nodedraintimeout being present in the control plane machine deployment topology.


    Log in to Supervisor as a root user.

    To add node draintimeout for worker nodes, run the following command, then add the nodeDrainTimeout value to machineDeployment.spec.template.spec.nodeDrainTimeout.

    kubectl edit md <machine-deployment> -n <namespace>

    For control plane nodes, get all control plane machines under the Kubernetes control plane using the following command.

    kubectl get machine -n ns01 -l cluster.x-k8s.io/control-plane=

    Example result (formatted for clarity):


    tkc-minimal-xz9vm-6bnqf | tkc-minimal | tkc-minimal-xz9vm-6bnqf | vsphere://422cc83a-81d9-55c7-402b-3d3510c52967 | Running | 75m | v1.22.6+vmware.1

    Then edit machine.spec.nodeDrainTimeout and add the nodeDrainTimeout value. Also edit the Kubernetes control plane spec section, and add node drain timeout value to kcp.spec.machineTemplate.nodeDrainTimeout.

  • TKR version v1.22.9 is listed in content library but not in kubectl command

    The content library for TKR images lists TKR v1.22.9. The command "kubectl get tkr" does not list this image as available because TKR v1.22.9 is not available for use and should not be used. It is an error that this image appears in the content library.

    Use a TKR other than TKR v1.22.9. Refer to the TKR Release Notes for a list of the available TKRs.

check-circle-line exclamation-circle-line close-line
Scroll to top icon