VMware Tanzu simplifies the operation of Kubernetes for multi-cloud deployment by centralizing management and governance for many clusters and teams across on-premises, public clouds, and edge. It delivers an open-source aligned Kubernetes distribution with consistent operations and management to support infrastructure and app modernization. This document provides a high-level architecture for deploying Tanzu editions. It also provides links for infrastructure provider specific reference architectures.

Tanzu Edition reference design diagram


VMware Tanzu Kubernetes Grid - Enables creation and lifecycle management operations of Kubernetes clusters.

Contour Ingress Controller - Provides layer 7 ingress control to deployed HTTP(S) applications

NSX Advanced Load Balancer Essentials - Provides layer 4 service type Load Balancer support, recommended for vSphere deployments without NSX-T, or which have unique scale requirements.

Harbor Image Registry - Provides a centralized location to push, pull, store, and scan container images used in Kubernetes workloads. It also supports storing many other artifacts such as helm charts and includes enterprise grade features such as RBAC, retention policies, automated garbage clean up, and docker hub proxying among many other things.

VMware Tanzu Mission Control - Provides a global view of Kubernetes clusters, and allows for centralized policy management across all deployed and attached clusters.

Pinniped - Pinniped is an authentication service for Kubernetes clusters. This tool provides integration with identity providers such as OKTA, Dex and any LDAP identity provider.

VMware Tanzu Observability Extensions

  • Fluent bit - provides export log streaming of cluster & workload logs to a wide range of supported aggregators provided in the extensions package for Tanzu Kubernetes Grid
  • Prometheus - provides out-of-the box health monitoring of Kubernetes clusters
  • Grafana - provides monitoring dashboards for displaying key health metrics of Kubernetes clusters

Tanzu Kubernetes Grid

Tanzu Kubernetes Grid allows you to create and manage ubiquitous Kubernetes clusters across multiple infrastructure providers using Kubernetes Cluster API. Tanzu Kubernetes Grid functions through the creation of a Management or Supervisor Kubernetes cluster which houses Cluster API. The Cluster API then interacts with the infrastructure provider to service workload Kubernetes cluster lifecycle requests.

Tanzu Kubernetes Grid Kickstart Install Screen

Tanzu Editions includes components for observability, as well as container registry. It is generally recommended to install the necessary components into a centralized Shared Services cluster.

When targeting vSphere IaaS, additional convenient features are introduced in the integrated vSphere with Tanzu product; see VMware Tanzu Kubernetes Grid on vSphere Reference Design, which highlights and disambiguates these features.

Network Overview


The following design encompasses the generic network architecture for the Tanzu Kubernetes Grid reference design. For some infrastructure providers, you will find that the networks can be the same subnet or segments, and in other architectures might be entirely separate domains, but each infrastructure provider's networks can be mapped into this general framework.

Network topology diagram

Air-gapped Installation

Installing Tanzu Kubernetes Grid in an environment without internet access (air-gapped) requires staging the container images necessary for Tanzu into your environment. If allowed, an internet connected workstation with access to both networks can be used to stage the files from the Tanzu Network registry into a container registry such as Harbor.

In public infrastructure providers, internetless environments also require the creation of private endpoints for their respective services.

The reference architecture for each infrastructure provider covers the necessary steps to implement an air gapped installation into that specific environment. The Tanzu Kubernetes Grid documentation also provides instructions for copying container images into an environment.

  • Harbor Container Registry provides an offline installer that can be deployed on a standalone VM via docker-compose instead of leveraging a Tanzu Kubernetes Grid shared-services cluster. This is ideal as the registry must be available prior to the bootstrap of management and subsequent workload clusters.

Authentication with Pinniped

Pinniped consists of two components: a Supervisor and Concierge.

  • The Pinniped Supervisor is an OIDC server which authenticates users through an external identity provider (IDP)/LDAP, and then issues its own federation ID tokens to be passed on to clusters based on the user information from the IDP.
  • The Pinniped Concierge is a credential exchange API which takes as input a credential from an identity source (e.g., Pinniped Supervisor, proprietary IDP), authenticates the user via that credential, and returns another credential which is understood by the host Kubernetes cluster or by an impersonation proxy which acts on behalf of the user.

Examples of Identity Providers are:

  • Okta Identity provider (OIDC) - Okta is an enterprise grade identity management service, built from the ground up in the cloud. With Okta, IT can manage access across any application, person or device and perform Single Sign On (SSO) using an Okta identity provider.
  • Active Directory - A Microsoft identity management system.

Authentication with pinniped

See the following for more information on how to integrate an OIDC provider like OKTA and Pinniped into Tanzu Kubernetes Grid Tanzu Kubernetes Grid OIDC/LDAP Integration with Pinniped and OKTA OIDC Provider.

Cluster Backup Operations with Velero

Velero is an open-source project from VMware enabling robust backup operations for Kubernetes clusters. Instead of infrastructure-centric backup strategies such as manual etcd snapshots, Velero targets Kubernetes objects and volumes via a set of controllers and custom resource definitions. This allows targeting Kubernetes-centric items such as namespaces, deployments and other cluster workloads, which can then be easily restored or even imported to different clusters.

In addition to Kubernetes objects, volumes can also be targeted for backup via cloud-provider specific plugins such as the velero-vsphere plugin. This allows velero to perform volume snapshots, storing the contents in object storage for future restoration.

The following guide describes the installation and backup workflow of a cluster workload via the vsphere-velero plugin.

For information about doing cluster backup with Velero, see Tanzu Kubernetes Grid Backup Operations with Velero vSphere Plugin.

Velero diagram

Tanzu Mission Control

Attaching clusters into global management through TMC allows you to manage your global portfolio of Kubernetes clusters. TMC can assist you with:

  • Centralized lifecycle management -- managing the creation and deletion of workload clusters using registered management or supervisor clusters
  • Centralized management -- viewing the inventory of clusters and the health of clusters and their components
  • Authorization -- Centralized authentication and authorization, with federated identity from multiple sources (e.g., AD, LDAP, and SAML), plus an easy-to-use policy engine for granting the right access to the right users across teams.
  • Compliance -- enforcing all clusters to have the same set of policies applied
  • Data protection -- managing Velero deployment, configuration & schedule to ensure cluster manifests & persistent volumes are backed up & restorable
  • Inspection -- running a Sonobouy conformance check suite to ensure kubernetes cluster functionality

VMware Tanzu Mission Control - global policy control plane diagram

If the workload cluster put under management requires a proxy to access the internet, you can use the TMC CLI to generate the YAML necessary to install TMC components on it.

For a complete list of features that TMC includes with Tanzu see this chart.


Metrics Monitoring with Tanzu Observability by Wavefront (recommended solution)

Observability can be significantly enhanced by using Tanzu Observability by Wavefront. Wavefront is a VMware SaaS system which is used to collect and display metrics and trace data from the full stack platform as well as from applications. The service provides the ability to create alerts tuned by advanced analytics, assist in the troubleshooting of systems and to understand the impact of running production code.

In the case of vSphere and Tanzu Kubernetes Grid, Wavefront is used to collect data from components in vSphere, from Kubernetes, and from applications running within Kubernetes.

You can configure Wavefront with an array of capabilities. Here are the recommended plugins for this design:

Plugin Purpose Key Metrics Example Metrics
Wavefront Kubernetes Integration Collect metrics from Kubernetes clusters and pods Kubernetes container and POD statistics POD CPU usage rate
Wavefront by VMware for Istio Adapts Istio collected metrics and forwards to Wavefront Istio metrics including request rates, trace rates, throughput, etc. Request rate (Transactions per Second)

kubernetes-metrics-1 kubernetes-metrics-2

Metrics Monitoring with Prometheus and Grafana (alternative solution)

Tanzu Kubernetes Grid also supports Prometheus and Grafana as an alternative on premise solution that can be used for monitoring kubernetes clusters.

Prometheus operates by exposing scrapable metrics endpoints for various monitoring targets throughout your cluster. Metrics are ingested by polling the endpoints on a set interval which are then stored in a time-series database. Metrics data can be explored via the Prometheus Query Language interface.

Grafana is responsible for vizualizing Prometheus metrics without the need to manually write PromQL queries. Custom charts and graphs can be created in addition to the pre-packaged optons.

The Tanzu Kubernetes Grid extensions bundles contain instructions & manifests for deploying these tools out.

Tanzu Observability CPU utilization dashboard

Tanzu Observability availability dashboard

Log Forwarding

Tanzu also includes fluent bit for integration with logging platforms such as vRealize LogInsight and other logging aggregators. Details on configuring fluent bit to your logging provider can be found in the documentation here.

Tanzu Kubernetes Grid on vSphere

Follow the reference architecture for Tanzu Kubernetes Grid on vSphere here.

check-circle-line exclamation-circle-line close-line
Scroll to top icon