Tanzu Kubernetes Grid includes binaries for tools that provide in-cluster and shared services to the clusters running in your Tanzu Kubernetes Grid instance. All of the provided binaries and container images are built and signed by VMware.
You can add functionalities to Tanzu Kubernetes clusters by installing extensions to different cluster locations as follows:
|Ingress Control||Contour||Tanzu Kubernetes or Shared Service cluster||Implementing Ingress Control with Contour|
|Service Discovery||External DNS||Tanzu Kubernetes or Shared Service cluster||Implementing Service Discovery with External DNS|
|Log Forwarding||Fluent Bit||Tanzu Kubernetes cluster||Implementing Log Forwarding with Fluentbit|
|Container Registry||Harbor||Shared Services cluster||Deploy Harbor Registry as a Shared Service|
|Monitoring||Prometheus||Tanzu Kubernetes cluster||Implementing Monitoring with Prometheus and Grafana|
|Grafana||Tanzu Kubernetes cluster|
Some extensions require or are enhanced by other extensions deployed to the same cluster:
Before you can deploy the Tanzu Kubernetes Grid extensions, you must prepare your bootstrap environment.
kubectlto apply preconfigured YAML files that pull data from the updated configuration files to create and update clusters that implement the extensions. The YAML files include calls to
kbldcommands, so these tools must be present on your bootstrap environment when you deploy the extensions. For information about installing
kbld, see Install the Carvel Tools.
The Tanzu Kubernetes Grid extension manifests are provided in a separate bundle to the Tanzu CLI and other binaries.
Use either the
tar command or the extraction tool of your choice to unpack the bundle of YAML manifest files for the Tanzu Kubernetes Grid extensions.
tar -xzf tkg-extensions-manifests-v1.3.1-vmware.1.tar.gz
For convenience, unpack the bundle in the same location as the one from which you run
tkg-extensions-v1.3.1+vmware.1. This folder contains subfolders for each type of extension, for example,
registry, and so on. At the top level of the folder there is an additional subfolder named
extensionsfolder also contains subfolders for
registry, and so on. In the procedures to deploy the extensions, take care to run commands from the location provided in the instructions. Commands are usually run from within the
Before you can deploy Tanzu Kubernetes Grid extensions, you must install
cert-manager, which provides automated certificate management, on workload clusters. The
cert-manager service already runs by default in management clusters.
All extensions other than Fluent Bit require
cert-manager to be running on workload clusters. Fluent Bit does not use
To install the
cert-manager service on a workload cluster, specify the cluster with
kubectl config use-context and then do the following:
cert-manager on the cluster.
kubectl apply -f cert-manager/
Check that the Kapp controller and cert-manager services are running as pods in the cluster.
kubectl get pods -A
The command output should show:
kapp-controllerpod with a name like
kapp-controller-cd55bbd6b-vt2c4running in the namespace
cert-manager-webhook-5fc8c6dc54-nlvzprunning in the namespace
1/1for all of these pods. If this status is not displayed, stop and troubleshoot the pods before proceeding.
The Harbor service runs on a shared services cluster, to serve all the other clusters in an installation. The Harbor service requires the Contour service to also run on the shared services cluster. In many environments, the Harbor service also benefits from External DNS running on its cluster, as described in Harbor Registry and External DNS.
Each Tanzu Kubernetes Grid instance can only have one shared services cluster.
To deploy a shared services cluster:
Create a cluster configuration YAML file for the target cluster. To deploy to a shared services cluster, for example named
tkg-services, it is recommended to use the
prod cluster plan rather than the
dev plan. For example:
INFRASTRUCTURE_PROVIDER: vsphere CLUSTER_NAME: tkg-services CLUSTER_PLAN: prod
vSphere: To deploy the cluster to vSphere, add a line to the configuration file that sets
VSPHERE_CONTROL_PLANE_ENDPOINT to a static virtual IP (VIP) address for the control plane of the shared services cluster. Ensure that this IP address is not in the DHCP range, but is in the same subnet as the DHCP range. If you mapped a fully qualified domain name (FQDN) to the VIP address, you can specify the FQDN instead of the VIP address. For example:
Deploy the cluster by passing the cluster configuration file to the
tanzu cluster create:
tanzu cluster create tkg-services --file tkg-services-config.yaml
Throughout the rest of these procedures, the cluster that you just deployed is referred to as the shared services cluster.
Set the context of
kubectl to the context of your management cluster. For example, if your cluster is named
mgmt-cluster, run the following command.
kubectl config use-context mgmt-cluster-admin@mgmt-cluster
Add the label
tanzu-services to the shared services cluster, as its cluster role. This label identifies the shared services cluster to the management cluster and workload clusters.
kubectl label cluster.cluster.x-k8s.io/tkg-services cluster-role.tkg.tanzu.vmware.com/tanzu-services="" --overwrite=true
You should see the confirmation
Check that the label has been correctly applied by running the following command.
tanzu cluster list --include-management-cluster
You should see that the
tkg-services cluster has the
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN another-cluster default running 1/1 1/1 v1.20.5+vmware.1 <none> dev tkg-services default running 3/3 3/3 v1.20.5+vmware.1 tanzu-services prod mgmt-cluster tkg-system running 1/1 1/1 v1.20.5+vmware.1 management dev
In a terminal, navigate to the folder that contains the unpacked Tanzu Kubernetes Grid extension manifest files,
You should see folders for
registry, and some YAML files. Run all of the commands in this procedure from this location.
admin credentials of the shared services cluster on which to deploy Harbor.
tanzu cluster kubeconfig get tkg-services --admin
Set the context of
kubectl to the shared services cluster.
kubectl config use-context tkg-services-admin@tkg-services
Previous versions of Tanzu Kubernetes Grid required the user to install the
kapp-controller service to any extension cluster. As of v1.3, all management and workload clusters are created with the
kapp-controller service pre-installed. If the cluster configuration file specifies a private registry with
TKG_CUSTOM_IMAGE_REPOSITORY_CA_CERTIFICATE variables, the
kapp-controller is configured to trust the private registry.
To enable a cluster's Kapp Controller to trust additional private registries, add their certificates to its configuration:
If needed, set the current
kubectl context to the cluster with the Kapp Controller you are changing:
kubectl config use-context CLUSTER-CONTEXT
Open the Kapp Controller's ConfigMap file in an editor:
kubectl edit configmap -n tkg-system kapp-controller-config
Edit the ConfigMap file to add new certificates to the
apiVersion: v1 kind: ConfigMap metadata: # Name must be `kapp-controller-config` for kapp controller to pick it up name: kapp-controller-config # Namespace must match the namespace kapp-controller is deployed to namespace: tkg-system data: # A cert chain of trusted ca certs. These will be added to the system-wide # cert pool of trusted ca's (optional) caCerts: | -----BEGIN CERTIFICATE----- <Existing Certificate> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <New Certificate> -----END CERTIFICATE----- # The url/ip of a proxy for kapp controller to use when making network requests (optional) httpProxy: "" # The url/ip of a tls capable proxy for kapp controller to use when making network requests (optional) httpsProxy: "" # A comma delimited list of domain names which kapp controller should bypass the proxy for when making requests (optional) noProxy: ""
Save the ConfigMap and exit the editor.
kapp-controller pod, so that it regenerates with the new configuration:
kubectl delete pod -n tkg-system -l app=kapp-controller
For information about how to upgrade the Tanzu Kubernetes Grid extensions from a previous release, see Upgrade Tanzu Kubernetes Grid Extensions.