Deploying Tanzu Kubernetes Grid Extensions and Shared Services

Tanzu Kubernetes Grid includes binaries for tools that provide in-cluster and shared services to the clusters running in your Tanzu Kubernetes Grid instance. All of the provided binaries and container images are built and signed by VMware.

You can add functionalities to Tanzu Kubernetes clusters by installing extensions to different cluster locations as follows:

Function	Extension	Location	Procedure
Ingress Control	Contour	Tanzu Kubernetes cluster	Implementing Ingress Control with Contour
Log Forwarding	Fluent Bit	Tanzu Kubernetes cluster	Implementing Log Forwarding with Fluentbit
Container Registry	Harbor	Shared Services cluster	Deploy Harbor Registry as a Shared Service
Monitoring	Prometheus	Tanzu Kubernetes cluster	Implementing Monitoring with Prometheus and Grafana
Monitoring	Grafana	Tanzu Kubernetes cluster	Implementing Monitoring with Prometheus and Grafana
User Authentication	Dex	Management cluster	Implementing User Authenticaton with Dex and Gangway
User Authentication	Gangway	Tanzu Kubernetes cluster	Implementing User Authenticaton with Dex and Gangway

Before you can deploy the Tanzu Kubernetes Grid extensions, you must prepare your bootstrap environment by performing the following tasks.

Download and Unpack the Tanzu Kubernetes Grid Extensions Bundle
Install the Carvel Tools on the Bootstrap Environment
If you are using Tanzu Kubernetes Grid in an Internet-restricted environment, see Deploying the Tanzu Kubernetes Grid Extensions in an Internet Restricted Environment.

Download and Unpack the Tanzu Kubernetes Grid Extensions Bundle

The Tanzu Kubernetes Grid extension manifests are provided in a separate bundle to the Tanzu Kubernetes Grid CLI and other binaries.

On the system that you use as the bootstrap machine, go to https://www.vmware.com/go/get-tkg and log in with your My VMware credentials.
Under Product Downloads, click Go to Downloads.
Scroll to VMware Tanzu Kubernetes Grid Extensions Manifest 1.2.0 and click Download Now.
Use either the tar command or the extraction tool of your choice to unpack the bundle of YAML manifest files for the Tanzu Kubernetes Grid extensions.
```
tar -xzf tkg-extensions-manifests-v1.2.0-vmware.1.tar.gz
```
For convenience, unpack the bundle in the same location as the one from which you run tkg and kubectl commands.

IMPORTANT: After you unpack the bundle, the extensions files are contained in a folder named tkg-extensions-v1.2.0+vmware.1. This folder contains subfolders for each type of extension, for example, authentication, ingress, registry, and so on. At the top level of the folder there is an additional subfolder named extensions. The extensions folder also contains subfolders for authentication, ingress, registry, and so on. In the procedures to deploy the extensions, take care to run commands from the location provided in the instructions. Commands are usually run from within the extensions folder.

Install the Carvel Tools on the Bootstrap Environment

The Tanzu Kubernetes Grid uses the following tools from the Carvel open-source project to configure and deploy all of the extensions:

ytt: YAML templating tool
kapp: Kubernetes applications CLI
kbld: Kubernetes builder

Tanzu Kubernetes Grid provides signed binaries for ytt, kapp, and kbld that are bundled with the Tanzu Kubernetes Grid CLI. For information about where to download the CLI bundle, see Install the Tanzu Kubernetes Grid CLI.

To deploy the extensions, you update configuration files with information about your environment. You then use kubectl to apply preconfigured YAML files that pull data from the updated configuration files to create and update clusters that implement the extensions. The YAML files include calls to ytt, kapp, and kbld commands, so these tools must be present on your bootstrap environment when you deploy the extensions.

Navigate to the location on your bootstrap environment machine where you saved the Tanzu Kubernetes Grid CLI bundle.

The bundles are tkg-darwin-amd64-VERSION for macOS, tkg-linux-amd64-VERSION for Linux, and tkg-windows-amd64-VERSION for Windows.
Rename the YTT binary for your platform to ytt and make it available to the system:
- For macOS and Linux platforms:
  1. Move the binary to /usr/local/bin:
```
mv ./ytt-linux-amd64-v0.30.0+vmware.1 /usr/local/bin/ytt
```
```
mv ./ytt-darwin-amd64-v0.30.0+vmware.1 /usr/local/bin/ytt
```
  2. Confirm that the binary is executable by running the ls command.
- For Windows platforms:
  1. Create a new Program Files\ytt folder and copy the ytt-windows-amd64-v0.30.0+vmware.1 binary into it.
  2. Rename ytt-windows-amd64-v0.30.0+vmware.1 to ytt.exe.
  3. Right-click the ytt folder, select Properties > Security, and make sure that your user account has the Full Control permission.
  4. Use Windows Search to search for env.
  5. Select Edit the system environment variables and click the Environment Variables button.
  6. Select the Path row under System variables, and click Edit.
  7. Click New to add a new row and enter the path to the ytt binary.
At the command line in a new terminal, run ytt version to check that the correct version of the binary is properly installed.

You should see information about the installed ytt version.
```
ytt version 0.30.0
```
Rename the Kapp binary for your platform to kapp and make it available to the system:
- For macOS and Linux platforms:
  1. Move the binary to /usr/local/bin:
```
mv ./kapp-linux-amd64-v0.33.0+vmware.1 /usr/local/bin/kapp
```
```
mv ./kapp-darwin-amd64-v0.33.0+vmware.1 /usr/local/bin/kapp
```
  2. Confirm that the binary is executable by running the ls command.
- For Windows platforms:
  1. Create a new Program Files\kapp folder and copy the kapp-windows-amd64-v0.33.0+vmware.1 binary into it.
  2. Rename kapp-windows-amd64-v0.33.0+vmware.1 to kapp.exe.
  3. Right-click the kapp folder, select Properties > Security, and make sure that your user account has the Full Control permission.
  4. Use Windows Search to search for env.
  5. Select Edit the system environment variables and click the Environment Variables button.
  6. Select the Path row under System variables, and click Edit.
  7. Click New to add a new row and enter the path to the kapp binary.
At the command line in a new terminal, run kapp version to check that the correct version of the binary is properly installed.

You should see information about the installed kapp version.
```
kapp version 0.33.0
```
Rename the Kbld binary for your platform to kbld and make it available to the system:
- For macOS and Linux platforms:
  1. Move the binary to /usr/local/bin:
```
mv ./kbld-linux-amd64-v0.24.0+vmware.1 /usr/local/bin/kbld
```
```
mv ./kbld-darwin-amd64-v0.24.0+vmware.1 /usr/local/bin/kbld
```
  2. Confirm that the binary is executable by running the ls command.
- For Windows platforms:
  1. Create a new Program Files\kbld folder and copy the kbld-windows-amd64-v0.24.0+vmware.1 binary into it.
  2. Rename kbld-windows-amd64-v0.24.0+vmware.1 to kbld.exe.
  3. Right-click the kbld folder, select Properties > Security, and make sure that your user account has the Full Control permission.
  4. Use Windows Search to search for env.
  5. Select Edit the system environment variables and click the Environment Variables button.
  6. Select the Path row under System variables, and click Edit.
  7. Click New to add a new row and enter the path to the kbld binary.
At the command line in a new terminal, run kbld version to check that the correct version of the binary is properly installed.

You should see information about the installed kbld version.
```
kbld version 0.24.0
```

Installing Extension Prerequisite Components to a Cluster

Procedures for installing Tanzu Kubernetes Grid extensions on a cluster, whether management or workload, include installing these components:

Tanzu Mission Control extension-manager
Kapp controller
For most extensions: cert-manager

To install these components on a cluster, specify the cluster with kubectl config use-context and then do the following:

Install the Tanzu Mission Control extension manager on the cluster.

The Tanzu Kubernetes Grid extensions and Tanzu Mission Control both use the same extension-manager service. You must install the extension manager even if you do not intend to use Tanzu Mission Control.

kubectl apply -f tmc-extension-manager.yaml

You should see confirmation that a namespace, resource definitions, a service account, an RBAC role, and a role binding for the extension-manager service are all created.

namespace/vmware-system-tmc created
customresourcedefinition.apiextensions.k8s.io/agents.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensions.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionresourceowners.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionintegrations.clusters.tmc.cloud.vmware.com created
customresourcedefinition.apiextensions.k8s.io/extensionconfigs.intents.tmc.cloud.vmware.com created
serviceaccount/extension-manager created
clusterrole.rbac.authorization.k8s.io/extension-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/extension-manager-rolebinding created
service/extension-manager-service created
deployment.apps/extension-manager created

Note: You can disregard any warning messages about deprecated APIs.

Install the Kapp controller on the cluster.

kubectl apply -f kapp-controller.yaml

You should see confirmation that a service account, resource definition, and RBAC role are created for the kapp-controller service.

serviceaccount/kapp-controller-sa created
customresourcedefinition.apiextensions.k8s.io/apps.kappctrl.k14s.io created
deployment.apps/kapp-controller created
clusterrole.rbac.authorization.k8s.io/kapp-controller-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/kapp-controller-cluster-role-binding created

Determine whether the extension image you are installing is stored in a private container registry that uses a self-signed certificate:
1. Look in the extension manifest file in the extensions directory. The file has a name of the form NAME-extension.yaml, for example harbor-extension.yaml or contour-extension.yaml.
2. In the manifest, check whether the spec.objects.spec.fetch.image.url field points to your own private registry or a public registry. Private registries typically use self-signed certificates.

If the extension manifest pulls the image from a registry with a self-signed certificate, configure the Kapp controller to trust the registry as follows:

Create a file custom-ca-certs.crt containing the CA certificate of your private container registry.

Create a ConfigMap with the certificate by running:

kubectl create configmap custom-ca-certs --from-file=custom-ca-certs.crt -n vmware-system-tmc

Back up your kapp-controller.yaml and replace the kind: Deployment block with the code below:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kapp-controller
  namespace: vmware-system-tmc
spec:
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: kapp-controller
  template:
    metadata:
      labels:
        app: kapp-controller
    spec:
      containers:
      - args: []
        command:
        - /kapp-controller
        env:
        - name: KAPPCTRL_MEM_TMP_DIR
          value: /etc/kappctrl-mem-tmp
        image: registry.tkg.vmware.run/kapp-controller:v0.9.0_vmware.1
        imagePullPolicy: IfNotPresent
        name: kapp-controller
        resources:
          requests:
            cpu: 120m
            memory: 100Mi
        securityContext:
          runAsGroup: 2000
          runAsUser: 1000
        volumeMounts:
        - mountPath: /etc/kappctrl-mem-tmp
          name: template-fs
        - mountPath: /etc/ssl/certs/custom-ca-certs.crt
          name: custom-ca-certs
          readOnly: true
          subPath: custom-ca-certs.crt
      securityContext:
        fsGroup: 3000
      serviceAccount: kapp-controller-sa
      volumes:
      - emptyDir:
          medium: Memory
        name: template-fs
      - configMap: 
          name: custom-ca-certs
        name: custom-ca-certs

Apply the Kapp controller definition to mount the CA cert file from the ConfigMap:
```
kubectl apply -f kapp-controller.yaml
```

For extensions other than Dex or Fluent Bit, deploy cert-manager, which provides automated certificate management, on the cluster.
```
kubectl apply -f cert-manager/
```
Dex and Fluent Bit do not use cert-manager.

Upgrading the Tanzu Kubernetes Grid Extensions

For information about how to upgrade the Tanzu Kubernetes Grid extensions from a previous release, see Upgrade Tanzu Kubernetes Grid Extensions from 1.1.x to 1.2.x.