Install Grafana in Workload Clusters Deployed by a Standalone Management Cluster

This topic explains how to deploy Grafana into a workload cluster. The procedures below apply to vSphere, Amazon Web Services (AWS), and Azure deployments.

Grafana

Grafana is open-source software that allows you to visualize and analyze metrics data collected by Prometheus on your clusters. Tanzu Kubernetes Grid includes a Grafana package that you can deploy on your workload clusters.

Prerequisites

A bootstrap machine with the following installed:
- Tanzu CLI, Tanzu CLI plugins, and kubectl, as described in Install the Tanzu CLI and Kubernetes CLI for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
- yq v4.5 or later.
A management cluster, either a vSphere with Tanzu Supervisor or a standalone management cluster running on vSphere, AWS, or Azure, with the Tanzu Standard package repository installed.
A workload cluster created by the management cluster above, with the following packages installed:
- Prometheus; see Deploy Prometheus on Workload Clusters.
- Contour; see Install Contour for Ingress Control.
- cert-manager; see Install Cert Manager for Certificate Management.

Note
As of v2.5, TKG does not support clusters on AWS or Azure. See the End of Support for TKG Management and Workload Clusters on AWS and Azure in the Tanzu Kubernetes Grid v2.5 Release Notes.

Important Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Grafana Deployment

To prepare the cluster:

Get the admin credentials of the workload cluster into which you want to deploy Grafana. For example:
```
tanzu cluster kubeconfig get my-cluster --admin
```

Set the context of kubectl to the cluster. For example:

kubectl config use-context my-cluster-admin@my-cluster

Enable Ingress for Grafana: By default, Grafana has Ingress enabled. This requires you to install the following packages:
- cert-manager: For information, see Install Cert Manager for Certificate Management.
- Contour: For information, see Install Contour for Ingress Control.

Continue to Deploy Grafana on a Workload Cluster.

Deploy Grafana into the Workload Cluster

To deploy Grafana:

If the cluster does not have a package repository with the Grafana package installed, such as the tanzu-standard repository, install one:
```
tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
```
Where:
- PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
- PACKAGE-REPO-ENDPOINT is the URL of the package repository.
  - For TKG v2.5.1 release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2024.4.12. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
Confirm that the Grafana package is available in your workload cluster:
```
tanzu package available list -A
```

Retrieve the version of the available package:

tanzu package available list grafana.tanzu.vmware.com -A
| Retrieving package versions for grafana.tanzu.vmware.com...
 NAME                           VERSION                          RELEASED-AT           NAMESPACE
 grafana.tanzu.vmware.com       10.0.1+vmware.1-tkg.1             2021-05-19T18:00:00Z  tanzu-package-repo-global

Create a configuration file for your Grafana configuration:
```
tanzu package available get grafana.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
Where PACKAGE-VERSION is the version of the Grafana package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, grafana-data-values.yaml.

See Grafana Configuration Parameters (Standalone MC) for a full list of available parameters.
Edit grafana-data-values.yaml and replace secret.admin_password with a Base64-encoded password. To generate a Base64-encoded password, run:
```
echo -n 'mypassword' | base64
```
You can also use the Base64 encoding tool at https://www.base64encode.org/ to encode your password. For example, using either method, mypassword results in the encoded password bXlwYXNzd29yZA==.
(Optional) Modify the Grafana datasource configuration in grafana-data-values.yaml. Grafana is configured with Prometheus as a default data source. If you have customized the Prometheus deployment namespace and it is not deployed in the default namespace, tanzu-system-monitoring, you need to change the Grafana datasource configuration in grafana-data-values.yaml. To change the datasource configuration, copy the section below into the position shown and modify url as needed.
```
#! The namespace in which to deploy grafana.
namespace: tanzu-system-dashboards

grafana:
  #! The grafana configuration.
  config:
    #! Refer to https://grafana.com/docs/grafana/latest/administration/provisioning/#example-data-source-config-file
    datasource_yaml: |-
      apiVersion: 1
      datasources:
        - name: Prometheus
          type: prometheus
          url: http://prometheus-server.<change-to-prometheus-namespace>.svc.cluster.local
          access: proxy
          isDefault: true
```
See Grafana Configuration Parameters (Standalone MC) for a full list of available parameters.
After you make any changes needed to your grafana-data-values.yaml file, remove all comments in it:
```
yq -i eval '... comments=""' grafana-data-values.yaml
```
Deploy the package:
```
tanzu package install grafana \
--package grafana.tanzu.vmware.com \
--version AVAILABLE-PACKAGE-VERSION \
--values-file grafana-data-values.yaml \
--namespace TARGET-NAMESPACE
```
Where:
- TARGET-NAMESPACE is the namespace in which you want to install the Grafana package. For example, the my-packages or tanzu-user-managed-packages namespace.
  - If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Grafana pods and any other resources associated with the Grafana component are created in the namespace you set in grafana-data-values.yaml; do not install the Grafana package into this namespace.
  - The specified namespace must already exist, for example from running kubectl create namespace my-packages.
- AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.
For example:
```
tanzu package install grafana --package grafana.tanzu.vmware.com --version 10.0.1+vmware.1-tkg.1 --values-file grafana-data-values.yaml --namespace my-packages
```
```
- Installing package 'grafana.tanzu.vmware.com'
| Getting namespace 'my-packages'
| Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-my-packages-sa'
| Creating cluster admin role 'grafana-my-packages-cluster-role'
| Creating cluster role binding 'grafana-my-packages-cluster-rolebinding'
| Creating secret 'grafana-my-packages-values'
- Creating package resource
\ Package install status: Reconciling

 Added installed package 'grafana' in namespace 'my-packages'
```
Note
If the installation fails with error: Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 4 (reason: BadRequest), see Secret not created when installing Grafana from default YAML file in Troubleshooting Workload Cluster Issues.

Verify Grafana Deployment

After you deploy Grafana, you can verify that the deployment is successful:

Confirm that the Grafana package is installed. For example:

tanzu package installed list -A
/ Retrieving installed packages...
NAME            PACKAGE-NAME                     PACKAGE-VERSION          STATUS               NAMESPACE
cert-manager    cert-manager.tanzu.vmware.com    1.12.2+vmware.1-tkg.1     Reconcile succeeded  my-packages
contour         contour.tanzu.vmware.com         1.25.2+vmware.1-tkg.1    Reconcile succeeded  my-packages
grafana         grafana.tanzu.vmware.com         10.0.1+vmware.1-tkg.1     Reconcile succeeded  tkg-system
prometheus      prometheus.tanzu.vmware.com      2.45.0+vmware.1-tkg.1    Reconcile succeeded  tkg-system
antrea          antrea.tanzu.vmware.com                                   Reconcile succeeded  tkg-system
[...]

The grafana package and the grafana app are installed in the namespace that you specify when running the tanzu package install command.

Confirm that the grafana app is successfully reconciled:

kubectl get apps -A

For example:

NAMESPACE     NAME                           DESCRIPTION           SINCE-DEPLOY   AGE
my-packages   cert-manager                   Reconcile succeeded   37s            21h
my-packages   contour                        Reconcile succeeded   33s            129m
my-packages   grafana                        Reconcile succeeded   19s            6m56s
my-packages   prometheus                     Reconcile succeeded   46s            21h
tkg-system    antrea                         Reconcile succeeded   3m50s          24h
[...]

If the status is not Reconcile succeeded, view the full status details of the grafana app. Viewing the full status can help you troubleshoot the problem:

kubectl get app grafana --namespace PACKAGE-NAMESPACE -o yaml

Where PACKAGE-NAMESPACE is the namespace in which you installed the package.

Confirm that the new services are running by listing all of the pods that are running in the cluster:

kubectl get pods -A

In the tanzu-system-dashboards namespace, you should see the grafana service running in a pod:

NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
[...]
tanzu-system-dashboards   grafana-6865dbb4f5-pk2qg              2/2     Running   0          7m7s
[...]

The Grafana pods and any other resources associated with the Grafana component are created in the namespace you provided in grafana-data-values.yaml. If you are using the default namespace, these are created in the tanzu-system-dashboards namespace.

Update a Running Grafana Deployment

To make changes to the configuration of the Grafana package after deployment, update your deployed Grafana package:

Update the Grafana configuration in the grafana-data-values.yaml file.

Update the installed package:

tanzu package installed update grafana \
--version 10.0.1+vmware.1-tkg.1 \
--values-file grafana-data-values.yaml \
--namespace my-packages

Expected output:

| Updating package 'grafana'
- Getting package install for 'grafana'
| Updating secret 'grafana-my-packages-values'
| Updating package install for 'grafana'

 Updated package install 'grafana' in namespace 'my-packages'

The Grafana package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Grafana Deployment

To remove the Grafana package on your cluster, run:

tanzu package installed delete grafana --namespace my-packages

For information about deleting, see Delete a Package.

Access the Grafana Dashboard

After Grafana is deployed, the grafana package creates a Contour HTTPProxy object with a Fully Qualified Domain Name (FQDN) of grafana.system.tanzu.

To use this FQDN to access the Grafana dashboard:

Create an entry in your local /etc/hosts file that points an IP address to this FQDN:
- AWS or Azure: Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.
- vSphere: Use the IP address of a worker node.
Navigate to https://grafana.system.tanzu.

Because the site uses self-signed certificates, you might need to navigate through a browser-specific security warning before you are able to access the dashboard.