Deploy Grafana on Workload Clusters

This topic explains how to deploy Grafana into a workload cluster. The procedures below apply to vSphere, Amazon Web Services (AWS), and Azure deployments.

Grafana

Grafana is open-source software that allows you to visualize and analyze metrics data collected by Prometheus on your clusters. Tanzu Kubernetes Grid includes a Grafana package that you can deploy on your workload clusters.

Prerequisites

• A bootstrap machine with the following installed:
  • Tanzu CLI, Tanzu CLI plugins, and kubectl, as described in Install the Tanzu CLI and Kubernetes CLI for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
  • yq v4.5 or later.
• A management cluster, either a vSphere with Tanzu Supervisor or a standalone management cluster running on vSphere, AWS, or Azure, with the Tanzu Standard package repository installed.
• A workload cluster created by the management cluster above, with the following packages installed:
  • Prometheus; see Deploy Prometheus on Workload Clusters.
  • Contour; see Install Contour for Ingress Control.
  • cert-manager; see Install cert-manager for Certificate Management.

Important

Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Grafana Deployment

To prepare the cluster:

1. Get the admin credentials of the workload cluster into which you want to deploy Grafana. For example:

   tanzu cluster kubeconfig get my-cluster --admin
   

2. Set the context of kubectl to the cluster. For example:

   kubectl config use-context my-cluster-admin@my-cluster
   

3. Enable Ingress for Grafana: By default, Grafana has Ingress enabled. This requires you to install the following packages:

   • cert-manager: For information, see Install cert-manager for Certificate Management.
   • Contour: For information, see Install Contour for Ingress Control.

Continue to Deploy Grafana on a Workload Cluster.

Deploy Grafana into the Workload Cluster

To deploy Grafana:

1. If the cluster does not have a package repository with the Grafana package installed, such as the tanzu-standard repository, install one:

   tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
   

   Where:

   • PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.

   • PACKAGE-REPO-ENDPOINT is the URL of the package repository.

     • For this release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2023.10.16. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.

2. Confirm that the Grafana package is available in your workload cluster:

   tanzu package available list -A
   

3. Retrieve the version of the available package:

   tanzu package available list grafana.tanzu.vmware.com -A
   | Retrieving package versions for grafana.tanzu.vmware.com...
    NAME                           VERSION                          RELEASED-AT           NAMESPACE
    grafana.tanzu.vmware.com       9.5.1+vmware.2-tkg.3             2021-05-19T18:00:00Z  tanzu-package-repo-global
   

4. Create a configuration file for your Grafana configuration:

   tanzu package available get grafana.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
   
   

   Where PACKAGE-VERSION is the version of the Grafana package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, grafana-data-values.yaml.

   For information about configuration parameters to use in grafana-data-values.yaml, see Grafana Package Configuration Parameters below.

5. Edit grafana-data-values.yaml and replace secret.admin_password with a Base64-encoded password. To generate a Base64-encoded password, run:

   echo -n 'mypassword' | base64
   

   You can also use the Base64 encoding tool at https://www.base64encode.org/ to encode your password. For example, using either method, mypassword results in the encoded password bXlwYXNzd29yZA==.

6. vSphere with Tanzu: If you are deploying Grafana to a workload cluster created by a vSphere with Tanzu Supervisor, set a non-null value for grafana.pvc.storageClassName in the grafana-data-values.yaml file:

   grafana:
   pvc:
     storageClassName: STORAGE-CLASS
   

   Where STORAGE-CLASS is the name of the cluster’s storage class, as returned by kubectl get storageclass.

7. (Optional) Modify the Grafana datasource configuration in grafana-data-values.yaml. Grafana is configured with Prometheus as a default data source. If you have customized the Prometheus deployment namespace and it is not deployed in the default namespace, tanzu-system-monitoring, you need to change the Grafana datasource configuration in grafana-data-values.yaml. To change the datasource configuration, copy the section below into the position shown and modify url as needed.

   #! The namespace in which to deploy grafana.
   namespace: tanzu-system-dashboards
   
   grafana:
     #! The grafana configuration.
     config:
       #! Refer to https://grafana.com/docs/grafana/latest/administration/provisioning/#example-data-source-config-file
       datasource_yaml: |-
         apiVersion: 1
         datasources:
           - name: Prometheus
             type: prometheus
             url: http://prometheus-server.<change-to-prometheus-namespace>.svc.cluster.local
             access: proxy
             isDefault: true
   

   For information about configuration parameters to use in grafana-data-values.yaml, see Grafana Package Configuration Parameters below.

8. After you make any changes needed to your grafana-data-values.yaml file, remove all comments in it:

   yq -i eval '... comments=""' grafana-data-values.yaml
   

9. Deploy the package:

   tanzu package install grafana \
   --package grafana.tanzu.vmware.com \
   --version AVAILABLE-PACKAGE-VERSION \
   --values-file grafana-data-values.yaml \
   --namespace TARGET-NAMESPACE
   

   Where:

   • TARGET-NAMESPACE is the namespace in which you want to install the Grafana package. For example, the my-packages or tanzu-user-managed-packages namespace.

     • If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Grafana pods and any other resources associated with the Grafana component are created in the namespace you set in grafana-data-values.yaml; do not install the Grafana package into this namespace.
     • The specified namespace must already exist, for example from running kubectl create namespace my-packages.

   • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

   For example:

   tanzu package install grafana --package grafana.tanzu.vmware.com --version 9.5.1+vmware.2-tkg.3 --values-file grafana-data-values.yaml --namespace my-packages
   

   - Installing package 'grafana.tanzu.vmware.com'
   | Getting namespace 'my-packages'
   | Getting package metadata for 'grafana.tanzu.vmware.com'
   | Creating service account 'grafana-my-packages-sa'
   | Creating cluster admin role 'grafana-my-packages-cluster-role'
   | Creating cluster role binding 'grafana-my-packages-cluster-rolebinding'
   | Creating secret 'grafana-my-packages-values'
   - Creating package resource
   \ Package install status: Reconciling
   
    Added installed package 'grafana' in namespace 'my-packages'
   

   Note

   If the installation fails with error: Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 4 (reason: BadRequest), see Secret not created when installing Grafana from default YAML file in Troubleshooting Workload Cluster Issues.

Verify Grafana Deployment

After you deploy Grafana, you can verify that the deployment is successful:

1. Confirm that the Grafana package is installed. For example:

   tanzu package installed list -A
   / Retrieving installed packages...
   NAME            PACKAGE-NAME                     PACKAGE-VERSION          STATUS               NAMESPACE
   cert-manager    cert-manager.tanzu.vmware.com    1.11.1+vmware.1-tkg.1     Reconcile succeeded  my-packages
   contour         contour.tanzu.vmware.com         1.24.4+vmware.1-tkg.1    Reconcile succeeded  my-packages
   grafana         grafana.tanzu.vmware.com         9.5.1+vmware.2-tkg.3     Reconcile succeeded  tkg-system
   prometheus      prometheus.tanzu.vmware.com      2.43.0+vmware.1-tkg.1    Reconcile succeeded  tkg-system
   antrea          antrea.tanzu.vmware.com                                   Reconcile succeeded  tkg-system
   [...]
   

   The grafana package and the grafana app are installed in the namespace that you specify when running the tanzu package install command.

2. Confirm that the grafana app is successfully reconciled:

   kubectl get apps -A
   

   For example:

   NAMESPACE     NAME                           DESCRIPTION           SINCE-DEPLOY   AGE
   my-packages   cert-manager                   Reconcile succeeded   37s            21h
   my-packages   contour                        Reconcile succeeded   33s            129m
   my-packages   grafana                        Reconcile succeeded   19s            6m56s
   my-packages   prometheus                     Reconcile succeeded   46s            21h
   tkg-system    antrea                         Reconcile succeeded   3m50s          24h
   [...]
   

   If the status is not Reconcile succeeded, view the full status details of the grafana app. Viewing the full status can help you troubleshoot the problem:

   kubectl get app grafana --namespace PACKAGE-NAMESPACE -o yaml
   

   Where PACKAGE-NAMESPACE is the namespace in which you installed the package.

3. Confirm that the new services are running by listing all of the pods that are running in the cluster:

   kubectl get pods -A
   

   In the tanzu-system-dashboards namespace, you should see the grafana service running in a pod:

   NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
   [...]
   tanzu-system-dashboards   grafana-6865dbb4f5-pk2qg              2/2     Running   0          7m7s
   [...]
   

The Grafana pods and any other resources associated with the Grafana component are created in the namespace you provided in grafana-data-values.yaml. If you are using the default namespace, these are created in the tanzu-system-dashboards namespace.

Grafana Package Configuration Parameters

There are two ways you can view configuration parameters of the Grafana package:

• Review the Package Schema
• Review Configuration Parameters

Review the Package Schema

To retrieve the package schema:

tanzu package available get grafana.tanzu.vmware.com/9.5.1+vmware.2-tkg.3 -n AVAILABLE-PACKAGE-NAMESPACE --values-schema

This command lists configuration parameters of the Grafana package and their default values. You can use the output to update your grafana-data-values.yml file created in Prepare the Grafana Package Configuration File above.

Review Configuration Parameters

The following table lists configuration parameters of the Grafana package and describes their default values.

You can set the following configuration values in your grafana-data-values.yml file created in Deploy Grafana on a Workload Cluster above.

Parameter	Description	Type	Default
namespace	Namespace where Grafana will be deployed.	String	tanzu-system-dashboards
grafana.deployment.replicas	Number of Grafana replicas.	Integer	1
grafana.deployment.containers.resources	Grafana container resource requests and limits.	Map	{}
grafana.deployment.k8sSidecar.containers.resources	k8s-sidecar container resource requests and limits.	Map	{}
grafana.deployment.podAnnotations	The Grafana deployments pod annotations.	Map	{}
grafana.deployment.podLabels	The Grafana deployments pod labels.	Map	{}
grafana.service.type	Type of service to expose Grafana. Supported Values: ClusterIP, NodePort, LoadBalancer. (For vSphere set this to NodePort)	String	LoadBalancer
grafana.service.port	Grafana service port.	Integer	80
grafana.service.targetPort	Grafana service target port.	Integer	9093
grafana.service.labels	Grafana service labels.	Map	{}
grafana.service.annotations	Grafana service annotations.	Map	{}
grafana.config.grafana_ini	For information about Grafana configuration, see Grafana Configuration Defaults in GitHub.	Config file	grafana.ini
grafana.config.datasource_yaml	For information about datasource config, see the Grafana documentation.	String	prometheus
grafana.config.dashboardProvider_yaml	For information about dashboard provider config, see the Grafana documentation.	YAML file	provider.yaml
grafana.pvc.annotations	Storage class to use for persistent volume claim. By default this is null and default provisioner is used.	String	null
grafana.pvc.storageClassName	Storage class to use for persistent volume claim. By default this is null and default provisioner is used.	String	null
grafana.pvc.accessMode	Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany.	String	ReadWriteOnce
grafana.pvc.storage	Define storage size for persistent volume claim.	String	2Gi
grafana.secret.type	Secret type defined for Grafana dashboard.	String	Opaque
grafana.secret.admin_user	Base64-encoded username to access Grafana dashboard. Defaults to YWRtaW4=, which is equivalent to admin in plain text.	String	YWRtaW4=
grafana.secret.admin_password	Base64-encoded password to access Grafana dashboard. Defaults to YWRtaW4=, which is equivalent to admin in plain text.	String	YWRtaW4=
ingress.enabled	Activate/Deactivate ingress for grafana.	Boolean	true
ingress.virtual_host_fqdn	Hostname for accessing grafana.	String	grafana.system.tanzu
ingress.prefix	Path prefix for grafana.	String	/
ingress.servicePort	Grafana service port to proxy traffic to.	Integer	80
ingress.tlsCertificate.tls.crt	Optional certificate for ingress if you want to use your own TLS cert. A self signed certificate is generated by default. Note tls.crt is a key and not nested.	String	Generated cert
ingress.tlsCertificate.tls.key	Optional certificate private key for ingress if you want to use your own TLS certificate. Note tls.key is a key and not nested.	String	Generated cert private key
ingress.tlsCertificate.ca.crt	Optional CA certificate. Note ca.crt is a key and not nested.	String	CA certificate

Update a Running Grafana Deployment

To make changes to the configuration of the Grafana package after deployment, update your deployed Grafana package:

1. Update the Grafana configuration in the grafana-data-values.yaml file.

2. Update the installed package:

   tanzu package installed update grafana \
   --version 9.5.1+vmware.2-tkg.3 \
   --values-file grafana-data-values.yaml \
   --namespace my-packages
   

   Expected output:

   | Updating package 'grafana'
   - Getting package install for 'grafana'
   | Updating secret 'grafana-my-packages-values'
   | Updating package install for 'grafana'
   
    Updated package install 'grafana' in namespace 'my-packages'
   

The Grafana package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Grafana Deployment

To remove the Grafana package on your cluster, run:

tanzu package installed delete grafana --namespace my-packages

For information about deleting, see Delete a Package.

Access the Grafana Dashboard

After Grafana is deployed, the grafana package creates a Contour HTTPProxy object with a Fully Qualified Domain Name (FQDN) of grafana.system.tanzu.

To use this FQDN to access the Grafana dashboard:

1. Create an entry in your local /etc/hosts file that points an IP address to this FQDN:

   • AWS or Azure: Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.
   • vSphere: Use the IP address of a worker node.

2. Navigate to https://grafana.system.tanzu.

Because the site uses self-signed certificates, you might need to navigate through a browser-specific security warning before you are able to access the dashboard.