Deploy Grafana on Workload Clusters

This topic explains how to deploy Grafana into a workload cluster. The procedures below apply to vSphere, Amazon Web Services (AWS), and Azure deployments.


Grafana is open-source software that allows you to visualize and analyze metrics data collected by Prometheus on your clusters. Tanzu Kubernetes Grid includes a Grafana package that you can deploy on your workload clusters.



Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Grafana Deployment

To prepare the cluster:

  1. Get the admin credentials of the workload cluster into which you want to deploy Grafana. For example:

    tanzu cluster kubeconfig get my-cluster --admin
  2. Set the context of kubectl to the cluster. For example:

    kubectl config use-context my-cluster-admin@my-cluster
  3. Enable Ingress for Grafana: By default, Grafana has Ingress enabled. This requires you to install the following packages:

Continue to Deploy Grafana on a Workload Cluster.

Deploy Grafana into the Workload Cluster

To deploy Grafana:

  1. If the cluster does not already have the standard package installed, install it:


    If you are targeting a plan-based cluster (legacy), skip this step. For plan-based clusters, the tanzu-standard package repository is automatically enabled in every cluster, in the tanzu-package-repo-global namespace.

    tanzu package repository add tanzu-standard --url PACKAGE-REPOSITORY-ENDPOINT --namespace tkg-system

    Where PACKAGE-REPOSITORY-ENDPOINT is the URL of the standard package repository. For this release, the URL is

    See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.

  2. Confirm that the Grafana package is available in your workload cluster:

    tanzu package available list -A
  3. Retrieve the version of the available package:

    tanzu package available list -A
    | Retrieving package versions for
     NAME                           VERSION                          RELEASED-AT           NAMESPACE       7.5.7+vmware.1-tkg.1             2021-05-19T18:00:00Z  tanzu-package-repo-global
  4. Create a configuration file for your Grafana configuration:

    tanzu package available get --default-values-file-output FILE-PATH

    Where PACKAGE-VERSION is the version of the Grafana package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, grafana-data-values.yaml.

    For information about configuration parameters to use in grafana-data-values.yaml, see Grafana Package Configuration Parameters below.

  5. Edit grafana-data-values.yaml and replace secret.admin_password with a Base64-encoded password. To generate a Base64 encoded password, run:

    echo -n 'mypassword' | base64

    You can also use the Base64 encoding tool at to encode your password. For example, using either method, mypassword results in the encoded password bXlwYXNzd29yZA==.

  6. vSphere with Tanzu: If you are deploying Grafana to a workload cluster created by a vSphere with Tanzu Supervisor, set a non-null value for ingress.pvc.storageClassName in the grafana-data-values.yaml file:

    virtual_host_fqdn: "grafana.system.tanzu"
      storageClassName: STORAGE-CLASS

    Where STORAGE-CLASS is the name of the cluster’s storage class, as returned by kubectl get storageclass.

  7. (Optional) Modify the Grafana datasource configuration in grafana-data-values.yaml. Grafana is configured with Prometheus as a default data source. If you have customized the Prometheus deployment namespace and it is not deployed in the default namespace, tanzu-system-monitoring, you need to change the Grafana datasource configuration in grafana-data-values.yaml. To change the datasource configuration, copy the section below into the position shown and modify url as needed.

    #! The namespace in which to deploy grafana.
    namespace: tanzu-system-dashboards
      #! The grafana configuration.
        #! Refer to
        datasource_yaml: |-
          apiVersion: 1
            - name: Prometheus
              type: prometheus
              url: prometheus-server.<change-to-prometheus-namespace>.svc.cluster.local
              access: proxy
              isDefault: true

    For information about configuration parameters to use in grafana-data-values.yaml, see Grafana Package Configuration Parameters below.

  8. After you make any changes needed to your grafana-data-values.yaml file, remove all comments in it:

    yq -i eval '... comments=""' grafana-data-values.yaml
  9. Deploy the package:

    tanzu package install grafana \
    --package \
    --values-file grafana-data-values.yaml \
    --namespace TARGET-NAMESPACE


    • TARGET-NAMESPACE is the namespace in which you want to install the Grafana package, Grafana package app, and any other Kubernetes resources that describe the package. For example, the my-packages or tanzu-user-managed-packages namespace. If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Grafana pods and any other resources associated with the Grafana component are created in the namespace you set in grafana-data-values.yaml; do not install the Grafana package into this namespace.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    For example:

    tanzu package install grafana --package --version 7.5.7+vmware.1-tkg.1 --values-file grafana-data-values.yaml --namespace my-packages
    - Installing package ''
    | Getting namespace 'my-packages'
    | Getting package metadata for ''
    | Creating service account 'grafana-my-packages-sa'
    | Creating cluster admin role 'grafana-my-packages-cluster-role'
    | Creating cluster role binding 'grafana-my-packages-cluster-rolebinding'
    | Creating secret 'grafana-my-packages-values'
    - Creating package resource
    \ Package install status: Reconciling
     Added installed package 'grafana' in namespace 'my-packages'
  10. vSphere with Tanzu: On vSphere 7.0 U2 with vSphere with Tanzu enabled, the tanzu package install grafana command may return the error:
    service init failed: failed to connect to database: failed to create SQLite database file "/var/lib/grafana/grafana.db": open /var/lib/grafana/grafana.db: permission denied.

    To fix this error, grant access to the Grafana database from a temporary pod with the same persistentVolumeClaim as Grafana:

    1. Create a pod spec file grafana-pvc-pod.yaml containing:

      apiVersion: v1
      kind: Pod
        name: my-graf-pod
        namespace: tanzu-package-repo-global
        - name: busybox
          image: ""
          - name: my-vol
            mountPath: "/var/lib/grafana"
          command: [ "sleep", "1000000" ]
          - name: my-vol
              claimName: grafana-pvc
    2. Create the pod:

      kubectl apply -f grafana-pvc-pod.yaml
    3. From within the pod, grant access to the Grafana database directory:

      kubectl exec -it my-graf-pod -n my-packages sh
      / $ ls -l /var/lib/grafana
      total 16
      drwx------    2 root     root         16384 Dec 10 16:57 lost+found
      / $ chmod -R 777 /var/lib/grafana
      / $ ls -l /var/lib/grafana
      total 16
      drwxrwxrwx    2 root     root         16384 Dec 10 16:57 lost+found
      / $ exit
    4. Delete the Grafana deployment to restart it:

      kubectl delete deploy grafana -n tanzu-package-repo-global
    5. Delete the pod:

      kubectl delete -f grafana-pvc-pod.yaml

Verify Grafana Deployment

After you deploy Grafana, you can verify that the deployment is successful:

  1. Confirm that the Grafana package is installed. For example:

    tanzu package installed list -A
    / Retrieving installed packages...
    NAME            PACKAGE-NAME                     PACKAGE-VERSION          STATUS               NAMESPACE
    cert-manager    1.1.0+vmware.1-tkg.2     Reconcile succeeded  my-packages
    contour         1.17.1+vmware.1-tkg.1    Reconcile succeeded  my-packages
    grafana         7.5.7+vmware.1-tkg.1     Reconcile succeeded  tkg-system
    prometheus      2.27.0+vmware.1-tkg.1    Reconcile succeeded  tkg-system
    antrea                                   Reconcile succeeded  tkg-system

    The grafana package and the grafana app are installed in the namespace that you specify when running the tanzu package install command.

  2. Confirm that the grafana app is successfully reconciled:

    kubectl get apps -A

    For example:

    NAMESPACE     NAME                           DESCRIPTION           SINCE-DEPLOY   AGE
    my-packages   cert-manager                   Reconcile succeeded   37s            21h
    my-packages   contour                        Reconcile succeeded   33s            129m
    my-packages   grafana                        Reconcile succeeded   19s            6m56s
    my-packages   prometheus                     Reconcile succeeded   46s            21h
    tkg-system    antrea                         Reconcile succeeded   3m50s          24h

    If the status is not Reconcile succeeded, view the full status details of the grafana app. Viewing the full status can help you troubleshoot the problem:

    kubectl get app grafana --namespace PACKAGE-NAMESPACE -o yaml

    Where PACKAGE-NAMESPACE is the namespace in which you installed the package.

  3. Confirm that the new services are running by listing all of the pods that are running in the cluster:

    kubectl get pods -A

    In the tanzu-system-dashboards namespace, you should see the grafana service running in a pod:

    NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
    tanzu-system-dashboards   grafana-6865dbb4f5-pk2qg              2/2     Running   0          7m7s

The Grafana pods and any other resources associated with the Grafana component are created in the namespace you provided in grafana-data-values.yaml. If you are using the default namespace, these are created in the tanzu-system-dashboards namespace.

Grafana Package Configuration Parameters

There are two ways you can view configuration parameters of the Grafana package:

Review the Package Schema

To retrieve the package schema:

tanzu package available get -n AVAILABLE-PACKAGE-NAMESPACE --values-schema

This command lists configuration parameters of the Grafana package and their default values. You can use the output to update your grafana-data-values.yml file created in Prepare the Grafana Package Configuration File above.

Review Configuration Parameters

The following table lists configuration parameters of the Grafana package and describes their default values.

You can set the following configuration values in your grafana-data-values.yml file created in Deploy Grafana on a Workload Cluster above.

Parameter Description Type Default
namespace Namespace where Grafana will be deployed. String tanzu-system-dashboards
grafana.deployment.replicas Number of Grafana replicas. Integer 1
grafana.deployment.containers.resources Grafana container resource requests and limits. Map {}
grafana.deployment.k8sSidecar.containers.resources k8s-sidecar container resource requests and limits. Map {}
grafana.deployment.podAnnotations The Grafana deployments pod annotations. Map {}
grafana.deployment.podLabels The Grafana deployments pod labels. Map {}
grafana.service.type Type of service to expose Grafana. Supported Values: ClusterIP, NodePort, LoadBalancer. (For vSphere set this to NodePort) String LoadBalancer
grafana.service.port Grafana service port. Integer 80
grafana.service.targetPort Grafana service target port. Integer 9093
grafana.service.labels Grafana service labels. Map {}
grafana.service.annotations Grafana service annotations. Map {}
grafana.config.grafana_ini For information about Grafana configuration, see Grafana Configuration Defaults in GitHub. Config file grafana.ini
grafana.config.datasource_yaml For information about datasource config, see the Grafana documentation. String prometheus
grafana.config.dashboardProvider_yaml For information about dashboard provider config, see the Grafana documentation. YAML file provider.yaml
grafana.pvc.annotations Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
grafana.pvc.storageClassName Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
grafana.pvc.accessMode Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany. String ReadWriteOnce Define storage size for persistent volume claim. String 2Gi
grafana.secret.type Secret type defined for Grafana dashboard. String Opaque
grafana.secret.admin_user Username to access Grafana dashboard. String YWRtaW4=
grafana.secret.admin_password Password to access Grafana dashboard. String admin
ingress.enabled Activate/Deactivate ingress for grafana. Boolean true
ingress.virtual_host_fqdn Hostname for accessing grafana. String grafana.system.tanzu
ingress.prefix Path prefix for grafana. String /
ingress.servicePort Grafana service port to proxy traffic to. Integer 80
ingress.tlsCertificate.tls.crt Optional certificate for ingress if you want to use your own TLS cert. A self signed certificate is generated by default. Note tls.crt is a key and not nested. String Generated cert
ingress.tlsCertificate.tls.key Optional certificate private key for ingress if you want to use your own TLS certificate. Note tls.key is a key and not nested. String Generated cert private key Optional CA certificate. Note ca.crt is a key and not nested. String CA certificate

Update a Running Grafana Deployment

To make changes to the configuration of the Grafana package after deployment, update your deployed Grafana package:

  1. Update the Grafana configuration in the grafana-data-values.yaml file.

  2. Update the installed package:

    tanzu package installed update grafana \
    --version 7.5.7+vmware.1-tkg.1 \
    --values-file grafana-data-values.yaml \
    --namespace my-packages

    Expected output:

    | Updating package 'grafana'
    - Getting package install for 'grafana'
    | Updating secret 'grafana-my-packages-values'
    | Updating package install for 'grafana'
     Updated package install 'grafana' in namespace 'my-packages'

The Grafana package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Grafana Deployment

To remove the Grafana package on your cluster, run:

tanzu package installed delete grafana --namespace my-packages

For information about deleting, see Delete a Package.

Access the Grafana Dashboard

After Grafana is deployed, the grafana package creates a Contour HTTPProxy object with a Fully Qualified Domain Name (FQDN) of grafana.system.tanzu.

To use this FQDN to access the Grafana dashboard:

  1. Create an entry in your local /etc/hosts file that points an IP address to this FQDN:

    • AWS or Azure: Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.
    • vSphere: Use the IP address of a worker node.
  2. Navigate to https://grafana.system.tanzu.

Because the site uses self-signed certificates, you might need to navigate through a browser-specific security warning before you are able to access the dashboard.

check-circle-line exclamation-circle-line close-line
Scroll to top icon