Deploy Grafana on Workload Clusters

This topic explains how to deploy Grafana on a workload cluster. The procedures below apply to vSphere, Amazon Web Services (AWS), and Azure deployments.

Grafana

Grafana is open-source software that allows you to visualize and analyze metrics data collected by Prometheus on your clusters. Tanzu Kubernetes Grid includes a Grafana package that you can deploy on your workload clusters.

Prerequisites

Important:

  • Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Grafana Deployment

To prepare the cluster:

  1. Get the admin credentials of the workload cluster into which you want to deploy Grafana. For example:

    tanzu cluster kubeconfig get my-cluster --admin
    
  2. Set the context of kubectl to the cluster. For example:

    kubectl config use-context my-cluster-admin@my-cluster
    
  3. Enable Ingress for Grafana: By default, Grafana has Ingress enabled. This requires you to install the following packages:

Continue to Deploy Grafana on a Workload Cluster.

Deploy Grafana into the Workload Cluster

Follow the procedure below to deploy Grafana:

  1. Confirm that the Grafana package is available in your workload cluster:

    tanzu package available list -A
    
  2. Retrieve the version of the available package:

    tanzu package available list grafana.tanzu.vmware.com -A
    | Retrieving package versions for grafana.tanzu.vmware.com...
     NAME                           VERSION                          RELEASED-AT           NAMESPACE
     grafana.tanzu.vmware.com       7.5.7+vmware.1-tkg.1             2021-05-19T18:00:00Z  tanzu-package-repo-global
    
  3. Create a configuration file for your Grafana configuration:

    tanzu package available get grafana.tanzu.vmware.com/PACKAGE-VERSION --generate-default-values-file
    
    

    Where PACKAGE-VERSION is the version of the Grafana package that you want to install. The above command creates a configuration file named grafana-default-values.yaml containing the default values. Note that in the previous versions, this file was called grafana-data-values.yaml.

    For information about configuration parameters to use in grafana-default-values.yaml, see Grafana Package Configuration Parameters below.

  4. Edit grafana-default-values.yaml and replace secret.admin_password with a Base64-encoded password. To generate a Base64 encoded password, run:

    echo -n 'mypassword' | base64
    

    You can also use the Base64 encoding tool at https://www.base64encode.org/ to encode your password. For example, using either method, mypassword results in the encoded password bXlwYXNzd29yZA==.

  5. vSphere with Tanzu: If you are deploying Grafana to a workload cluster created by a vSphere Supervisor cluster, set a non-null value for ingress.pvc.storageClassName in the grafana-default-values.yaml file:

    ingress:
    virtual_host_fqdn: "grafana.system.tanzu"
    pvc:
      storageClassName: STORAGE-CLASS
    

    Where STORAGE-CLASS is the name of the cluster’s storage class, as returned by kubectl get storageclass.

  6. (Optional) Modify the Grafana datasource configuration in grafana-default-values.yaml. Grafana is configured with Prometheus as a default data source. If you have customized the Prometheus deployment namespace and it is not deployed in the default namespace, tanzu-system-monitoring, you need to change the Grafana datasource configuration in grafana-default-values.yaml. To change the datasource configuration, copy the section below into the position shown and modify url as needed.

    #! The namespace in which to deploy grafana.
    namespace: tanzu-system-dashboards
    
    grafana:
      #! The grafana configuration.
      config:
        #! Refer to https://grafana.com/docs/grafana/latest/administration/provisioning/#example-data-source-config-file
        datasource_yaml: |-
          apiVersion: 1
          datasources:
            - name: Prometheus
              type: prometheus
              url: prometheus-server.<change-to-prometheus-namespace>.svc.cluster.local
              access: proxy
              isDefault: true
    

    For information about configuration parameters to use in grafana-default-values.yaml, see Grafana Package Configuration Parameters below.

  7. After you make any changes needed to your grafana-default-values.yaml file, remove all comments in it:

    yq -i eval '... comments=""' grafana-default-values.yaml
    
  8. Deploy the package:

    If the target namespace exists in the cluster, run:

    tanzu package install grafana \
    --package-name grafana.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file grafana-default-values.yaml \
    --namespace TARGET-NAMESPACE
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the Grafana package, Grafana package app, and any other Kubernetes resources that describe the package. For example, the my-packages or tanzu-user-managed-packages namespace. If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Grafana pods and any other resources associated with the Grafana component are created in the namespace you set in grafana-default-values.yaml; do not install the Grafana package into this namespace.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    For example:

    tanzu package install grafana --package-name grafana.tanzu.vmware.com --version 7.5.7+vmware.1-tkg.1 --values-file grafana-default-values.yaml --namespace my-packages
    
    - Installing package 'grafana.tanzu.vmware.com'
    | Getting namespace 'my-packages'
    | Getting package metadata for 'grafana.tanzu.vmware.com'
    | Creating service account 'grafana-my-packages-sa'
    | Creating cluster admin role 'grafana-my-packages-cluster-role'
    | Creating cluster role binding 'grafana-my-packages-cluster-rolebinding'
    | Creating secret 'grafana-my-packages-values'
    - Creating package resource
    \ Package install status: Reconciling
    
     Added installed package 'grafana' in namespace 'my-packages'
    

    If the target namespace does not exist in the cluster, run:

    tanzu package install grafana \
    --package-name grafana.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file grafana-default-values.yaml \
    --namespace TARGET-NAMESPACE
    --create-namespace
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the Grafana package, Grafana package app, and any other Kubernetes resources that describe the package. For example, the my-packages or tanzu-user-managed-packages namespace. If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Grafana pods and any other resources associated with the Grafana component are created in the namespace you set in grafana-default-values.yaml; do not install the Grafana package into this namespace.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    For example:

    tanzu package install grafana --package-name grafana.tanzu.vmware.com --version 7.5.7+vmware.1-tkg.1 --values-file grafana-default-values.yaml --namespace tanzu-package-repo-global --create-namespace
    
  9. vSphere with Tanzu: On vSphere 7.0 U2 with vSphere with Tanzu enabled, the tanzu package install grafana command may return the error:
    service init failed: failed to connect to database: failed to create SQLite database file "/var/lib/grafana/grafana.db": open /var/lib/grafana/grafana.db: permission denied.

    To fix this error, grant access to the Grafana database from a temporary pod with the same persistentVolumeClaim as Grafana:

    1. Create a pod spec file grafana-pvc-pod.yaml containing:

      apiVersion: v1
      kind: Pod
      metadata:
        name: my-graf-pod
        namespace: tanzu-package-repo-global
      spec:
        containers:
        - name: busybox
          image: "k8s.gcr.io/busybox"
          volumeMounts:
          - name: my-vol
            mountPath: "/var/lib/grafana"
          command: [ "sleep", "1000000" ]
        volumes:
          - name: my-vol
            persistentVolumeClaim:
              claimName: grafana-pvc
      
    2. Create the pod:

      kubectl apply -f grafana-pvc-pod.yaml
      
    3. From within the pod, grant access to the Grafana database directory:

      kubectl exec -it my-graf-pod -n my-packages sh
      
      / $ ls -l /var/lib/grafana
      total 16
      drwx------    2 root     root         16384 Dec 10 16:57 lost+found
      / $ chmod -R 777 /var/lib/grafana
      / $ ls -l /var/lib/grafana
      total 16
      drwxrwxrwx    2 root     root         16384 Dec 10 16:57 lost+found
      / $ exit
      
    4. Delete the Grafana deployment to restart it:

      kubectl delete deploy grafana -n tanzu-package-repo-global
      
    5. Delete the pod:

      kubectl delete -f grafana-pvc-pod.yaml
      

Verify Grafana Deployment

After you deploy Grafana, you can verify that the deployment is successful:

  1. Confirm that the Grafana package is installed. For example:

    tanzu package installed list -A
    / Retrieving installed packages...
    NAME            PACKAGE-NAME                     PACKAGE-VERSION          STATUS               NAMESPACE
    cert-manager    cert-manager.tanzu.vmware.com    1.1.0+vmware.1-tkg.2     Reconcile succeeded  my-packages
    contour         contour.tanzu.vmware.com         1.17.1+vmware.1-tkg.1    Reconcile succeeded  my-packages
    grafana         grafana.tanzu.vmware.com         7.5.7+vmware.1-tkg.1     Reconcile succeeded  tkg-system
    prometheus      prometheus.tanzu.vmware.com      2.27.0+vmware.1-tkg.1    Reconcile succeeded  tkg-system
    antrea          antrea.tanzu.vmware.com                                   Reconcile succeeded  tkg-system
    [...]
    

    The grafana package and the grafana app are installed in the namespace that you specify when running the tanzu package install command.

  2. Confirm that the grafana app is successfully reconciled:

    kubectl get apps -A
    

    For example:

    NAMESPACE     NAME                           DESCRIPTION           SINCE-DEPLOY   AGE
    my-packages   cert-manager                   Reconcile succeeded   37s            21h
    my-packages   contour                        Reconcile succeeded   33s            129m
    my-packages   grafana                        Reconcile succeeded   19s            6m56s
    my-packages   prometheus                     Reconcile succeeded   46s            21h
    tkg-system    antrea                         Reconcile succeeded   3m50s          24h
    [...]
    

    If the status is not Reconcile succeeded, view the full status details of the grafana app. Viewing the full status can help you troubleshoot the problem:

    kubectl get app grafana --namespace PACKAGE-NAMESPACE -o yaml
    

    Where PACKAGE-NAMESPACE is the namespace in which you installed the package.

  3. Confirm that the new services are running by listing all of the pods that are running in the cluster:

    kubectl get pods -A
    

    In the tanzu-system-dashboards namespace, you should see the grafana service running in a pod:

    NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
    [...]
    tanzu-system-dashboards   grafana-6865dbb4f5-pk2qg              2/2     Running   0          7m7s
    [...]
    

The Grafana pods and any other resources associated with the Grafana component are created in the namespace you provided in grafana-default-values.yaml. If you are using the default namespace, these are created in the tanzu-system-dashboards namespace.

Grafana Package Configuration Parameters

There are two ways you can view configuration parameters of the Grafana package:

Review the Package Schema

To retrieve the package schema:

tanzu package available get grafana.tanzu.vmware.com/7.5.7+vmware.1-tkg.1 -n AVAILABLE-PACKAGE-NAMESPACE --values-schema

This command lists configuration parameters of the Grafana package and their default values. You can use the output to update your grafana-data-values.yml file created in Prepare the Grafana Package Configuration File above.

Review Configuration Parameters

The following table lists configuration parameters of the Grafana package and describes their default values.

You can set the following configuration values in your grafana-data-values.yml file created in Deploy Grafana on a Workload Cluster above.

Parameter Description Type Default
namespace Namespace where Grafana will be deployed. String tanzu-system-dashboards
grafana.deployment.replicas Number of Grafana replicas. Integer 1
grafana.deployment.containers.resources Grafana container resource requests and limits. Map {}
grafana.deployment.k8sSidecar.containers.resources k8s-sidecar container resource requests and limits. Map {}
grafana.deployment.podAnnotations The Grafana deployments pod annotations. Map {}
grafana.deployment.podLabels The Grafana deployments pod labels. Map {}
grafana.service.type Type of service to expose Grafana. Supported Values: ClusterIP, NodePort, LoadBalancer. (For vSphere set this to NodePort) String LoadBalancer
grafana.service.port Grafana service port. Integer 80
grafana.service.targetPort Grafana service target port. Integer 9093
grafana.service.labels Grafana service labels. Map {}
grafana.service.annotations Grafana service annotations. Map {}
grafana.config.grafana_ini For information about Grafana configuration, see Grafana Configuration Defaults in GitHub. Config file grafana.ini
grafana.config.datasource_yaml For information about datasource config, see the Grafana documentation. String prometheus
grafana.config.dashboardProvider_yaml For information about dashboard provider config, see the Grafana documentation. YAML file provider.yaml
grafana.pvc.annotations Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
grafana.pvc.storageClassName Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
grafana.pvc.accessMode Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany. String ReadWriteOnce
grafana.pvc.storage Define storage size for persistent volume claim. String 2Gi
grafana.secret.type Secret type defined for Grafana dashboard. String Opaque
grafana.secret.admin_user Username to access Grafana dashboard. String YWRtaW4=
grafana.secret.admin_password Password to access Grafana dashboard. String admin
ingress.enabled Enable/disable ingress for grafana. Boolean true
ingress.virtual_host_fqdn Hostname for accessing grafana. String grafana.system.tanzu
ingress.prefix Path prefix for grafana. String /
ingress.servicePort Grafana service port to proxy traffic to. Integer 80
ingress.tlsCertificate.tls.crt Optional certificate for ingress if you want to use your own TLS cert. A self signed certificate is generated by default. Note: tls.crt is a key and not nested. String Generated cert
ingress.tlsCertificate.tls.key Optional certificate private key for ingress if you want to use your own TLS certificate. Note: tls.key is a key and not nested. String Generated cert private key
ingress.tlsCertificate.ca.crt Optional CA certificate. Note: ca.crt is a key and not nested. String CA certificate

Update a Running Grafana Deployment

To make changes to the configuration of the Grafana package after deployment, update your deployed Grafana package:

  1. Update the Grafana configuration in the grafana-default-values.yaml file.

  2. Update the installed package:

    tanzu package installed update grafana \
    --version 7.5.7+vmware.1-tkg.1 \
    --values-file grafana-default-values.yaml \
    --namespace my-packages
    

    Expected output:

    | Updating package 'grafana'
    - Getting package install for 'grafana'
    | Updating secret 'grafana-my-packages-values'
    | Updating package install for 'grafana'
    
     Updated package install 'grafana' in namespace 'my-packages'
    

The Grafana package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Grafana Deployment

To remove the Grafana package on your cluster, run:

tanzu package installed delete grafana --namespace my-packages

For information about deleting, see Delete a Package.

Access the Grafana Dashboard

After Grafana is deployed, the grafana package creates a Contour HTTPProxy object with a Fully Qualified Domain Name (FQDN) of grafana.system.tanzu.

To use this FQDN to access the Grafana dashboard:

  1. Create an entry in your local /etc/hosts file that points an IP address to this FQDN:

    • AWS or Azure: Use the IP address of the LoadBalancer for the Envoy service in the tanzu-system-ingress namespace.
    • vSphere: Use the IP address of a worker node.
  2. Navigate to https://grafana.system.tanzu.

Because the site uses self-signed certificates, you might need to navigate through a browser-specific security warning before you are able to access the dashboard.

check-circle-line exclamation-circle-line close-line
Scroll to top icon