Grafana lets you query, visualize, alert on, and explore metrics no matter where they are stored. Grafana provides tools to form graphs and visualizations from application data. Deploy the TKG Extension for Grafana to generate and view metrics for Tanzu Kubernetes clusters.

Grafana Extension Requirements

This topic describes how to deploy and manage the TKG Extension v1.3.1 for Grafana.

Adhere to the following prerequisites to deploy the extension. In addition, the Grafana monitoring extension requires a default persistent storage class. You can either create a cluster with a default persistent storage class, or specify one in the Prometheus configuration file when deploying the extension. See Review Persistent Storage Requirements for TKG Extensions.

Deploy the Grafana Extension for Visualization and Analytics

The TKG Extension for Grafana deploys a single container. For more information, see https://grafana.com/.
Container Resource Type Replicas Description
Grafana Deployment 2 Data visualization
The extension is configured to pull the containers from the VMware public registry at https://projects.registry.vmware.com/. If you are using a private registry, change the endpoint URL in the data values and extension configurations to match. See Configure the Grafana Extension.
  1. Verify that you have completed each of the extension prerequisites. See Grafana Extension Requirements.
  2. Change directory to the Grafana extension.
    cd /tkg-extensions-v1.3.1+vmware.1/extensions/monitoring/grafana
  3. Create the tanzu-system-monitoring namespace and Grafana service account and role objects.
    kubectl apply -f namespace-role.yaml
  4. Create a Grafana data values file.
    The example data values file provides the minimum required configuration.
    cp grafana-data-values.yaml.example grafana-data-values.yaml
  5. Configure the Grafana extension by updating grafana-data-values.yaml.

    Customize the configuration as needed. See Configure the Grafana Extension.

    The admin_password should be base64 encoded, but it will not block the deployment of the extension if it is not. In the example below, the password "admin" is base64 encoded. Encode your own Grafana password here: https://www.base64encode.org/.

    If the cluster is not provisioned with a default storage class, you can specify it in the data values file. Also, make sure the namespace has sufficient storage for the persistent volume claims.
    monitoring:
      grafana:
        image:
          repository: "projects.registry.vmware.com/tkg/grafana"
        pvc:
          storage_class: vwt-storage-policy
          storage: "8Gi"  
        secret:
          admin_password: "YWRtaW4="
      grafana_init_container:
        image:
          repository: "projects.registry.vmware.com/tkg/grafana"
      grafana_sc_dashboard:
        image:
          repository: "projects.registry.vmware.com/tkg/grafana"
    
  6. Create the Grafana secret with grafana-data-values file.
    kubectl create secret generic grafana-data-values --from-file=values.yaml=grafana-data-values.yaml -n tanzu-system-monitoring

    The grafana-data-values secret is created in the tanzu-system-monitoring namespace. Verify using kubectl get secrets -n tanzu-system-monitoring.

  7. Deploy the Grafana extension.
    kubectl apply -f grafana-extension.yaml

    On success the Grafana app is created: app.kappctrl.k14s.io/grafana created.

  8. Check the status of the Grafana app.
    kubectl get app grafana -n tanzu-system-monitoring
    The status should change from Reconciling to Reconcile succeeded. If the status is Reconcile failed, see Troubleshooting.
  9. View detailed status on the Grafana app.
    kubectl get app grafana -n tanzu-system-monitoring -o yaml
  10. Verify the Grafana Deployment.
    kubectl get deployments -n tanzu-system-monitoring

Troubleshoot Grafana Deployment

If the deployment or reconciliation fails, run kubectl get pods -A to view pod status. The contour and envoy pods should be Running. If a pod status is ImagePullBackOff or ImageCrashLoopBackOff, the container image could not be pulled. Check the registry URL in the data values and the extension YAML files and make sure they are accurate.

Check the container logs, where name-XXXX is the unique pod name when you run kubectl get pods -A:
kubectl logs pod/grafana-XXXX -c grafana -n tanzu-system-monitoring

Update the Grafana Extension

Update the Grafana extension that is deployed to a Tanzu Kubernetes cluster.

  1. Get the current Grafana data values from the grafana-data-values secret.
    kubectl get secret grafana-data-values -n tanzu-system-monitoring -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > grafana-data-values.yaml
    
  2. Update the Grafana data values in grafana-data-values.yaml. See Configure the Grafana Extension.
  3. Update the Grafana data values secret.
    kubectl create secret generic grafana-data-values --from-file=values.yaml=grafana-data-values.yaml -n tanzu-system-monitoring -o yaml --dry-run | kubectl replace -f-
    
    The Grafana extension is reconciled with the updated data values.
    Note: By default, kapp-controller will sync apps every 5 minutes. The update should take effect in 5 minutes or less. If you want the update to take effect immediately, change syncPeriod in grafana-extension.yaml to a lesser value and apply the Grafana extension using kubectl apply -f grafana-extension.yaml.
  4. Check the status of the extension.
    kubectl get app grafana -n tanzu-system-monitoring

    The status should change to Reconcile Succeeded once Grafana is updated.

  5. View detailed status and troubleshoot if necessary.
    kubectl get app grafana -n tanzu-system-monitoring -o yaml

Delete the Grafana Extension

Delete the Grafana extension from a Tanzu Kubernetes cluster.
Note: Complete the steps in order. Do not delete the namespace, service account, and role objects before the Grafana app is fully deleted. Doing so can lead to system errors.
Note: The Prometheus and Grafana extensions are deployed to the same namespace: tanzu-system-monitoring. If you have deployed both extensions to the same cluster, delete each extension before you delete the namespace.
  1. Change directory to the Grafana extension.
    cd /tkg-extensions-v1.3.1+vmware.1/extensions/monitoring/grafana
  2. Delete the Grafana app.
    kubectl delete app grafana -n tanzu-system-monitoring

    Expected result: app.kappctrl.k14s.io "grafana" deleted.

  3. Verify that the Grafana app is deleted.
    kubectl get app grafana -n tanzu-system-monintoring

    Expected result: apps.kappctrl.k14s.io "grafana" not found.

  4. Delete the tanzu-system-monitoring namespace and the Grafana service account and role objects.
    Warning: Do not perform this step if Prometheus is deployed.
    kubectl delete -f namespace-role.yaml
  5. If you want to redeploy Grafana, remove the secret grafana-data-values.
    kubectl delete secret grafana-data-values -n tanzu-system-monitoring

    Expected result: secret "grafana-data-values" deleted.

Upgrade the Grafana Extension

If you have an existing Grafana extension deployed, you can upgrade it to use the latest version.
  1. Export the Grafana configmap and save it as backup.
    kubectl get configmap grafana -n tanzu-system-monitoring -o 'go-template={{ index .data "grafana.yaml" }}' > grafana-configmap.yaml
  2. Delete the existing Grafana extension. See Delete the Grafana Extension.
  3. Deploy the Grafana extension. See Deploy the Grafana Extension for Visualization and Analytics.

Configure the Grafana Extension

The Grafana configuration is set in /tkg-extensions-v1.3.1+vmware.1/extensions/monitoring/grafana/grafana-data-values.yaml.
Table 1. Grafana Configuration Parameters
Parameter Description Type Default
monitoring.namespace Namespace where Prometheus will be deployed string tanzu-system-monitoring
monitoring.create_namespace The flag indicates whether to create the namespace specified by monitoring.namespace boolean false
monitoring.grafana.cluster_role.apiGroups api group defined for grafana clusterrole list [""]
monitoring.grafana.cluster_role.resources resources defined for grafana clusterrole list ["configmaps", "secrets"]
monitoring.grafana.cluster_role.verbs access permission defined for clusterrole list ["get", "watch", "list"]
monitoring.grafana.config.grafana_ini Grafana configuration file details config file grafana.ini

In this file, grafana_net URL is used to integrate with Grafana, for example, to import the dashboard directly from Grafana.com.

monitoring.grafana.config.datasource.type Grafana datasource type string prometheus
monitoring.grafana.config.datasource.access access mode. proxy or direct (Server or Browser in the UI) string proxy
monitoring.grafana.config.datasource.isDefault mark as default Grafana datasource boolean true
monitoring.grafana.config.provider_yaml Config file to define grafana dashboard provider yaml file provider.yaml
monitoring.grafana.service.type Type of service to expose Grafana. Supported Values: ClusterIP, NodePort, LoadBalancer string vSphere: NodePort, aws/azure: LoadBalancer
monitoring.grafana.pvc.storage_class Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany string ReadWriteOnce
monitoring.grafana.pvc.storage Define storage size for persistent volume claim string 2Gi
monitoring.grafana.deployment.replicas Number of grafana replicas integer 1
monitoring.grafana.image.repository Location of the repository with the Grafana image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment). string projects.registry.vmware.com/tkg/grafana
monitoring.grafana.image.name Name of Grafana image string grafana
monitoring.grafana.image.tag Grafana image tag. This value may need to be updated if you are upgrading the version. string v7.3.5_vmware.1
monitoring.grafana.image.pullPolicy Grafana image pull policy string IfNotPresent
monitoring.grafana.secret.type Secret type defined for Grafana dashboard string Opaque
monitoring.grafana.secret.admin_user username to access Grafana dashboard string YWRtaW4=

Value is base64 encoded; to decode: echo "xxxxxx" | base64 --decode

monitoring.grafana.secret.admin_password password to access Grafana dashboard string null
monitoring.grafana.secret.ldap_toml If using ldap auth, ldap configuration file path string ""
monitoring.grafana_init_container.image.repository Repository containing grafana init container image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment). string projects.registry.vmware.com/tkg/grafana
monitoring.grafana_init_container.image.name Name of grafana init container image string k8s-sidecar
monitoring.grafana_init_container.image.tag Grafana init container image tag. This value may need to be updated if you are upgrading the version. string 0.1.99
monitoring.grafana_init_container.image.pullPolicy grafana init container image pull policy string IfNotPresent
monitoring.grafana_sc_dashboard.image.repository Repository containing the Grafana dashboard image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment). string projects.registry.vmware.com/tkg/grafana
monitoring.grafana_sc_dashboard.image.name Name of grafana dashboard image string k8s-sidecar
monitoring.grafana_sc_dashboard.image.tag Grafana dashboard image tag. This value may need to be updated if you are upgrading the version. string 0.1.99
monitoring.grafana_sc_dashboard.image.pullPolicy grafana dashboard image pull policy string IfNotPresent
monitoring.grafana.ingress.enabled Enable/disable ingress for grafana boolean true
monitoring.grafana.ingress.virtual_host_fqdn Hostname for accessing grafana string grafana.system.tanzu
monitoring.grafana.ingress.prefix Path prefix for grafana string /
monitoring.grafana.ingress.tlsCertificate.tls.crt Optional cert for ingress if you want to use your own TLS cert. A self signed cert is generated by default string Generated cert
monitoring.grafana.ingress.tlsCertificate.tls.key Optional cert private key for ingress if you want to use your own TLS cert. string Generated cert key