Prometheus is an open-source systems monitoring and alerting toolkit. Tanzu Kubernetes Grid includes signed binaries for Prometheus that you can deploy on Tanzu Kubernetes clusters to monitor cluster health and services.

Prerequisites

  • A bootstrap machine with the following installed:
  • A Tanzu Kubernetes Grid management cluster and workload cluster running on vSphere, Amazon EC2, or Azure, with the package repository installed. For information about deploying, see Deploy Tanzu Kubernetes Clusters.
  • (Optional) If ingress for Prometheus is enabled, you must install the cert-managerand contour packages.

IMPORTANT

  • Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Prometheus Deployment

This procedure applies to all clusters, running on vSphere, Amazon EC2, and Azure.

To prepare the cluster:

  1. Get the admin credentials of the workload cluster into which you want to deploy Prometheus. For example:

    tanzu cluster kubeconfig get my-cluster --admin
    
  2. Set the context of kubectl to the cluster. For example:

    kubectl config use-context my-cluster-admin@my-cluster
    

(Optional) Enable Ingress for Prometheus

To enable ingress, you can install the below optional packages:

  1. Install Cert Manager. For information, see Installing Cert Manager.
  2. Install Contour. For information, see Implement Ingress control with Contour.

Continue to Deploy Prometheus on a Tanzu Kubernetes Cluster below.

Deploy Prometheus on a Tanzu Kubernetes Cluster

To install Prometheus:

  1. Confirm that the prometheus package is available in your workload cluster:

    tanzu package available list -A
    
  2. Retrieve the version of the available package:

    tanzu package available list prometheus.tanzu.vmware.com -A
    | Retrieving package versions for prometheus.tanzu.vmware.com...
     NAME                           VERSION                          RELEASED-AT           NAMESPACE
     prometheus.tanzu.vmware.com    2.27.0+vmware.1-tkg.1            2020-11-24T18:00:00Z  tanzu-package-repo-global
    

When you are ready to deploy Prometheus on a workload cluster, you can:

Deploy Prometheus with Default Configurations

After you confirm the package version and retrieve it, you can deploy it.

  1. Install the Prometheus package using its default values:

    If the target namespace exists in the cluster, run:

    tanzu package install prometheus \
    --package-name prometheus.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file prometheus-data-values.yaml \
    --namespace TARGET-NAMESPACE
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the package, such as my-packages. If this flag is not specified, the Tanzu CLI installs the package in the default namespace.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    For example:

    tanzu package install prometheus --package-name prometheus.tanzu.vmware.com --namespace my-packages --version 2.27.0+vmware.1-tkg.1
    
    \ Installing package 'prometheus.tanzu.vmware.com'
    | Creating namespace 'my-packages'
    / Getting package metadata for 'prometheus.tanzu.vmware.com'
    | Creating service account 'prometheus-my-packages-sa'
    | Creating cluster admin role 'prometheus-my-packages-cluster-role'
    | Creating cluster role binding 'prometheus-my-packages-cluster-rolebinding'
    - Creating package resource
    \ Package install status: Reconciling
    
    Added installed package 'prometheus' in namespace 'my-packages'
    

    If the target namespace does not exist in the cluster, run:

    tanzu package install prometheus \
    --package-name prometheus.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file prometheus-data-values.yaml \
    --namespace TARGET-NAMESPACE
    --create-namespace
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the package. For example, such as my-packages.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    For example:

    tanzu package install prometheus --package-name prometheus.tanzu.vmware.com --namespace my-packages --version 2.27.0+vmware.1-tkg.1 --create-namespace
    

Alternatively, you can create the namespace before installing the package by running the kubectl create namespace TARGET-NAMESPACE command.

Continue to Verify Prometheus Deployment below.

Deploy Prometheus with Custom Values

To install the Prometheus package using user-provided values:

  1. Retrieve the template of the Prometheus package's default configuration:

    image_url=$(kubectl -n tanzu-package-repo-global get packages prometheus.tanzu.vmware.com.PACKAGE-VERSION -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
    
    imgpkg pull -b $image_url -o /tmp/prometheus-package-PACKAGE-VERSION
    
    cp /tmp/prometheus-package-PACKAGE-VERSION/config/values.yaml prometheus-data-values.yaml
    

    For example:

    image_url=$(kubectl -n tanzu-package-repo-global get packages prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1 -o jsonpath='{.spec.template.spec.fetch[0].imgpkgBundle.image}')
    
    imgpkg pull -b $image_url -o /tmp/prometheus-package-2.27.0+vmware.1-tkg.1
    
    cp /tmp/prometheus-package-2.27.0+vmware.1-tkg.1/config/values.yaml prometheus-data-values.yaml
    

    This creates a configuration file named prometheus-data-values.yaml that you can modify.

    For information about configuration parameters to use in prometheus-data-values.yaml, see Prometheus Package Configuration Parameters below.

  2. After you make any changes needed to your prometheus-data-values.yaml file, remove all comments in it:

    yq -i eval '... comments=""' prometheus-data-values.yaml
    
  3. Deploy the package:

    tanzu package install prometheus \
    --package-name prometheus.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file prometheus-data-values.yaml \
    --namespace TARGET-NAMESPACE
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the package. For example, my-packages.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

    If the target namespace does not exist in the cluster, run:

    tanzu package install prometheus \
    --package-name prometheus.tanzu.vmware.com \
    --version AVAILABLE-PACKAGE-VERSION \
    --values-file prometheus-data-values.yaml \
    --namespace TARGET-NAMESPACE
    --create-namespace
    

    Where:

    • TARGET-NAMESPACE is the namespace in which you want to install the package. For example, my-packages.
    • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above.

Continue to Verify Prometheus Deployment below.

Verify Prometheus Deployment

After you deploy Prometheus, you can verify that the deployment is successful:

  1. Confirm that the prometheus package is installed. For example:

    tanzu package installed list -A
    / Retrieving installed packages...
    NAME            PACKAGE-NAME                       PACKAGE-VERSION                STATUS                   NAMESPACE
    cert-manager    cert-manager.tanzu.vmware.com      1.1.0+vmware.1-tkg.2           Reconcile succeeded      my-packages
    prometheus      prometheus.tanzu.vmware.com        2.27.0+vmware.1-tkg.1          Reconcile succeeded      my-packages
    antrea          antrea.tanzu.vmware.com                                           Reconcile succeeded      tkg-system
    metrics-server  metrics-server.tanzu.vmware.com                                   Reconcile succeeded      tkg-system
    vsphere-cpi     vsphere-cpi.tanzu.vmware.com                                      Reconcile succeeded      tkg-system
    vsphere-csi     vsphere-csi.tanzu.vmware.com                                      Reconcile succeeded      tkg-system
    

    The prometheus package and its resources, such as the prometheus app, are installed in the namespace that you specify when running the tanzu package install command.

  2. Confirm that the prometheus app is successfully reconciled in your AVAILABLE-PACKAGE-NAMESPACE:

    kubectl get apps -A
    

    For example:

    NAMESPACE     NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
    my-packages   cert-manager                        Reconcile succeeded   74s            29m
    my-packages   prometheus                          Reconcile succeeded   20s            33m
    tkg-system    antrea                              Reconcile succeeded   70s            3h43m
    [...]
    

    If the status is not Reconcile succeeded, view the full status details of theprometheus app. Viewing the full status can help you troubleshoot the problem:

    kubectl get app prometheus --namespace AVAILABLE-PACKAGE-NAMESPACE -o yaml
    

    Where AVAILABLE-PACKAGE-NAMESPACE is the namespace in which you installed the package.

  3. Confirm that the new services are running by listing all of the Pods that are running in the cluster.

    kubectl get pods -A
    

    In the tanzu-system-monitoring namespace, you should see the prometheus, alertmanager, node_exporter, pushgateway, cadvisor and kube_state_metrics services running in a Pod:

    NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
    [...]
    tanzu-system-monitoring   alertmanager-d6bb4d94d-7fgmb                             1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-cadvisor-pgfck                                1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-kube-state-metrics-868b5b749d-9w5f2           1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-node-exporter-97x6c                           1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-node-exporter-dnrkk                           1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-pushgateway-84cc9b85c6-tgmv6                  1/1     Running   0          35m
    tanzu-system-monitoring   prometheus-server-6479964fb6-kk9g2                       2/2     Running   0          35m
    [...]
    

The Prometheus Pods and any other resources associated with the Prometheus component are created in the namespace you provided in prometheus-data-values.yaml. If you are using the default namespace, these are created in tanzu-system-monitoring.

Prometheus Package Configuration Parameters

Note: Configuration parameters have changed since TKG v1.3. Review your configurations before deployment. For information about migrating from the TKG v1.3 extension, see Migrate the Prometheus Extension to a Package.

There are two ways you can view configuration parameters of the Prometheus package:

Review the Package Schema

You can review the package schema to expose configuration parameters:

tanzu package available get prometheus.tanzu.vmware.com/2.27.0+vmware.1-tkg.1 -n AVAILABLE-PACKAGE-NAMESPACE --values-schema

You can use the output to expose values to set in your prometheus-data-values.yml file created in Deploy Prometheus with Custom Values above.

Review Configuration Parameters

The following table lists configuration parameters of the Prometheus package and describes their default values.

You can set the following configuration values in your prometheus-data-values.yml file created in Deploy Prometheus with Custom Values above.

Parameter Description Type Default
namespace Namespace where Prometheus will be deployed. String tanzu-system-monitoring
prometheus.deployment.replicas Number of Prometheus replicas. String 1
prometheus.deployment.containers.args Prometheus container arguments. You can configure this parameter to change retention time. For information about configuring Prometheus storage parameters, see the Prometheus documentation. Note: Longer retention times require more storage capacity than shorter retention times. It might be necessary to increase the persistent volume claim size if you are significantly increasing the retention time. List n/a
prometheus.deployment.containers.resources Prometheus container resource requests and limits. Map {}
prometheus.deployment.podAnnotations The Prometheus deployments Pod annotations. Map {}
prometheus.deployment.podLabels The Prometheus deployments Pod labels. Map {}
prometheus.deployment.configMapReload.containers.args Configmap-reload container arguments. List n/a
prometheus.deployment.configMapReload.containers.resources Configmap-reload container resource requests and limits. Map {}
prometheus.service.type Type of service to expose Prometheus. Supported Values: ClusterIP. String ClusterIP
prometheus.service.port Prometheus service port. Integer 80
prometheus.service.targetPort Prometheus service target port. Integer 9090
prometheus.service.labels Prometheus service labels. Map {}
prometheus.service.annotations Prometheus service annotations. Map {}
prometheus.pvc.annotations Storage class annotations. Map {}
prometheus.pvc.storageClassName Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
prometheus.pvc.accessMode Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany. String ReadWriteOnce
prometheus.pvc.storage Define storage size for persistent volume claim. String 150Gi
prometheus.config.prometheus_yml For information about the global Prometheus configuration, see the Prometheus documentation. YAML file prometheus.yaml
prometheus.config.alerting_rules_yml For information about the Prometheus alerting rules, see the Prometheus documentation. YAML file alerting_rules.yaml
prometheus.config.recording_rules_yml For information about the Prometheus recording rules, see the Prometheus documentation. YAML file recording_rules.yaml
prometheus.config.alerts_yml Additional prometheus alerting rules are configured here. YAML file alerts_yml.yaml
prometheus.config.rules_yml Additional prometheus recording rules are configured here. YAML file rules_yml.yaml
alertmanager.deployment.replicas Number of alertmanager replicas. Integer 1
alertmanager.deployment.containers.resources Alertmanager container resource requests and limits. Map {}
alertmanager.deployment.podAnnotations The Alertmanager deployments Pod annotations. Map {}
alertmanager.deployment.podLabels The Alertmanager deployments Pod labels. Map {}
alertmanager.service.type Type of service to expose Alertmanager. Supported Values: ClusterIP. String ClusterIP
alertmanager.service.port Alertmanager service port. Integer 80
alertmanager.service.targetPort Alertmanager service target port. Integer 9093
alertmanager.service.labels Alertmanager service labels. Map {}
alertmanager.service.annotations Alertmanager service annotations. Map {}
alertmanager.pvc.annotations Storage class annotations. Map {}
alertmanager.pvc.storageClassName Storage class to use for persistent volume claim. By default this is null and default provisioner is used. String null
alertmanager.pvc.accessMode Define access mode for persistent volume claim. Supported values: ReadWriteOnce, ReadOnlyMany, ReadWriteMany. String ReadWriteOnce
alertmanager.pvc.storage Define storage size for persistent volume claim. String 2Gi
alertmanager.config.alertmanager_yml For information about the global YAML configuration for Alert Manager, see the Prometheus documentation. YAML file alertmanager_yml
kube_state_metrics.deployment.replicas Number of kube-state-metrics replicas. Integer 1
kube_state_metrics.deployment.containers.resources kube-state-metrics container resource requests and limits. Map {}
kube_state_metrics.deployment.podAnnotations The kube-state-metrics deployments Pod annotations. Map {}
kube_state_metrics.deployment.podLabels The kube-state-metrics deployments Pod labels. Map {}
kube_state_metrics.service.type Type of service to expose kube-state-metrics. Supported Values: ClusterIP. String ClusterIP
kube_state_metrics.service.port kube-state-metrics service port. Integer 80
kube_state_metrics.service.targetPort kube-state-metrics service target port. Integer 8080
kube_state_metrics.service.telemetryPort kube-state-metrics service telemetry port. Integer 81
kube_state_metrics.service.telemetryTargetPort kube-state-metrics service target telemetry port. Integer 8081
kube_state_metrics.service.labels kube-state-metrics service labels. Map {}
kube_state_metrics.service.annotations kube-state-metrics service annotations. Map {}
node_exporter.daemonset.replicas Number of node-exporter replicas. Integer 1
node_exporter.daemonset.containers.resources node-exporter container resource requests and limits. Map {}
node_exporter.daemonset.hostNetwork Host networking requested for this Pod. boolean false
node_exporter.daemonset.podAnnotations The node-exporter deployments Pod annotations. Map {}
node_exporter.daemonset.podLabels The node-exporter deployments Pod labels. Map {}
node_exporter.service.type Type of service to expose node-exporter. Supported Values: ClusterIP. String ClusterIP
node_exporter.service.port node-exporter service port. Integer 9100
node_exporter.service.targetPort node-exporter service target port. Integer 9100
node_exporter.service.labels node-exporter service labels. Map {}
node_exporter.service.annotations node-exporter service annotations. Map {}
pushgateway.deployment.replicas Number of pushgateway replicas. Integer 1
pushgateway.deployment.containers.resources pushgateway container resource requests and limits. Map {}
pushgateway.deployment.podAnnotations The pushgateway deployments Pod annotations. Map {}
pushgateway.deployment.podLabels The pushgateway deployments Pod labels. Map {}
pushgateway.service.type Type of service to expose pushgateway. Supported Values: ClusterIP. String ClusterIP
pushgateway.service.port pushgateway service port. Integer 9091
pushgateway.service.targetPort pushgateway service target port. Integer 9091
pushgateway.service.labels pushgateway service labels. Map {}
pushgateway.service.annotations pushgateway service annotations. Map {}
cadvisor.daemonset.replicas Number of cadvisor replicas. Integer 1
cadvisor.daemonset.containers.resources cadvisor container resource requests and limits. Map {}
cadvisor.daemonset.podAnnotations The cadvisor deployments Pod annotations. Map {}
cadvisor.daemonset.podLabels The cadvisor deployments Pod labels. Map {}
ingress.enabled Enable/disable ingress for prometheus and alertmanager. Boolean false
ingress.virtual_host_fqdn Hostname for accessing promethues and alertmanager. String prometheus.system.tanzu
ingress.prometheus_prefix Path prefix for prometheus. String /
ingress.alertmanager_prefix Path prefix for alertmanager. String /alertmanager/
ingress.prometheusServicePort Prometheus service port to proxy traffic to. Integer 80
ingress.alertmanagerServicePort Alertmanager service port to proxy traffic to. Integer 80
ingress.tlsCertificate.tls.crt Optional certificate for ingress if you want to use your own TLS certificate. A self signed certificate is generated by default. Note: tls.crt is a key and not nested. String Generated cert
ingress.tlsCertificate.tls.key Optional certificate private key for ingress if you want to use your own TLS certificate.
Note: tls.key is a key and not nested.
String Generated cert key
ingress.tlsCertificate.ca.crt Optional CA certificate. Note: ca.crt is a key and not nested. String CA certificate

Update a Running Prometheus Deployment

To make changes to the configuration of the Prometheus package after deployment, update your deployed Prometheus package:

  1. Update the Prometheus configuration in the prometheus-data-values.yaml file.

  2. Update the installed package:

    tanzu package installed update prometheus \
    --version 2.27.0+vmware.1-tkg.1 \
    --values-file prometheus-data-values.yaml \
    --namespace my-packages
    

    Expected output:

    | Updating package 'prometheus'
    - Getting package install for 'prometheus'
    | Updating secret 'prometheus-my-packages-values'
    | Updating package install for 'prometheus'
    
     Updated package install 'prometheus' in namespace 'my-packages'
    

The Prometheus package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Prometheus Deployment

To remove the Prometheus package on your cluster, run:

tanzu package installed delete prometheus --namespace my-packages

For information about deleting, see Delete a Package.

Configure Notifications in Alert Manager

To configure notifications for Alert Manager, edit the alertmanager.config.alertmanager_yml section in your prometheus-data-values.yml file.

For information about configuring notifications, such as Slack or Email, see Configuration in the Prometheus documentation.

Access the Prometheus Dashboard

By default, ingress is not enabled on Prometheus. This is because access to the Prometheus dashboard is not authenticated. To access the Prometheus dashboard:

  1. Deploy Contour on the cluster.

    For information about deploying Contour, see Implement Ingress control with Contour.

  2. Copy the ingress.enabled section below into prometheus-data-values.yaml.

    ingress:
      enabled: false
      virtual_host_fqdn: "prometheus.system.tanzu"
      prometheus_prefix: "/"
      alertmanager_prefix: "/alertmanager/"
      prometheusServicePort: 80
      alertmanagerServicePort: 80
      #! [Optional] The certificate for the ingress if you want to use your own TLS certificate.
      #! We will issue the certificate by cert-manager when it's empty.
      tlsCertificate:
        #! [Required] the certificate
        tls.crt:
        #! [Required] the private key
        tls.key:
        #! [Optional] the CA certificate
        ca.crt:
    
  3. Update ingress.enabled from false to true.

  4. Create a DNS record to map prometheus.system.tanzu to the address of the Envoy load balancer.

    To obtain the address of the Envoy load balancer, see Implement Ingress control with Contour.

  5. Access the Prometheus dashboard by navigating to https://prometheus.system.tanzu in a browser.

    Prometheus dashboard

What to Do Next

The Prometheus package is now running and scraping data from your cluster. To visualize the data in Grafana dashboards, see Deploy Grafana on Tanzu Kubernetes Clusters.

check-circle-line exclamation-circle-line close-line
Scroll to top icon