Install Prometheus in Workload Clusters Deployed by a Standalone Management Cluster

This topic explains how to deploy Prometheus into a workload cluster. The procedures below apply to vSphere, Amazon Web Services (AWS), and Azure deployments.

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. Tanzu Kubernetes Grid includes signed binaries for Prometheus that you can deploy on workload clusters to monitor cluster health and services.

Prerequisites

• A bootstrap machine with the following installed:
  • Tanzu CLI, Tanzu CLI plugins, and kubectl, as described in Install the Tanzu CLI and Kubernetes CLI for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Kubernetes CLI for Use with Standalone Management Clusters.
  • yq v4.5 or later.
• A management cluster, either a vSphere with Tanzu Supervisor or a standalone management cluster running on vSphere, AWS, or Azure, with the Tanzu Standard package repository installed.
• A workload cluster created by the management cluster above, with the following packages installed:
  • Prometheus; see Deploy Prometheus on Workload Clusters.
  • Contour; see Install Contour for Ingress Control.
  • cert-manager; see Install Cert Manager for Certificate Management.
• (Optional) If ingress for Prometheus is enabled, you must install the cert-manager and contour packages.

Important

Support for IPv6 addresses in Tanzu Kubernetes Grid is limited; see Deploy Clusters on IPv6 (vSphere Only). If you are not deploying to an IPv6-only networking environment, you must provide IPv4 addresses in the following steps.

Prepare the Workload Cluster for Prometheus Deployment

To prepare the cluster:

1. Get the admin credentials of the workload cluster into which you want to deploy Prometheus. For example:

   tanzu cluster kubeconfig get my-cluster --admin
   

2. Set the context of kubectl to the cluster. For example:

   kubectl config use-context my-cluster-admin@my-cluster
   

(Optional) Enable Ingress for Prometheus

To enable ingress, you can install the below optional packages:

1. Install Cert Manager. For information, see Install Cert Manager for Certificate Management.
2. Install Contour. For information, see Install Contour for Ingress control.

Continue to Deploy Prometheus into the Workload Cluster below.

Deploy Prometheus into the Workload Cluster

To install Prometheus:

1. If the cluster does not have a package repository with the Prometheus package installed, such as the tanzu-standard repository, install one:

   tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
   

   Where:

   • PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.

   • PACKAGE-REPO-ENDPOINT is the URL of the package repository.

     • For TKG v2.3.1 release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2023.10.16. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.

2. Confirm that the Prometheus package is available in your workload cluster:

   tanzu package available list -A
   

3. Retrieve the version of the available package:

   tanzu package available list prometheus.tanzu.vmware.com -A
   | Retrieving package versions for prometheus.tanzu.vmware.com...
    NAME                           VERSION                          RELEASED-AT           NAMESPACE
    prometheus.tanzu.vmware.com    2.45.0+vmware.1-tkg.1            2020-11-24T18:00:00Z  tanzu-package-repo-global
   

When you are ready to deploy Prometheus, you can:

• Deploy Prometheus with Default Configurations
• Deploy Prometheus with Custom Values

Deploy Prometheus with Default Configurations

After you confirm the package version and retrieve it, you can install the package.

1. Install the Prometheus package using its default values:

   tanzu package install prometheus \
   --package prometheus.tanzu.vmware.com \
   --version AVAILABLE-PACKAGE-VERSION \
   --namespace TARGET-NAMESPACE
   

   Where:

   • TARGET-NAMESPACE is the namespace in which you want to install the Prometheus package. For example, the my-packages or tanzu-cli-managed-packages namespace.

     • If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Prometheus pods and any other resources associated with the Prometheus component are created in the tanzu-system-monitoring namespace; do not install the Prometheus package into this namespace.
     • The specified namespace must already exist, for example from running kubectl create namespace my-packages.

   • AVAILABLE-PACKAGE-VERSION is the version that you retrieved above, for example 2.45.0+vmware.1-tkg.1.

   For example:

   tanzu package install prometheus --package prometheus.tanzu.vmware.com --namespace my-packages --version 2.45.0+vmware.1-tkg.1
   

   \ Installing package 'prometheus.tanzu.vmware.com'
   | Getting package metadata for 'prometheus.tanzu.vmware.com'
   | Creating service account 'prometheus-my-packages-sa'
   | Creating cluster admin role 'prometheus-my-packages-cluster-role'
   | Creating cluster role binding 'prometheus-my-packages-cluster-rolebinding'
   - Creating package resource
   \ Package install status: Reconciling
   
   Added installed package 'prometheus' in namespace 'my-packages'
   

Continue to Verify Prometheus Deployment below.

Deploy Prometheus with Custom Values

To install the Prometheus package using user-provided values:

1. Create a configuration file. This file configures the Prometheus package.

   tanzu package available get prometheus.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
   

   Where PACKAGE-VERSION is the version of the Prometheus package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, prometheus-data-values.yaml. The above command creates a configuration file named prometheus-data-values.yaml containing the default values. Note that in the previous versions, this file was called prometheus-data-values.yaml.

   See Prometheus Configuration Parameters (Standalone MC) for a full list of available parameters.

2. After you make any changes needed to your prometheus-data-values.yaml file, remove all comments in it:

   yq -i eval '... comments=""' prometheus-data-values.yaml
   

3. Deploy the package:

   tanzu package install prometheus \
   --package prometheus.tanzu.vmware.com \
   --version PACKAGE-VERSION \
   --values-file prometheus-data-values.yaml \
   --namespace TARGET-NAMESPACE
   

   Where:

   • TARGET-NAMESPACE is the namespace in which you want to install the Prometheus package, Prometheus package app, and any other Kubernetes resources that describe the package. For example, the my-packages or tanzu-cli-managed-packages namespace. If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Prometheus pods and any other resources associated with the Prometheus component are created in the tanzu-system-monitoring namespace; do not install the Prometheus package into this namespace.
   • PACKAGE-VERSION is the version that you retrieved above, for example 2.45.0+vmware.1-tkg.1.

Continue to Verify Prometheus Deployment below.

Verify Prometheus Deployment

After you deploy Prometheus, you can verify that the deployment is successful:

1. Confirm that the Prometheus package is installed. For example:

   tanzu package installed list -A
   / Retrieving installed packages...
   NAME            PACKAGE-NAME                       PACKAGE-VERSION                STATUS                   NAMESPACE
   cert-manager    cert-manager.tanzu.vmware.com      1.12.2+vmware.1-tkg.2           Reconcile succeeded      my-packages
   prometheus      prometheus.tanzu.vmware.com        2.45.0+vmware.1-tkg.1          Reconcile succeeded      my-packages
   antrea          antrea.tanzu.vmware.com                                           Reconcile succeeded      tkg-system
   metrics-server  metrics-server.tanzu.vmware.com                                   Reconcile succeeded      tkg-system
   vsphere-cpi     vsphere-cpi.tanzu.vmware.com                                      Reconcile succeeded      tkg-system
   vsphere-csi     vsphere-csi.tanzu.vmware.com                                      Reconcile succeeded      tkg-system
   

   The prometheus package and the prometheus app are installed in the namespace that you specify when running the tanzu package install command.

2. Confirm that the prometheus app is successfully reconciled:

   kubectl get apps -A
   

   For example:

   NAMESPACE     NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
   my-packages   cert-manager                        Reconcile succeeded   74s            29m
   my-packages   prometheus                          Reconcile succeeded   20s            33m
   tkg-system    antrea                              Reconcile succeeded   70s            3h43m
   [...]
   

   If the status is not Reconcile succeeded, view the full status details of the prometheus app. Viewing the full status can help you troubleshoot the problem:

   kubectl get app prometheus --namespace PACKAGE-NAMESPACE -o yaml
   

   Where PACKAGE-NAMESPACE is the namespace in which you installed the package.

3. Confirm that the new services are running by listing all of the pods that are running in the cluster:

   kubectl get pods -A
   

   In the tanzu-system-monitoring namespace, you should see the prometheus, alertmanager, node_exporter, pushgateway, cadvisor and kube_state_metrics services running in a pod:

   NAMESPACE               NAME                                    READY   STATUS    RESTARTS   AGE
   [...]
   tanzu-system-monitoring   alertmanager-d6bb4d94d-7fgmb                             1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-cadvisor-pgfck                                1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-kube-state-metrics-868b5b749d-9w5f2           1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-node-exporter-97x6c                           1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-node-exporter-dnrkk                           1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-pushgateway-84cc9b85c6-tgmv6                  1/1     Running   0          35m
   tanzu-system-monitoring   prometheus-server-6479964fb6-kk9g2                       2/2     Running   0          35m
   [...]
   

The Prometheus pods and any other resources associated with the Prometheus component are created in the namespace you provided in prometheus-data-values.yaml. If you are using the default namespace, these are created in the tanzu-system-monitoring namespace.

Update a Running Prometheus Deployment

To make changes to the configuration of the Prometheus package after deployment, update your deployed Prometheus package:

1. Update the Prometheus configuration in the prometheus-data-values.yaml file.

2. Update the installed package:

   tanzu package installed update prometheus \
   --version 2.45.0+vmware.1-tkg.1 \
   --values-file prometheus-data-values.yaml \
   --namespace my-packages
   

   Expected output:

   | Updating package 'prometheus'
   - Getting package install for 'prometheus'
   | Updating secret 'prometheus-my-packages-values'
   | Updating package install for 'prometheus'
   
    Updated package install 'prometheus' in namespace 'my-packages'
   

The Prometheus package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Prometheus Deployment

To remove the Prometheus package on your cluster, run:

tanzu package installed delete prometheus --namespace my-packages

For information about deleting, see Delete a Package.

Configure Notifications in Alert Manager

To configure notifications for Alert Manager, edit the alertmanager.config.alertmanager_yml section in your prometheus-data-values.yaml file.

For information about configuring notifications, such as Slack or Email, see Configuration in the Prometheus documentation.

Access the Prometheus Dashboard

By default, ingress is not enabled on Prometheus. This is because access to the Prometheus dashboard is not authenticated. To access the Prometheus dashboard:

1. Deploy Contour on the cluster.

   For information about deploying Contour, see Install Contour for Ingress Control.

2. Copy the ingress.enabled section below into prometheus-data-values.yaml.

   ingress:
     enabled: false
     virtual_host_fqdn: "prometheus.system.tanzu"
     prometheus_prefix: "/"
     alertmanager_prefix: "/alertmanager/"
     prometheusServicePort: 80
     alertmanagerServicePort: 80
     #! [Optional] The certificate for the ingress if you want to use your own TLS certificate.
     #! We will issue the certificate by cert-manager when it's empty.
     tlsCertificate:
       #! [Required] the certificate
       tls.crt:
       #! [Required] the private key
       tls.key:
       #! [Optional] the CA certificate
       ca.crt:
   

3. Update ingress.enabled from false to true.

4. Create a DNS record to map prometheus.system.tanzu to the address of the Envoy load balancer.

   To obtain the address of the Envoy load balancer, see Install Contour for Ingress Control.

5. Access the Prometheus dashboard by navigating to https://prometheus.system.tanzu in a browser.

   

What to Do Next

The Prometheus package is now running and scraping data from your cluster. To visualize the data in Grafana dashboards, see Deploy Grafana on Workload Clusters.