Install Prometheus in Workload Clusters Deployed by a Supervisor

This topic explains how to deploy Prometheus to Tanzu Kubernetes Grid (TKG) workload clusters deployed to vSphere by a vSphere with Tanzu Supervisor.

Prometheus is a system and service monitoring system. Prometheus collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and triggers alerts if conditions are met. Alertmanager handles alerts generated by Prometheus and routes them to their receiving endpoints.

You can install Prometheus on a workload cluster in two ways:

Using the Tanzu CLI
Using kubectl

Install Prometheus Using the Tanzu CLI

Prerequisites

Adhere to the following prerequisites.

Prepare to Install Tanzu Packages.
Install Cert Manager Using the Tanzu CLI.
Install Contour Using the Tanzu CLI (Contour with Envoy is required to access the Prometheus dashboard).

Reference

Refer to the following topic as needed.

Prometheus, Alertmanager, and Grafana Components, Configuration, Data Values.

Install Prometheus with Alertmanager

Complete these steps to install the Prometheus package.

Create the namespace.

kubectl create ns tanzu-system-monitoring

Get the latest Prometheus package version for your repository.

kubectl -n tkg-system get packages

For example, the latest version is 2.37.0+vmware.3-tkg.1 which is what we will install.

prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1                    prometheus.tanzu.vmware.com                    2.27.0+vmware.1-tkg.1   49m9s
prometheus.tanzu.vmware.com.2.27.0+vmware.2-tkg.1                    prometheus.tanzu.vmware.com                    2.27.0+vmware.2-tkg.1   49m9s
prometheus.tanzu.vmware.com.2.36.2+vmware.1-tkg.1                    prometheus.tanzu.vmware.com                    2.36.2+vmware.1-tkg.1   49m9s
prometheus.tanzu.vmware.com.2.37.0+vmware.1-tkg.1                    prometheus.tanzu.vmware.com                    2.37.0+vmware.1-tkg.1   49m9s
prometheus.tanzu.vmware.com.2.37.0+vmware.2-tkg.1                    prometheus.tanzu.vmware.com                    2.37.0+vmware.2-tkg.1   49m9s
prometheus.tanzu.vmware.com.2.37.0+vmware.3-tkg.1                    prometheus.tanzu.vmware.com                    2.37.0+vmware.3-tkg.1   49m9s

Create the prometheus-data-values.yaml file using either of the following methods:
1. Copy the example provided. See Prometheus with Alertmanager Components, Configuration, Data Values.
2. Or, generate the file prometheus-default-values.yaml using the following command:
```
tanzu package available get prometheus.tanzu.vmware.com/2.37.0+vmware.3-tkg.1 --default-values-file-output prometheus-data-values.yaml
```

Edit the prometheus-data-values.yaml file and configure the following values. These values are required to access the Prometheus dashboard.

Parameter	Description
`ingress.tlsCertificate.tls.crt`	A self-signed TLS cert is generated for ingress. Optionally you can override and provide your own.
`ingress.tlsCertificate.tls.key`	A self-signed TLS private key is generated for ingress. Optionally you can override and provide your own.
`ingress.enabled`	Set the value to `true` (default is false).
`ingress.virtual_host_fqdn`	Set the value to `prometheus.<your.domain>` (default is `prometheus.system.tanzu`).
`alertmanager.pvc.storageClassName`	Enter the name of the vSphere storage policy.
`prometheus.pvc.storageClassName`	Enter the name of the vSphere storage policy.

See Prometheus with Alertmanager Components, Configuration, Data Values for a full list of all available configuration parameters.

Install the Prometheus package.

tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.37.0+vmware.3-tkg.1 --values-file prometheus-data-values.yaml -n tanzu-system-monitoring

Verify Prometheus installation using the Tanzu CLI.

tanzu package installed list -n tanzu-system-monitoring

NAME        PACKAGE-NAME                 PACKAGE-VERSION        STATUS
prometheus  prometheus.tanzu.vmware.com  2.37.0+vmware.3-tkg.1  Reconcile succeeded

tanzu package installed get prometheus -n tanzu-system-monitoring

NAME:                    prometheus
PACKAGE-NAME:            prometheus.tanzu.vmware.com
PACKAGE-VERSION:         2.37.0+vmware.3-tkg.1
STATUS:                  Reconcile succeeded

Verify Prometheus installation using Kubectl.

kubectl -n tanzu-system-monitoring get all

kubectl -n tanzu-system-monitoring get pvc

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
alertmanager        Bound    pvc-a53f7091-9823-4b70-a9b4-c3d7a1e27a4b   2Gi        RWO            k8s-policy     2m30s
prometheus-server   Bound    pvc-41745d1d-9401-41d7-b44d-ba430ecc5cda   20Gi       RWO            k8s-policy     2m30s

Troubleshoot Prometheus Installation

If the tanzu package install prometheus operation returns the error “Failed to get final advertise address: No private IP address found, and explicit IP not provided,” apply a package overlay to reconfigure the alertmanager component.

Create the file overlay-alertmanager.yaml.

---
#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "Deployment"}), overlay.subset({"metadata": {"name": "alertmanager"}}))
---
spec:
  template:
    spec:
      containers:
        #@overlay/match by="name",expects="0+"
        - name: alertmanager
          args:
            - --cluster.listen-address=

Use Kubectl to create a secret from the file overlay-alertmanager.yaml.

kubectl create secret generic alertmanager-overlay -n tkg-system -o yaml --dry-run=client --from-file=overlay-alertmanager.yaml | kubectl apply -f -

Use Kubectl to annotate the Prometheus package with the overlay secret.

kubectl annotate PackageInstall prometheus -n tkg-system ext.packaging.carvel.dev/ytt-paths-from-secret-name.1=alertmanager-overlay

Run the installation command again.

tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.37.0+vmware.3-tkg.1 --values-file prometheus-data-values.yaml -n tanzu-system-monitoring

Access the Prometheus Dashboard

Once Prometheus is installed, complete the following steps to access the Prometheus dashboard.

Ensure that the ingress section of the prometheus-data-values.yaml file is populated with all required fields.

ingress:
  enabled: true
  virtual_host_fqdn: "prometheus.system.tanzu"
  prometheus_prefix: "/"
  alertmanager_prefix: "/alertmanager/"
  prometheusServicePort: 80
  alertmanagerServicePort: 80
  #! [Optional] The certificate for the ingress if you want to use your own TLS certificate.
  #! We will issue the certificate by cert-manager when it's empty.
  tlsCertificate:
    #! [Required] the certificate
    tls.crt:
    #! [Required] the private key
    tls.key:
    #! [Optional] the CA certificate
    ca.crt:

Get the public (external) IP address of the Contour with Envoy load balancer.

See Install Contour Using the Tanzu CLI.
Create a DNS record that maps the Prometheus FQDN you used (default is prometheus.system.tanzu) to the IP address of the Envoy load balancer.
Access the Prometheus dashboard by navigating to the Prometheus FQDN using a browser.

Install Prometheus Using Kubectl

Prometheus is a system and service monitoring system. Install the Prometheus package to monitor and collect metrics for TKG clusters on Supervisor.

Prerequisites

Adhere to the following prerequisites before you install the Prometheus package.

Reference

Refer to the following topic as needed.

Prometheus, Alertmanager, and Grafana Components, Configuration, Data Values.