Install Prometheus with Alertmanager

Follow these instructions to install Prometheus with Alertmanager on a TKG Service cluster that is running TKr for vSphere 8.x.

Prerequisites

Adhere to the following prerequisites.

General Requirements
Create the Package Repository.
Install Cert Manager.
Install Contour with Envoy (required to access the Prometheus dashboard).
Prometheus Package Reference

Create Prometheus Data Values

Prepare to install Prometheus by creating the data values file.

Get the latest Prometheus package version for your repository.
```
tanzu package available get prometheus.tanzu.vmware.com -n tkg-system
```
Or, using kubectl.
```
kubectl -n tkg-system get packages | grep prometheus
```
Note: Typically you should use the latest version unless your requirements differ.
Generate the prometheus-data-values.yaml file.
```
tanzu package available get prometheus.tanzu.vmware.com/2.45.0+vmware.1-tkg.2 --default-values-file-output prometheus-data-values.yaml
```
Where:
- 2.45.0+vmware.1-tkg.2 is the target package version
- prometheus-data-values.yaml is the name and path of the data values file to generate

Edit the prometheus-data-values.yaml file and configure the following values which are required to access the Prometheus dashboard. See Prometheus Package Reference for an example data values file and full list of configuration parameters.

Parameter	Description
`ingress.tlsCertificate.tls.crt`	A self-signed TLS cert is generated for ingress. Optionally you can override and provide your own.
`ingress.tlsCertificate.tls.key`	A self-signed TLS private key is generated for ingress. Optionally you can override and provide your own.
`ingress.enabled`	Set the value to `true` (default is false).
`ingress.virtual_host_fqdn`	Set the value to `prometheus.<your.domain>` (default is `prometheus.system.tanzu`).
`alertmanager.pvc.storageClassName`	Enter the name of the vSphere storage policy.
`prometheus.pvc.storageClassName`	Enter the name of the vSphere storage policy.

Install Prometheus

Complete these steps to install the Prometheus package.

Create the namespace.

kubectl create ns tanzu-system-monitoring

Install Prometheus.

tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.45.0+vmware.1-tkg.2 --values-file prometheus-data-values.yaml -n tanzu-system-monitoring

Verify Prometheus installation.

tanzu package installed list -n tanzu-system-monitoring

tanzu package installed get prometheus -n tanzu-system-monitoring

Verify Prometheus and Altermanager objects.

kubectl -n tanzu-system-monitoring get all

kubectl -n tanzu-system-monitoring get pvc

NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
alertmanager        Bound    pvc-a53f7091-9823-4b70-a9b4-c3d7a1e27a4b   2Gi        RWO            k8s-policy     2m30s
prometheus-server   Bound    pvc-41745d1d-9401-41d7-b44d-ba430ecc5cda   20Gi       RWO            k8s-policy     2m30s

Troubleshoot Prometheus Installation

If the tanzu package install prometheus operation returns the error "Failed to get final advertise address: No private IP address found, and explicit IP not provided," apply a package overlay to reconfigure the alertmanager component.

Create the file overlay-alertmanager.yaml.

---
#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "Deployment"}), overlay.subset({"metadata": {"name": "alertmanager"}}))
---
spec:
  template:
    spec:
      containers:
        #@overlay/match by="name",expects="0+"
        - name: alertmanager
          args:
            - --cluster.listen-address=

Use Kubectl to create a secret from the file overlay-alertmanager.yaml.

kubectl create secret generic alertmanager-overlay -n tkg-system -o yaml --dry-run=client --from-file=overlay-alertmanager.yaml | kubectl apply -f -

Use Kubectl to annotate the Prometheus package with the overlay secret.

kubectl annotate PackageInstall prometheus -n tkg-system ext.packaging.carvel.dev/ytt-paths-from-secret-name.1=alertmanager-overlay

Run the installation command again.

tanzu package install prometheus -p prometheus.tanzu.vmware.com -v 2.37.0+vmware.3-tkg.1 --values-file prometheus-data-values.yaml -n tanzu-system-monitoring

Access the Prometheus Dashboard

Once Prometheus is installed, complete the following steps to access the Prometheus dashboard.

Ensure that the ingress section of the prometheus-data-values.yaml file is populated with all required fields.

ingress:
  enabled: true
  virtual_host_fqdn: "prometheus.system.tanzu"
  prometheus_prefix: "/"
  alertmanager_prefix: "/alertmanager/"
  prometheusServicePort: 80
  alertmanagerServicePort: 80
  #! [Optional] The certificate for the ingress if you want to use your own TLS certificate.
  #! We will issue the certificate by cert-manager when it's empty.
  tlsCertificate:
    #! [Required] the certificate
    tls.crt:
    #! [Required] the private key
    tls.key:
    #! [Optional] the CA certificate
    ca.crt:

Get the public (external) IP address of the Contour with Envoy load balancer.
See Install Contour with Envoy.
Create a DNS record that maps the Prometheus FQDN you used (default is prometheus.system.tanzu) to the IP address of the Envoy load balancer.
Access the Prometheus dashboard by navigating to the Prometheus FQDN using a browser.