Use this reference when configuring additional parameters of prometheus addon via the Custom Resources(CRs) tab.

Configurable parameters

Parameter

Description

Type

Default value

Note

prometheus.deployment.replicas

Number of Prometheus replicas.

integer

1

prometheus.deployment.containers.args

Prometheus container arguments. You can configure this parameter to change retention time. For information about configuring Prometheus storage parameters, see the Prometheus documentation. Note: Longer retention times require more storage capacity. It might be necessary to increase the persistent volume claim size if you are significantly increasing the retention time.

list

- --storage.tsdb.retention.time=42d

- --config.file=/etc/config/prometheus.yml

- --storage.tsdb.path=/data

- --web.console.libraries=/etc/prometheus/console_libraries2

- --web.console.templates=/etc/prometheus/consoles

- --web.enable-lifecycle

Prometheus will replace the whole arg list, make sure the customized arg list contains all of these args.

prometheus.deployment.containers.resources

Prometheus container resource requests and limits.

map

{}

prometheus.deployment.podAnnotations

The Prometheus deployments pod annotations.

map

{}

prometheus.deployment.podLabels

The Prometheus deployments pod labels.

map

{}

prometheus.deployment.configMapReload.containers.args

Configmap-reload container arguments.

list

prometheus.deployment.configMapReload.containers.resources

Configmap-reload container resource requests and limits.

map

{}

prometheus.service.type

Type of service to expose Prometheus. 

Enum["ClusterIP","NodePort","LoadBalancer"]

ClusterIP

Immutable

prometheus.service.port

Prometheus service port.

Integer

80

Immutable

prometheus.service.targetPort

Prometheus service target port.

Integer

9090

Immutable

prometheus.service.labels

Prometheus service labels.

map

{}

prometheus.service.annotations

Prometheus service annotations.

map

{}

prometheus.pvc.annotations

PVC annotations.

map

{}

prometheus.pvc.storageClassName

Storage class to use for persistent volume claim.  The default storage class is used if it is not set.

string

Immutable, formatted on UI

prometheus.pvc.accessMode

Define access mode for persistent volume claim.

Enum["ReadWriteOnce", "ReadOnlyMany", "ReadWriteMany"]

ReadWriteOnce

Immutable, formatted on UI

prometheus.pvc.storage

Define storage size for persistent volume claim.

string

150Gi

Immutable, formatted on UI

prometheus.config.prometheus_yml

For information about the global Prometheus configuration, see the Prometheus documentation.

YAML file

prometheus.yaml

prometheus.config.alerting_rules_yml

For information about the Prometheus alerting rules, see the Prometheus documentation.

YAML file

alerting_rules.yaml

prometheus.config.recording_rules_yml

For information about the Prometheus recording rules, see the Prometheus documentation.

YAML file

recording_rules.yaml

prometheus.config.alerts_yml

Additional prometheus alerting rules are configured here.

YAML file

alerts_yml.yaml

prometheus.config.rules_yml

Additional prometheus recording rules are configured here.

YAML file

rules_yml.yaml

alertmanager.deployment.replicas

Number of alertmanager replicas.

Integer

1

alertmanager.deployment.containers.resources

Alertmanager container resource requests and limits.

map

{}

alertmanager.deployment.podAnnotations

The Alertmanager deployments pod annotations.

map

{}

alertmanager.deployment.podLabels

The Alertmanager deployments pod labels.

map

{}

alertmanager.service.type

Type of service to expose Alertmanager.

Enum["ClusterIP"]

ClusterIP

Immutable

alertmanager.service.port

Alertmanager service port.

Integer

80

Immutable

alertmanager.service.targetPort

Alertmanager service target port.

Integer

9093

Immutable

alertmanager.service.labels

Alertmanager service labels.

map

{}

alertmanager.service.annotations

Alertmanager service annotations.

map

{}

alertmanager.pvc.annotations

Alertmanager PVC annotations.

map

{}

alertmanager.pvc.storageClassName

Storage class to use for persistent volume claim.  The default provisioner is used if it is not set.

string

Immutable

alertmanager.pvc.accessMode

Define access mode for persistent volume claim.

Enum["ReadWriteOnce", "ReadOnlyMany", "ReadWriteMany"]

ReadWriteOnce

Immutable

alertmanager.pvc.storage

Define storage size for persistent volume claim.

string

2Gi

Immutable

alertmanager.config.alertmanager_yml

For information about the global YAML configuration for Alert Manager, see the Prometheus documentation.

YAML file

alertmanager_yml

kube_state_metrics.deployment.replicas

Number of kube-state-metrics replicas.

integer

1

kube_state_metrics.deployment.containers.resources

kube-state-metrics container resource requests and limits.

map

{}

kube_state_metrics.deployment.podAnnotations

The kube-state-metrics deployments pod annotations.

map

{}

kube_state_metrics.deployment.podLabels

The kube-state-metrics deployments pod labels.

map

{}

kube_state_metrics.service.type

Type of service to expose kube-state-metrics

Enum["ClusterIP"]

ClusterIP

Immutable

kube_state_metrics.service.port

kube-state-metrics service port.

Integer

80

Immutable

kube_state_metrics.service.targetPort

kube-state-metrics service target port.

Integer

8080

Immutable

kube_state_metrics.service.telemetryPort

kube-state-metrics service telemetry port.

Integer

81

Immutable

kube_state_metrics.service.telemetryTargetPort

kube-state-metrics service target telemetry port.

Integer

8081

Immutable

kube_state_metrics.service.labels

kube-state-metrics service labels.

map

{}

kube_state_metrics.service.annotations

kube-state-metrics service annotations.

map

{}

node_exporter.daemonset.replicas

Number of node-exporter replicas.

Integer

1

node_exporter.daemonset.containers.resources

node-exporter container resource requests and limits.

map

{}

node_exporter.daemonset.hostNetwork

Host networking requested for this pod.

boolean

false

node_exporter.daemonset.podAnnotations

The node-exporter deployments pod annotations.

map

{}

node_exporter.daemonset.podLabels

The node-exporter deployments pod labels.

map

{}

node_exporter.service.type

Type of service to expose node-exporter

Enum["ClusterIP"]

ClusterIP

Immutable

node_exporter.service.port

node-exporter service port.

Integer

9100

Immutable

node_exporter.service.targetPort

node-exporter service target port.

Integer

9100

Immutable

node_exporter.service.labels

node-exporter service labels.

map

{}

node_exporter.service.annotations

node-exporter service annotations.

map

{}

pushgateway.deployment.replicas

Number of pushgateway replicas.

Integer

1

pushgateway.deployment.containers.resources

pushgateway container resource requests and limits.

map

{}

pushgateway.deployment.podAnnotations

The pushgateway deployments pod annotations.

map

{}

pushgateway.deployment.podLabels

The pushgateway deployments pod labels.

map

{}

pushgateway.service.type

Type of service to expose pushgateway

Enum["ClusterIP"]

ClusterIP

Immutable

pushgateway.service.port

pushgateway service port.

Integer

9091

Immutable

pushgateway.service.targetPort

pushgateway service target port.

Integer

9091

Immutable

pushgateway.service.labels

pushgateway service labels.

map

{}

pushgateway.service.annotations

pushgateway service annotations.

map

{}

cadvisor.daemonset.replicas

Number of cadvisor replicas.

Integer

1

cadvisor.daemonset.containers.resources

cadvisor container resource requests and limits.

map

{}

cadvisor.daemonset.podAnnotations

The cadvisor deployments pod annotations.

map

{}

cadvisor.daemonset.podLabels

The cadvisor deployments pod labels.

map

{}

ingress.enabled

Enable/disable ingress for prometheus and alertmanager.

boolean

false

Immutable, depends on cert-manager addon and contour ingress controller

ingress.virtual_host_fqdn

Hostname for accessing promethues and alertmanager.

string

prometheus.system.tanzu

Immutable

ingress.prometheus_prefix

Path prefix for prometheus.

string

/

Immutable

ingress.alertmanager_prefix

Path prefix for alertmanager.

string

/alertmanager/

Immutable

ingress.prometheusServicePort

Prometheus service port to proxy traffic to.

Integer

80

Immutable

ingress.alertmanagerServicePort

Alertmanager service port to proxy traffic to.

Integer

80

Immutable

ingress.tlsCertificate.tls.crt

Optional certificate for ingress if you want to use your own TLS certificate. A self signed certificate is generated by default.

string

Generated cert

tls.crt is a key and not nested.

ingress.tlsCertificate.tls.key

Optional certificate private key for ingress if you want to use your own TLS certificate.

string

Generated cert key

tls.key is a key and not nested.

Ingress.tlsCertificate.ca.crt

Optional CA certificate.

string

CA certificate

ca.crt is a key and not nested.

A sample prometheus addon CR is:

metadata:
  name: prometheus
spec:
  clusterRef:
    name: wc0
    namespace: wc0
  name: prometheus
  namespace: wc0
  config:
    stringData:
      values.yaml: |
        prometheus:
          deployment:
            replicas: 1
            containers:
              args:
                - --storage.tsdb.retention.time=5d
                - --config.file=/etc/config/prometheus.yml
                - --storage.tsdb.path=/data
                - --web.console.libraries=/etc/prometheus/console_libraries2
                - --web.console.templates=/etc/prometheus/consoles
                - --web.enable-lifecycle
          service:
            type: NodePort
            port: 80
            targetPort: 9090
          pvc:
            accessMode: ReadWriteOnce
            storage: 150Gi
          config:
            prometheus_yml: |
              global:
                evaluation_interval: 1m
                scrape_interval: 1m
                scrape_timeout: 10s
              rule_files:
              - /etc/config/alerting_rules.yml
              - /etc/config/recording_rules.yml
              - /etc/config/alerts
              - /etc/config/rules
              scrape_configs:
              - job_name: 'prometheus'
                scrape_interval: 5s
                static_configs:
                - targets: ['localhost:9090']
              - job_name: 'kube-state-metrics'
                static_configs:
                - targets: ['prometheus-kube-state-metrics.tanzu-system-monitoring.svc.cluster.local:8080']
              - job_name: 'node-exporter'
                static_configs:
                - targets: ['prometheus-node-exporter.tanzu-system-monitoring.svc.cluster.local:9100']
              - job_name: 'kubernetes-pods'
                kubernetes_sd_configs:
                - role: pod
                relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                  action: keep
                  regex: true
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                  action: replace
                  target_label: __metrics_path__
                  regex: (.+)
                - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                  action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  target_label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                - source_labels: [__meta_kubernetes_namespace]
                  action: replace
                  target_label: kubernetes_namespace
                - source_labels: [__meta_kubernetes_pod_name]
                  action: replace
                  target_label: kubernetes_pod_name
                - source_labels: [__meta_kubernetes_pod_node_name]
                  action: replace
                  target_label: node
              - job_name: kubernetes-nodes-cadvisor
                kubernetes_sd_configs:
                - role: node
                relabel_configs:
                - action: labelmap
                  regex: __meta_kubernetes_node_label_(.+)
                - replacement: kubernetes.default.svc:443
                  target_label: __address__
                - regex: (.+)
                  replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
                  source_labels:
                  - __meta_kubernetes_node_name
                  target_label: __metrics_path__
                scheme: https
                tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: true
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              - job_name: kubernetes-apiservers
                kubernetes_sd_configs:
                - role: endpoints
                relabel_configs:
                - action: keep
                  regex: default;kubernetes;https
                  source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_service_name
                  - __meta_kubernetes_endpoint_port_name
                scheme: https
                tls_config:
                  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  insecure_skip_verify: true
                bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              alerting:
                alertmanagers:
                - scheme: http
                  static_configs:
                  - targets:
                    - alertmanager.tanzu-system-monitoring.svc:80
                - kubernetes_sd_configs:
                    - role: pod
                  relabel_configs:
                  - source_labels: [__meta_kubernetes_namespace]
                    regex: default
                    action: keep
                  - source_labels: [__meta_kubernetes_pod_label_app]
                    regex: prometheus
                    action: keep
                  - source_labels: [__meta_kubernetes_pod_label_component]
                    regex: alertmanager
                    action: keep
                  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_probe]
                    regex: .*
                    action: keep
                  - source_labels: [__meta_kubernetes_pod_container_port_number]
                    regex:
                    action: drop
            alerting_rules_yml: |
              {}
            recording_rules_yml: |
              groups:
                - name: vmw-telco-namespace-cpu-rules
                  interval: 1m
                  rules:
                  - record: tkg_namespace_cpu_usage_seconds
                    expr: sum by (namespace) (rate (container_cpu_usage_seconds_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_cpu_throttled_seconds
                    expr: sum by (namespace) (((rate(container_cpu_cfs_throttled_seconds_total[5m])) ) > 0 or  kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_cpu_request_core
                    expr: sum by (namespace) (kube_pod_container_resource_requests_cpu_cores)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_cpu_limits_core
                    expr: sum by (namespace) (kube_pod_container_resource_limits_cpu_cores > 0.0 or kube_pod_info < bool 0.1)
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-namespace-mem-rules
                  interval: 1m
                  rules:
                  - record: tkg_namespace_mem_usage_mb
                    expr: sum by (namespace) (container_memory_usage_bytes{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_mem_rss_mb
                    expr: sum by (namespace) (container_memory_rss{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job:  kubernetes-nodes-cadvisor
                  - record: tkg_namespace_mem_workingset_mb
                    expr: sum by (namespace) (container_memory_working_set_bytes{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_mem_request_mb
                    expr: sum by (namespace) (kube_pod_container_resource_requests_memory_bytes) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_mem_limit_mb
                    expr: sum by (namespace) ((kube_pod_container_resource_limits_memory_bytes / (1024*1024) )> 0 or kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-namespace-network-rules
                  interval: 1m
                  rules:
                  - record: tkg_namespace_network_tx_bytes
                    expr: sum by (namespace) (rate (container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_rx_bytes
                    expr: sum by (namespace) (rate (container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_tx_packets
                    expr: sum by (namespace) (rate (container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_rx_packets
                    expr: sum by (namespace) (rate (container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job:  kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_tx_drop_packets
                    expr: sum by (namespace) (rate (container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_rx_drop_packets
                    expr: sum by (namespace) (rate (container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_tx_errors
                    expr: sum by (namespace) (rate (container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_rx_errors
                    expr: sum by (namespace) (rate (container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_total_bytes
                    expr: sum by (namespace) (rate (container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_total_packets
                    expr: sum by (namespace) (rate (container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_total_drop_packets
                    expr: sum by (namespace) (rate (container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_network_total_errors
                    expr: sum by (namespace) (rate (container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-namespace-storage-rules
                  interval: 1m
                  rules:
                  - record: tkg_namespace_storage_pvc_bound
                    expr: sum by (namespace) ((kube_persistentvolumeclaim_status_phase{phase="Bound"}) > 0 or kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_storage_pvc_count
                    expr: sum by (namespace) ((kube_pod_spec_volumes_persistentvolumeclaims_info)> 0 or kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-namespace-other-rules
                  interval: 1m
                  rules:
                  - record: tkg_namespace_pods_qty_count
                    expr: sum by (namespace) (kube_pod_info)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_pods_reboot_5m_count
                    expr: sum by (namespace) (changes(kube_pod_status_ready{condition="true"}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_namespace_pods_broken_count
                    expr: sum by (namespace) (kube_pod_status_ready{condition="false"})
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-pod-cpu-rules
                  interval: 1m
                  rules:
                  - record: tkg_pod_cpu_usage_seconds
                    expr: sum by (pod) (rate (container_cpu_usage_seconds_total{container!~"POD",pod!="",image!=""}[5m])) * 100
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_cpu_request_core
                    expr: sum by (pod) (kube_pod_container_resource_requests_cpu_cores)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_cpu_limit_core
                    expr: sum by (pod) (kube_pod_container_resource_limits_cpu_cores > 0.0 or kube_pod_info < bool 0.1)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_cpu_throttled_seconds
                    expr: sum by (pod) (((rate(container_cpu_cfs_throttled_seconds_total[5m])) ) > 0 or  kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-pod-mem-rules
                  interval: 1m
                  rules:
                  - record: tkg_pod_mem_usage_mb
                    expr: sum by (pod) (container_memory_usage_bytes{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_mem_rss_mb
                    expr: sum by (pod) (container_memory_rss{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_mem_workingset_mb
                    expr: sum by (pod) (container_memory_working_set_bytes{container!~"POD",container!=""}) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_mem_request_mb
                    expr: sum by (pod) (kube_pod_container_resource_requests_memory_bytes) / (1024*1024)
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_mem_limit_mb
                    expr: sum by (pod) ((kube_pod_container_resource_limits_memory_bytes / (1024*1024) )> 0 or kube_pod_info < bool 0)
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-pod-network-rules
                  interval: 1m
                  rules:
                  - record: tkg_pod_network_tx_bytes
                    expr: sum by (pod) (rate (container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_rx_bytes
                    expr: sum by (pod) (rate (container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_tx_packets
                    expr: sum by (pod) (rate (container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_rx_packets
                    expr: sum by (pod) (rate (container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_tx_dropped_packets
                    expr: sum by (pod) (rate (container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_rx_dropped_packets
                    expr: sum by (pod) (rate (container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_tx_errors
                    expr: sum by (pod) (rate (container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_rx_errors
                    expr: sum by (pod) (rate (container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_total_bytes
                    expr: sum by (pod) (rate (container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_total_packets
                    expr: sum by (pod) (rate (container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_total_drop_packets
                    expr: sum by (pod) (rate (container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_network_total_errors
                    expr: sum by (pod) (rate (container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-pod-other-rules
                  interval: 1m
                  rules:
                  - record: tkg_pod_health_container_restarts_1hr_count
                    expr: sum by (pod) (increase(kube_pod_container_status_restarts_total[1h]))
                    labels:
                      job: kubernetes-nodes-cadvisor
                  - record: tkg_pod_health_unhealthy_count
                    expr: min_over_time(sum by (pod) (kube_pod_status_phase{phase=~"Pending|Unknown|Failed"})[15m:1m])
                    labels:
                      job: kubernetes-nodes-cadvisor
                - name: vmw-telco-node-cpu-rules
                  interval: 1m
                  rules:
                  - record: tkg_node_cpu_capacity_core
                    expr: sum by (node) (kube_node_status_capacity_cpu_cores)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_cpu_allocate_core
                    expr: sum by (node) (kube_node_status_allocatable_cpu_cores)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_cpu_usage_seconds
                    expr: (label_replace(sum by (instance) (rate(container_cpu_usage_seconds_total[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_cpu_throttled_seconds
                    expr: sum by (instance) (rate(container_cpu_cfs_throttled_seconds_total[5m]))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_cpu_request_core
                    expr: sum by (node) (kube_pod_container_resource_requests_cpu_cores)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_cpu_limits_core
                    expr: sum by (node) (kube_pod_container_resource_limits_cpu_cores)
                    labels:
                      job: kubernetes-service-endpoints
                - name: vmw-telco-node-mem-rules
                  interval: 1m
                  rules:
                  - record: tkg_node_mem_capacity_mb
                    expr: sum by (node) (kube_node_status_capacity_memory_bytes / (1024*1024))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_allocate_mb
                    expr: sum by (node) (kube_node_status_allocatable_memory_bytes / (1024*1024))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_request_mb
                    expr: sum by (node) (kube_pod_container_resource_requests_memory_bytes) / (1024*1024)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_limits_mb
                    expr: sum by (node) (kube_pod_container_resource_limits_memory_bytes) / (1024*1024)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_available_mb
                    expr: sum by (node) ((node_memory_MemAvailable_bytes / (1024*1024) ))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_free_mb
                    expr: sum by (node) ((node_memory_MemFree_bytes / (1024*1024) ))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_usage_mb
                    expr: (label_replace(sum by (instance) (container_memory_usage_bytes{container!~"POD",container!=""}) / (1024*1024), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_mem_free_pc
                    expr: sum ((node_memory_MemFree_bytes{job="kubernetes-pods"} / node_memory_MemTotal_bytes) *100) by (node)
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_oom_kill
                    expr: sum by(node) (node_vmstat_oom_kill)
                    labels:
                      job: kubernetes-service-endpoints
                - name: vmw-telco-node-network-rules
                  interval: 1m
                  rules:
                  - record: tkg_node_network_tx_bytes
                    expr: (label_replace(sum by (instance) (rate(container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_rx_bytes
                    expr: (label_replace(sum by (instance) (rate(container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_tx_packets
                    expr: (label_replace(sum by (instance) (rate(container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_rx_packets
                    expr: (label_replace(sum by (instance) (rate(container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_tx_dropped_packets
                    expr: (label_replace(sum by (instance) (rate(container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_rx_dropped_packets
                    expr: (label_replace(sum by (instance) (rate(container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_tx_errors
                    expr: (label_replace(sum by (instance) (rate(container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_rx_errors
                    expr: (label_replace(sum by (instance) (rate(container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m])), "node", "$1", "instance", "(.*)"))
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_total_bytes
                    expr: label_replace((sum by (instance) (rate (container_network_transmit_bytes_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_bytes_total{container!~"POD",pod!="",image!=""}[5m]))), "node", "$1", "instance", "(.*)")
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_total_packets
                    expr: label_replace((sum by (instance) (rate (container_network_transmit_packets_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_packets_total{container!~"POD",pod!="",image!=""}[5m]))), "node", "$1", "instance", "(.*)")
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_total_drop_packets
                    expr: label_replace((sum by (instance) (rate (container_network_transmit_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_packets_dropped_total{container!~"POD",pod!="",image!=""}[5m]))), "node", "$1", "instance", "(.*)")
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_network_total_errors
                    expr: label_replace((sum by (instance) (rate (container_network_transmit_errors_total{container!~"POD",pod!="",image!=""}[5m]) + rate (container_network_receive_errors_total{container!~"POD",pod!="",image!=""}[5m]))), "node", "$1", "instance", "(.*)")
                    labels:
                      job: kubernetes-service-endpoints
                - name: vmw-telco-node-other-rules
                  interval: 1m
                  rules:
                  - record: tkg_node_status_mempressure_count
                    expr: sum by (node) (kube_node_status_condition{condition="MemoryPressure",status="true"})
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_status_diskpressure_count
                    expr: sum by (node) (kube_node_status_condition{condition="DiskPressure",status="true"})
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_status_pidpressure_count
                    expr: sum by (node) (kube_node_status_condition{condition="PIDPressure",status="true"})
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_status_networkunavailable_count
                    expr: sum by (node) (kube_node_status_condition{condition="NetworkUnavailable",status="true"})
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_status_etcdb_bytes
                    expr: (label_replace(etcd_db_total_size_in_bytes, "instance", "$1", "instance", "(.+):(\\d+)")) * on (instance) group_left (node) (avg by (instance, node) (label_replace ((kube_pod_info), "instance", "$1", "host_ip", "(.*)")) )
                    labels:
                      job: kubernetes-service-endpoints
                  - record: tkg_node_status_apiserver_request_total
                    expr: sum((label_replace(apiserver_request_total, "instance", "$1", "instance", "(.+):(\\d+)")) * on (instance) group_left (node) (avg by (instance, node) (label_replace ((kube_pod_info), "instance", "$1", "host_ip", "(.*)")) )) by (node)
                    labels:
                      job: kubernetes-service-endpoints
        ingress:
          enabled: false
          virtual_host_fqdn: prometheus.system.tanzu
          prometheus_prefix: /
          alertmanager_prefix: /alertmanager/
          prometheusServicePort: 80
          alertmanagerServicePort: 80
        alertmanager:
          deployment:
            replicas: 1
          service:
            type: ClusterIP
            port: 80
            targetPort: 9093
          pvc:
            accessMode: ReadWriteOnce
            storage: 2Gi
          config:
            alertmanager_yml: |
              global: {}
              receivers:
              - name: default-receiver
              templates:
              - '/etc/alertmanager/templates/*.tmpl'
              route:
                group_interval: 5m
                group_wait: 10s
                receiver: default-receiver
                repeat_interval: 3h
        kube_state_metrics:
          deployment:
            replicas: 1
          service:
            type: ClusterIP
            port: 80
            targetPort: 8080
            telemetryPort: 81
            telemetryTargetPort: 8081
        node_exporter:
          daemonset:
            hostNetwork: false
            updatestrategy: RollingUpdate
          service:
            type: ClusterIP
            port: 9100
            targetPort: 9100
        pushgateway:
          deployment:
            replicas: 1
          service:
            type: ClusterIP
            port: 9091
            targetPort: 9091
        cadvisor:
          daemonset:
            updatestrategy: RollingUpdate

In this sample CR:

  • The TSDB retention time in parameter prometheus.deployment.containers.args is changed to 5 days instead of default 42 days.

  • Some recording rules are added to prometheus.config.recording_rules_yml. Customize them or add more as needed.

  • The prometheus.service.type is changed to NodePort so that it can be integrated with external components(e.g. vROPS or Grafana). See Prometheus service type.

Prometheus service type

By default, Prometheus is deployed with a service type of ClusterIP, this means it is NOT exposable to the outside world.

There are three options available for the prometheus.service.type:

  • ClusterIP – use the default configuration then prometheus service only can be accessed in workload cluster. The service can also be exposed via ingress however this depends on ingress controller and some other munual configuraitons.

  • NodePort(recommended) – exposes the prometheus service on a nodeport. TCA does not support to specify the actual Nodeport, K8s will allocate a random nodeport number(a high-range port number between 30,000 and 32,767). To determine what this nodeport number is (post configuration), user must view the service configuration from the TCA cluster with command kubectl get svc -n tanzu-system-monitoring prometheus-server, as can be seen in the following output, the prometheus-server is exposed on node port 32020, then other external components can integrate with Prometheus with URL http://<cluster-endpoint-ip>:32020

    capv@cp0-control-plane-kz5k6 [ ~ ]$ kubectl get svc -n tanzu-system-monitoring prometheus-server
    NAME              TYPE     CLUSTER-IP   EXTERNAL-IP PORT(S)      AGE
    prometheus-server NodePort 100.65.8.127 <node>      80:32020/TCP 25s
  • Loadbalancer – leverages Avi load balancer to expose service, this deployment method depends on load-balancer-and-ingress-service addon. TCA does not support to specify static VIP for prometheus service, Avi will allocate a VIP from default VIP pool for prometheus service, then other external components can integrate with Prometheus with URL http://<prometheus-VIP>.