This topic describes how to set up monitoring tools to capture metrics from Windows worker-based Kubernetes clusters deployed by Tanzu Kubernetes Grid Integrated Edition (TKGI).
You can use any of the following monitoring tools to capture metrics from TKGI Windows worker-based Kubernetes clusters:
You can use the Healthwatch Healthwatch Exporter for TKGI tile to monitor the health of the TKGI Control Plane and your Linux and Windows cluster control plane nodes.
Healthwatch enables you to monitor the functionality of your TKGI environment and can be configured to expose metrics to a service or database external to your Ops Manager foundation. For more information, see Overview of the Healthwatch Exporter for TKGI Tile.
To configure cluster discovery in Healthwatch, see Configuring TKGI Cluster Discovery in the Healthwatch documentation.
Wavefront runs as an external service that you set up to monitor Windows worker-based clusters the same way that you set it up to monitor clusters running Linux worker nodes:
Install Helm on your local machine, if you do not already have it, by following Install and Configure Helm in the topic Using Helm with Tanzu Kubernetes Grid Integrated Edition.
Use the Helm CLI to deploy Wavefront to the target cluster:
helm install wavefront wavefront/wavefront --namespace wavefront \
--set clusterName=CLUSTER-NAME \
--set wavefront.url=https://INSTANCE-NAME.wavefront.com \
--set wavefront.token=API-TOKEN \
--set collector.usePKSPrefix=true \
--set collector.useDaemonset=false
Where:
CLUSTER-NAME
is the name of your Kubernetes cluster.INSTANCE-NAME
is the subdomain name for your Wavefront instance.API-TOKEN
is the Wavefront API token for your Wavefront subscription.Do one of the following:
Prometheus and Grafana are open source tools you can use to monitor your Kubernetes Windows clusters and worker nodes, and to visualize and alert on events occurring in those clusters and worker nodes.
Before installing Prometheus ensure you have installed kubectl. For more information about installing kubectl, see Install and Set Up kubectl in the Kubernetes documentation.
For additional Prometheus and Grafana prerequisites, see Prerequisites in the kube-prometheus GitHub repository.
To monitor Windows clusters and worker node metrics using Prometheus and Grafana:
To install Prometheus and Grafana:
To download the Prometheus source code:
Clone or download the Prometheus operator source code from Latest release in the prometheus-operator/kube-prometheus
GitHub repository.
Clone or download the Prometheus monitoring mixin for kubernetes source code from
Latest release in the kubernetes-monitoring/kubernetes-mixin
GitHub repository.
To generate a Prometheus dashboard configuration:
kubernetes-mixin/dashboards
source code directory.dashboards.libsonnet
configuration file.To add the Windows dashboard to the configuration add the following to the dashboards.libsonnet
file:
(import 'windows.libsonnet')
kubernetes-mixin
source code root directory.To generate the Prometheus dashboard, run:
make dashboards_out
Review the kubernetes-mixin/dashboards_out
directory and confirm the following Windows dashboard definition files were created:
k8s-resources-windows-cluster.json
k8s-resources-windows-namespace.json
k8s-resources-windows-pod.json
k8s-windows-cluster-rsrc-use.json
k8s-windows-node-rsrc-use.json
To generate Prometheus monitoring rules:
Copy the rule definition file from the kubernetes_mixin/rules
directory to the kube-prometheus/jsonnet/kube-prometheus/rules
directory:
cp KUBERNETES-MIXIN-PATH/rules/windows.libsonnet PROMETHEUS-OP-PATH/jsonnet/kube-prometheus/rules/windows.libsonnet
Where:
KUBERNETES-MIXIN-PATH
is the kubernetes_mixin
source directory.PROMETHEUS-OP-PATH
is the Prometheus operator path.kube-prometheus/jsonnet/kube-prometheus/rules
source code directory.rules.libsonnet
configuration file.To add the Windows rule to the configuration add the following to the rules.libsonnet
file:
(import 'windows.libsonnet')
kube-prometheus
Prometheus source code root directory.To generate all rules, run:
make manifests
To deploy Prometheus and Grafana:
kubeconfig
for your Windows cluster. For more information, see Retrieving Cluster Credentials and Configuration.kube-prometheus
Prometheus source code root directory.To deploy Prometheus and Grafana, run:
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/
To forward the Prometheus and Grafana ports to localhost run:
kubectl --namespace monitoring port-forward svc/prometheus-k8s PROMETHEUS-PORT
kubectl --namespace monitoring port-forward svc/grafana GRAFANA-PORT
Where:
PROMETHEUS-PORT
is a localhost port for Prometheus, for example 9090
.GRAFANA-PORT
is a localhost port for Grafana, for example 3000
.For more information about installing Prometheus and Grafana, see Quickstart in the prometheus-operator/kube-prometheus GitHub repository.
To verify Grafana is running:
To open the Grafana Dashboard, open a browser to:
http://localhost:GRAFANA-PORT
Where GRAFANA-PORT
is the localhost port for Grafana, for example 3000
.
To log in to the Grafana Dashboard authenticate with user admin
, password admin
.
To monitor Windows worker node metrics, you need endpoints, the Windows Node Exporter service, and the Windows Node Exporter service monitor.
To install and configure the Windows Node Exporter:
The Windows Node Exporter must be installed on all of the Windows worker nodes you want to monitor.
To determine the names and IP addresses of all of your Windows worker nodes:
Run the following:
kubectl get nodes -L spec.ip
To deploy the Windows Node Exporter to all of your Windows worker nodes, repeat the steps below for each Windows worker node you want to monitor:
To download the windows_exporter-0.13.0-amd64.msi
to the worker from v0.13.0 in the prometheus-community/windows_exporter repository on GitHub:
Powershell Invoke-WebRequest https://github.com/prometheus-community/windows_exporter/releases/download/v0.13.0/windows_exporter-0.13.0-amd64.msi ^
-OutFile DOWNLOAD-PATH
Where DOWNLOAD-PATH
is the full path to the output location and filename for the retrieved file. For example, c:\windows_exporter-0.13.0-amd64.msi
.
To start the windows_exporter
msi as a Windows service, open a command line:
msiexec /i DOWNLOAD-PATH ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,system,container,memory"
Where DOWNLOAD-PATH
is the full path to the output location and filename for the retrieved file. For example, c:\windows_exporter-0.13.0-amd64.msi
.
To support monitoring by Prometheus and Grafana, the Windows Node Exporter service must be configured to access your Windows worker node metrics.
To configure the Windows Node Exporter:
windows-exporter.yml
.Populate the file with the following:
apiVersion: v1
kind: Endpoints
metadata:
labels:
k8s-app: APP-NAME
name: ENDPOINT-NAME
namespace: kube-system
subsets:
- addresses:
- ip: NODE-IP
targetRef:
kind: Node
name: NODE-NAME
ports:
- name: http-metrics
port: 9182
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: APP-NAME
name: SERVICE-NAME
namespace: kube-system
spec:
clusterIP: None
ports:
- name: http-metrics
port: 9182
protocol: TCP
targetPort: 9182
sessionAffinity: None
type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: APP-NAME
name: SERVICEMONITOR-NAME
namespace: monitoring
spec:
endpoints:
- interval: 30s
port: http-metrics
jobLabel: k8s-app
namespaceSelector:
matchNames:
- kube-system
selector:
matchLabels:
k8s-app: APP-NAME
Where:
APP-NAME
is the name for the Kubernetes app.ENDPOINT-NAME
is the name for the endpoint.SERVICE-NAME
is the name for the service.SERVICEMONITOR-NAME
is the name for the service monitor.NODE-IP
is the IP address of the Windows worker node to monitor.NODE-NAME
is the name of the Windows worker node to monitor.For example:
apiVersion: v1 kind: Endpoints metadata: labels: k8s-app: wmi-exporter name: wmi-exporter namespace: kube-system subsets: - addresses: - ip: 192.168.1.1 targetRef: kind: Node name: nodeone - ip: 192.168.1.1 targetRef: kind: Node name: nodetwo ports: - name: http-metrics port: 9182 protocol: TCP
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: k8s-app: wmi-exporter name: wmi-exporter namespace: monitoring spec: endpoints: - interval: 30s port: http-metrics jobLabel: k8s-app namespaceSelector: matchNames: - kube-system selector: matchLabels: k8s-app: wmi-exporter
Create an additional subsets.addresses
item for each Windows worker node you want to monitor:
subsets:
- addresses:
- ip: NODE-ONE-IP
targetRef:
kind: Node
name: NODE-ONE-NAME
- ip: NODE-TWO-IP
targetRef:
kind: Node
name: NODE-TWO-NAME
- ip: NODE-THREE-IP
targetRef:
kind: Node
name: NODE-THREE-NAME
Where:
NODE-ONE-IP
is the IP address of the first Windows worker node to monitor.NODE-ONE-NAME
is the name of the first Windows worker node to monitor.NODE-TWO-IP
is the IP address of the second Windows worker node to monitor.NODE-TWO-NAME
is the name of the second Windows worker node to monitor.NODE-THREE-IP
is the IP address of the third Windows worker node to monitor.NODE-THREE-NAME
is the name of the third Windows worker node to monitor.Save your changes to the windows-exporter.yml
file.
To create the Windows node exporter objects, run:
kubectl create -f windows-exporter.yml
To verify your new targets in Prometheus, open a browser to:
http://localhost:PROMETHEUS-PORT/targets
Where PROMETHEUS-PORT
is the localhost port for Prometheus, for example 9090
.
To import the dashboard files generated above to Grafana:
Upload the five dashboard files from the kubernetes-mixin/dashboards_out
directory one at a time:
k8s-resources-windows-cluster.json
k8s-resources-windows-namespace.json
k8s-resources-windows-pod.json
k8s-windows-cluster-rsrc-use.json
k8s-windows-node-rsrc-use.json
If you prefer to use a different Grafana dashboard, you can use and configure the preferred dashboard to monitor the Windows Node Exporter. For more information, see Dashboards at the Grafana Labs site.
After completing the minimum installation steps described above:
admin/admin
user/password pair.Review the Grafana and Prometheus documentation for any additional steps you should complete: