This topic describes how to create and manage sink resources for a Kubernetes cluster provisioned with VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), or for a namespace within a cluster.
Sinks collect logs and metrics about Kubernetes worker nodes in your TKGI deployment and workloads that are running on them.
You can create two types of sinks:
For more conceptual information about sinks, see Sink Architecture in Tanzu Kubernetes Grid Integrated Edition.
Before creating a sink resource:
Configure sink resources in the Tanzu Kubernetes Grid Integrated Edition tile > In-Cluster Monitoring:
ClusterLogSink
or LogSink
resource, select the Enable Log Sink Resources checkbox.ClusterMetricSink
or MetricSink
resource, select the Enable Metric Sink Resources checkbox.ClusterMetricSink
as described in Create a ClusterMetricSink Resource for Node Exporter Metrics below, select the Enable node exporter on workers checkbox. Install the Kubernetes CLI, kubectl
. For installation instructions, see Installing the Kubernetes CLI.
You can create log and metric sinks for clusters and namespaces.
ClusterLogSink
or LogSink
, see ClusterLogSink and LogSink Resources below.ClusterMetricSink
or MetricSink
, see ClusterMetricSink and MetricSink Resources below.To create ClusterLogSink
or LogSink
resources, you can:
Note: Log sinks created in TKGI do not support UDP connections.
Note: TKGI requires a secure connection for log forwarding when using ClusterLogSink
and LogSink
resources of type syslog
or webhook
. To forward logs using an unsecured connection, see Unsecured ClusterLogSink and LogSink Log Forwarding below.
ClusterLogSink
and LogSink
resources of type syslog
deliver logs using the TCP-based syslog protocol.
By default, TKGI uses a system root certificate authority (CA) certificate to secure syslog
ClusterLogSink
and LogSink
log forwarding connections, but you can optionally use a custom CA certificate to secure the connections.
To define a syslog
ClusterLogSink
or LogSink
resource, perform the following steps:
Create a YAML file that specifies your log destination in one of the following formats:
To use the default system root CA certificate to secure log forwarding connections:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
type: syslog
host: YOUR-LOG-DESTINATION
port: YOUR-LOG-DESTINATION-PORT
enable_tls: true
insecure_skip_verify: false
Where:
YOUR-SINK-RESOURCE
is the sink resource that you want to create. This must be either ClusterLogSink
or LogSink
. For information about these sink resources, see Overview.YOUR-SINK
is a name that you choose for your sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line if you are creating ClusterLogSink
.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.Note: enable_tls
must be true
.
To use a custom CA certificate to secure log forwarding connections:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
type: syslog
host: YOUR-LOG-DESTINATION
port: YOUR-LOG-DESTINATION-PORT
enable_tls: true
insecure_skip_verify: false
fluent_bit_ca_cert: YOUR-CA-CERT
Where:
YOUR-SINK-RESOURCE
is the sink resource that you want to create. This must be either ClusterLogSink
or LogSink
. For information about these sink resources, see Overview.YOUR-SINK
is a name that you choose for your sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line if you are creating ClusterLogSink
.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.YOUR-CA-CERT
is your custom CA certificate to secure the syslog
connection.Note: enable_tls
must be true
.
Save the YAML file with an appropriate file name. For example, my-cluster-log-sink.yml
.
Apply the ClusterLogSink
or LogSink
resource to your cluster by running the following command:
kubectl apply -f YOUR-SINK.yml
Where YOUR-SINK.yml
is the name of your YAML file. For example:
$ kubectl apply -f my-cluster-log-sink.yml
ClusterLogSink
and LogSink
resources of type webhook
batch logs into one-second units, wrap the resulting payload in JSON, and use the POST
method to deliver the logs to the address of your log management service.
To define a webhook ClusterLogSink
or LogSink
resource, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
type: webhook
url: YOUR-LOG-DESTINATION
Where:
YOUR-SINK-RESOURCE
is the sink resource you want to create. This must be either ClusterLogSink
or LogSink
. For information about these sink resources, see Overview.YOUR-SINK
is a name you choose for your sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line if you are creating ClusterLogSink
.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.Save the YAML file with an appropriate filename. For example, my-cluster-log-sink.yml
.
Apply the ClusterLogSink
or LogSink
resource to your cluster by running the following command:
kubectl apply -f YOUR-SINK.yml
Where YOUR-SINK.yml
is the name of your YAML file. For example:
$ kubectl apply -f my-cluster-log-sink.yml
ClusterLogSink
and LogSink
resources with a Fluent Bit output plugin deliver logs to the output plugin that you specify in your resource configuration.
To define a ClusterLogSink
or LogSink
resource with a Fluent Bit output plugin, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
type: http
output_properties:
Host: example.com
Format: json
Port: 443
tls: on
tls.verify: off
Where:
YOUR-SINK-RESOURCE
is the sink resource you want to create. This must be either ClusterLogSink
or LogSink
. For information about these sink resources, see Overview.YOUR-SINK
is a name you choose for your log sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line if you are creating ClusterLogSink
.Note: This is a sample plugin configuration for http
. For a full list of supported plugins, see the Fluent Bit documentation.
Save the YAML file with an appropriate filename. For example, my-cluster-log-sink.yml
.
Apply the ClusterLogSink
or LogSink
resource to your cluster by running the following command:
kubectl apply -f YOUR-SINK.yml
Where YOUR-SINK.yml
is the name of your YAML file. For example:
$ kubectl apply -f my-cluster-log-sink.yml
By default, TKGI uses a secure connection for log forwarding when using ClusterLogSink
and LogSink
resources of type syslog
or webhook
.
For debugging purposes on a local machine, you may want to temporarily forward logs using an unsecured connection. To do this, you must:
Deactivate sink forwarding validation by running the following command:
kubectl delete validatingwebhookconfigurations validator.pksapi.io
Set enable_tls
to false
in your log destination YAML file.
Warning: Deactivating secure log forwarding is not recommended.
ClusterMetricSink and MetricSink resources collect metrics from different sources:
ClusterMetricSink
resources collect metrics from a cluster. If you want to collect pod usage metrics, you have to use a ClusterMetricSink.MetricSink
resources collect metrics from a namespace within a cluster. If you want to isolate a certain workload’s metrics to its own output, you have to use a namespaced MetricSink.By default, a ClusterMetricSink
resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs that you specify in your ClusterMetricSink
configuration:
Do not use a ClusterMetricSink to isolate a specific workload’s metrics to its own output. A ClusterMetricSink cannot isolate one input from all the others because all of a ClusterMetricSink’s configurations are located in a shared ConfigMap. This means that each ClusterMetricSink’s input will also go to all other ClusterMetricSinks’ outputs.
A MetricSink
resource runs in a single node only and collects metrics from a namespace within a cluster using prometheus.io/scrape
annotations set to true
. A MetricSink creates a unique telegraf agent pod and ConfigMap in the namespace, then writes the metrics to one or more outputs that you specify in your MetricSink
configuration.
Do not use MetricSinks to collect pod usage metrics. Pod metrics come from the underlying container runtime, which does not isolate metrics based on namespace, so only ClusterMetricSink is able to fetch pod metrics.
For a list of supported output plugins, see Output Plugins in the telegraf GitHub repository.
Case 1. Isolating Output => MetricSink
If you want to isolate a certain workload’s metrics to its own output, you have to use a namespaced MetricSink. A MetricSink creates a unique telegraf agent pod and ConfigMap in the namespace.
ClusterMetricSinks cannot isolate one input from all the others because all ClusterMetricSinks’ configurations are located in a shared ConfigMap. This means that each ClusterMetricSink’s inputs will also go to all other ClusterMetricSinks’ outputs.
Note that MetricSinks do not have access to pod usage metrics, as the source of those metrics does not filter by namespace and gives usage for all pods on a node.
Case 2. Pod Usage Metrics (CPU, Memory) => ClusterMetricSink
If you want to get pod usage metrics, you have to use a ClusterMetricSink.
To get pod usage metrics, Telegraf agents need to run on every node in a cluster, and this is what the ClusterMetricSink DaemonSet does.
Pod metrics come from the underlying container runtime, which does not isolate metrics based on namespace, so only ClusterMetricSink is able to fetch pod metrics.
ClusterMetricSinks use the Kubernetes input plugin, which gives access to pod usage metrics for each kubernetes node. MetricSinks only run in a single node, so they do not have access to pod usage metrics.
To define a ClusterMetricSink
or MetricSink
resource, perform the following steps:
Create a YAML file in the following format:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
inputs:
outputs:
- type: YOUR-OUTPUT-PLUGIN
telegraf_agent_config:
interval: INTERVAL-LENGTH
Where:
YOUR-SINK-RESOURCE
is the sink resource you want to create. This must be either ClusterMetricSink
or MetricSink
. For information about these sink resources, see Overview.YOUR-SINK
is a name you choose for your sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line if you are creating ClusterMetricSink
.YOUR-OUTPUT-PLUGIN
is the name of the output plugin you want to use for your metrics.INTERVAL-LENGTH
is the interval, in seconds, separating sink collection of input data. The assigned INTERVAL-LENGTH
must be a positive integer less than 9223372037. The default is 10
. Note: You can leave the inputs
field blank. For ClusterMetricSink
, this field is configured to include metrics from the kubelet by default. For MetricSink
, the field includes all prometheus.io/scrape
annotations set to true
by default.
For example:
apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
name: http
spec:
inputs:
outputs:
- type: http
url: https://example.com
method: POST
data_format: json
This will send all cluster metrics provided by the kubernetes input plugin via json POST to https://example.com
To define a ClusterMetricSink
resource for collecting Node Exporter metrics, perform the following steps:
Create a YAML file in the following format:
apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
name: YOUR-SINK
spec:
inputs:
- monitor_kubernetes_pods: true
type: prometheus
outputs:
- type: YOUR-OUTPUT-PLUGIN
Where:
YOUR-SINK
is a name you choose for your sink.YOUR-OUTPUT-PLUGIN
is the name of the output plugin you want to use for your metrics.For example:
apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
name: http
spec:
inputs:
- monitor_kubernetes_pods: true
type: prometheus
outputs:
- type: http
url: https://example.com
method: POST
data_format: json
Save the YAML file with an appropriate filename. For example, my-cluster-metric-sink.yml
.
Apply the ClusterMetricSink
resource to your cluster by running the following command:
kubectl apply -f YOUR-SINK.yml
Where YOUR-SINK.yml
is the name of your YAML file. For example:
$ kubectl apply -f my-cluster-metric-sink.yml
The LogSink
and ClusterLogSink
resources allow users to set filters to include or exclude logs or events. For more information, see Monitoring Clusters with Log Sinks.
To filter log sinks, add a filter properties section to the YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
name: YOUR-SINK
namespace: YOUR-NAMESPACE
spec:
type: syslog
host: YOUR-LOG-DESTINATION
port: YOUR-LOG-DESTINATION-PORT
enable_tls: true
filters:
include-events: true
include-logs: false
Where:
YOUR-SINK-RESOURCE
is the sink resource type that you created. This must be either ClusterLogSink
or LogSink
.YOUR-SINK
is the name you chose for your sink.YOUR-NAMESPACE
is the name of your namespace. Omit this line for ClusterLogSink
.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.The default values for these filter properties is true. If you do not specify filter properties, both logs and events are included.
To list sinks for clusters and namespaces, use the commands in the following sections.
To list cluster log sinks, run the following command:
kubectl get clusterlogsinks
To list namespace log sinks, run the following command:
kubectl -n YOUR-NAMESPACE get logsinks
Where YOUR-NAMESPACE
is the name of your namespace.
To list cluster metric sinks, run the following command:
kubectl get clustermetricsinks
To list namespace metric sinks, run the following command:
kubectl -n YOUR-NAMESPACE get metricsinks
Where YOUR-NAMESPACE
is the name of your namespace.
To delete sinks for clusters and namespaces, use the commands in the following sections.
To delete a cluster log sink, run the following command:
kubectl delete clusterlogsink YOUR-SINK
Where YOUR-SINK
is the name of your sink.
To delete a namespace log sink, run the following command:
kubectl -n YOUR-NAMESPACE delete logsink YOUR-SINK
Where:
YOUR-NAMESPACE
is the name of your namespace.YOUR-SINK
is the name of your log sink.To delete a cluster metric sink, use the following command:
kubectl delete clustermetricsink YOUR-SINK
Where YOUR-SINK
is the name of your sink.
To delete a namespace metric sink, use the following command:
kubectl -n YOUR-NAMESPACE delete metricsink YOUR-SINK
Where:
YOUR-NAMESPACE
is the name of your namespace.YOUR-SINK
is the name of your metric sink.