Creating and Managing Sink Resources

This topic describes how to create and manage sink resources for a Kubernetes cluster provisioned with VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), or for a namespace within a cluster.

Overview

Sinks collect logs and metrics about Kubernetes worker nodes in your TKGI deployment and workloads that are running on them.

You can create two types of sinks:

Log sinks
Metric sinks

For more conceptual information about sinks, see Sink Architecture in Tanzu Kubernetes Grid Integrated Edition.

Prerequisites

Before creating a sink resource:

Review Sink Types in Sink Architecture in Tanzu Kubernetes Grid Integrated Edition.
Configure sink resources in the Tanzu Kubernetes Grid Integrated Edition tile > In-Cluster Monitoring:
- If you want to create a ClusterLogSink or LogSink resource, select the Enable Log Sink Resources check box.
- If you want to create a ClusterMetricSink or MetricSink resource, select the Enable Metric Sink Resources check box.
- If you want to use Node Exporter to send worker node metrics to metric sinks of kind ClusterMetricSink as described in Create a ClusterMetricSink Resource for Node Exporter Metrics below, select the Enable node exporter on workers check box.
  
  For more information about these configuration settings, see the TKGI installation topic for your IaaS:
Install the Kubernetes CLI, kubectl. For installation instructions, see Installing the Kubernetes CLI.

Create LogSink or ClusterLogSink Resources

To create ClusterLogSink or LogSink resources, you can:

Create a Syslog ClusterLogSink or LogSink Resource
Create a Webhook ClusterLogSink or LogSink Resource
Create a ClusterLogSink or LogSink Resource with a Fluent Bit Output Plugin

Note: Log sinks created in TKGI do not support UDP connections.

Note: TKGI requires a secure connection for log forwarding when using ClusterLogSink and LogSink resources of type syslog or webhook. To forward logs using an unsecured connection, see Unsecured ClusterLogSink and LogSink Log Forwarding below.

Create a Syslog ClusterLogSink or LogSink Resource

ClusterLogSink and LogSink resources of type syslog deliver logs using the TCP-based syslog protocol.

By default, TKGI uses a system root certificate authority (CA) certificate to secure syslog ClusterLogSink and LogSink log forwarding connections, but you can optionally use a custom CA certificate to secure the connections.

To define a syslog ClusterLogSink or LogSink resource, perform the following steps:

Create a YAML file that specifies your log destination in one of the following formats:
- To use the default system root CA certificate to secure log forwarding connections:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
   name: YOUR-SINK
   namespace: YOUR-NAMESPACE
spec:
   type: syslog
   host: YOUR-LOG-DESTINATION
   port: YOUR-LOG-DESTINATION-PORT
   enable_tls: true
   insecure_skip_verify: false
```
  Where:
  - YOUR-SINK-RESOURCE is the sink resource that you want to create. This must be either ClusterLogSink or LogSink. For information about these sink resources, see Overview.
  - YOUR-SINK is a name that you choose for your sink.
  - YOUR-NAMESPACE is the name of your namespace. Omit this line if you are creating ClusterLogSink.
  - YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  - YOUR-LOG-DESTINATION-PORT is the port number of your log management service.
  Note: enable_tls must be true.
- To use a custom CA certificate to secure log forwarding connections:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
   name: YOUR-SINK
   namespace: YOUR-NAMESPACE
spec:
   type: syslog
   host: YOUR-LOG-DESTINATION
   port: YOUR-LOG-DESTINATION-PORT
   enable_tls: true
   insecure_skip_verify: false
   fluent_bit_ca_cert: YOUR-CA-CERT
```
  Where:
  - YOUR-SINK-RESOURCE is the sink resource that you want to create. This must be either ClusterLogSink or LogSink. For information about these sink resources, see Overview.
  - YOUR-SINK is a name that you choose for your sink.
  - YOUR-NAMESPACE is the name of your namespace. Omit this line if you are creating ClusterLogSink.
  - YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  - YOUR-LOG-DESTINATION-PORT is the port number of your log management service.
  - YOUR-CA-CERT is your custom CA certificate to secure the syslog connection. The custom CA certificate must include a SAN field. If the certificate does not include a SAN field, Fluent Bit will not send logs.
  Note: enable_tls must be true.
(Optional) To filter the logging output, include a filters section in your configuration. For more information, see Define a Filter for LogSink and ClusterLogSink Resources below.
Save the YAML file with an appropriate file name. For example, my-cluster-log-sink.yml.
Apply the ClusterLogSink or LogSink resource to your cluster by running the following command:
```
kubectl apply -f YOUR-SINK.yml
```
Where YOUR-SINK.yml is the name of your YAML file.

For example:
```
$ kubectl apply -f my-cluster-log-sink.yml
```

Create a Webhook ClusterLogSink or LogSink Resource

ClusterLogSink and LogSink resources of type webhook batch logs into one-second units, wrap the resulting payload in JSON, and use the POST method to deliver the logs to the address of your log management service.

To define a webhook ClusterLogSink or LogSink resource, perform the following steps:

Create a YAML file that specifies your log destination in the following format:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
  name: YOUR-SINK
  namespace: YOUR-NAMESPACE
spec:
  type: webhook
  url: YOUR-LOG-DESTINATION
```
Where:
- YOUR-SINK-RESOURCE is the sink resource you want to create. This must be either ClusterLogSink or LogSink. For information about these sink resources, see Overview.
- YOUR-SINK is a name you choose for your sink.
- YOUR-NAMESPACE is the name of your namespace. Omit this line if you are creating ClusterLogSink.
- YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
(Optional) To filter the logging output, include a filters section in your configuration. For more information, see Define a Filter for LogSink and ClusterLogSink Resources below.
Save the YAML file with an appropriate filename. For example, my-cluster-log-sink.yml.
Apply the ClusterLogSink or LogSink resource to your cluster by running the following command:
```
kubectl apply -f YOUR-SINK.yml
```
Where YOUR-SINK.yml is the name of your YAML file.

For example:
```
$ kubectl apply -f my-cluster-log-sink.yml
```

Create a ClusterLogSink or LogSink Resource with a Fluent Bit Output Plugin

ClusterLogSink and LogSink resources with a Fluent Bit output plugin deliver logs to the output plugin that you specify in your resource configuration.

To define a ClusterLogSink or LogSink resource with a Fluent Bit output plugin, perform the following steps:

Create a YAML file that specifies your log destination in the following format:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
  name: YOUR-SINK
  namespace: YOUR-NAMESPACE
spec:
  type: http
  output_properties:
    Host: example.com
    Format: json
    Port: 443
    tls: on
    tls.verify: off
```
Where:
- YOUR-SINK-RESOURCE is the sink resource you want to create. This must be either ClusterLogSink or LogSink. For information about these sink resources, see Overview.
- YOUR-SINK is a name you choose for your log sink.
- YOUR-NAMESPACE is the name of your namespace. Omit this line if you are creating ClusterLogSink.
Note: This is a sample plugin configuration for http. For a full list of supported plugins, see the Fluent Bit documentation.
(Optional) To filter the logging output, include a filters section in your configuration. For more information, see Define a Filter for LogSink and ClusterLogSink Resources below.
Save the YAML file with an appropriate filename. For example, my-cluster-log-sink.yml.
Apply the ClusterLogSink or LogSink resource to your cluster by running the following command:
```
kubectl apply -f YOUR-SINK.yml
```
Where YOUR-SINK.yml is the name of your YAML file.

For example:
```
$ kubectl apply -f my-cluster-log-sink.yml
```

(Optional) Define a Filter for LogSink and ClusterLogSink Resources

You can set filters on your LogSink and ClusterLogSink resources:

You can include or exclude all logs or all events from sink output. For more information, see Exclude Logs or Events from Sink Output below.
If you are using Fluent Bit, you can filter logs using conditions and rules. For more information, see Create a Fluent Bit ClusterLogSink or LogSink Filter below.

Exclude Logs or Events from Sink Output

The LogSink and ClusterLogSink resources allow users to set filters to include or exclude all logs or all events from sink output.

To filter all logs or all events from sink output:

Add a filter properties section to the YAML file that specifies the sink’s log output destination and which types of log entities to include or exclude from the output:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
   name: YOUR-SINK
   namespace: YOUR-NAMESPACE
spec:
   type: syslog
   host: YOUR-LOG-DESTINATION
   port: YOUR-LOG-DESTINATION-PORT
   enable_tls: true
   filters:
    include-events: true
    include-logs: false
```
Where:
- YOUR-SINK-RESOURCE is the sink resource type that you created. This must be either ClusterLogSink or LogSink.
- YOUR-SINK is the name you chose for your sink.
- YOUR-NAMESPACE is the name of your namespace. Omit this line for ClusterLogSink type sink resources.
- YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
- YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

The default values for these filter properties is true. If you do not specify filters properties, both logs and events are included in the sink’s output.

For more information, see Monitoring Clusters with Log Sinks.

Fluent Bit ClusterLogSink or LogSink FilterSpecs

As you consider your sink, you might realize that the sink needs a filter more complex than “all logs” or “all events” or even more complex than a single query. You might even need to route your log output to more than one destination target.

If you are using a Fluent Bit ClusterLogSink or LogSink sink resource, you can define a filterSpec with condition rules for filtering incoming logs. Your Fluent Bit sink resource will selectively direct matching log entries to the output destination for the sink. The output target for a Fluent Bit sink is the Output_properties destination defined for the sink resource.

For example, you might want only the error log entries of your Production Pods routed to a specific service that you are monitoring and for the remaining Production Pod status log entries to be managed separately from your non-production Pod logs. For this scenario, you require the ability to create a complex filtering query and to control the routing destination of filtered log entries.

Support for Multiple Filtering Rules

You can define multiple filtering rules in a filterSpec:

Define multiple condition tests in a filterSpec filter:
- Define filter conditions that filter based on log content.
- Define filter conditions that filter based on the log metadata. You can filter log metadata by: Namespace, Host Name, Container Name, Pod Name, and Pod labels.
When you define multiple condition tests in a filterSpec filter, a log entry is considered a match only if all of the defined conditions in the filter are matched.
Define multiple filters for a filterSpec:

When you define multiple filters in your filterSpec configuration, a log entry is considered a match if one or more of the filters is matched.

Support for Multiple Target Destinations

If you have multiple output target destinations, you need to define a separate sink for each destination. When you have multiple sinks, log entry output is directed to only one destination, no matter how many sinks the log entry is a match for.

Which destination a log entry is routed to depends on which sinks the entry matches:

If one or more of the sinks is configured without a filterSpec, all log entries are directed to the output target of the first sink without a filterSpec.

If all of the sinks are configured with a filterSpec, the log entries are directed to the output target as follows:

Log Entry Match Scenario	Log Entry Target Destination
Matches a single sink	The log entry is directed to the output target destination for the sink.
Matches multiple LogSinks	The log entry is directed to only the output target destination for the first matched LogSink.
Matches multiple ClusterLogSinks	The log entry is directed to only the output target destination for the first matched ClusterLogSink.
Matches both LogSinks and ClusterLogSinks	The log entry is directed to only the output target destination for the first matched LogSink.

Create a Fluent Bit ClusterLogSink or LogSink FilterSpec

To define a filter for a ClusterLogSink or LogSink resource, perform the following steps:

Configure include-logs as true in the filters section of your Sink resource configuration file.
Add a filterSpec section to the filters section in your Sink resource configuration file:
```
  filters:
    include-logs: true
    filterSpec:
      - filter:
          - condition: CONDITION-TYPE
            key: KEY
            value: VALUE
    Output_properties:
      output_target
```
Where:
- CONDITION-TYPE is the type of conditional test to apply to the log entries.
- KEY is the name of the key to validate.
- VALUE is the value of the key to validate.
For more information, see Define a Filter Condition below.
Create additional condition sections for each condition you want to apply to your filter.
Create additional filter sections as needed.

Define a Filter Condition

When you define a Fluent Bit Sink resource filter, you must specify the filter’s condition type, key, and value:

The filter condition type is the type of rule to apply to the filter. For example, common condition types include key_value_equals and key_value_does_not_equal.
The filter key indicates which property in the log entry JSON to test against.
The filter value is the value of the key to test for.

To build a filter for your filterSpec, see:

Chose a Filter Condition Type
Create a Filter Key

For example, consider a situation where you wish to include only production Pod log entries in your logs. If the administrator has tagged production Pods as a production, the output JSON for error log entries might have a format similar to the following:

"log":"[ERROR]counter 2: 2439 Wed Feb 23 06:51:08 UTC 2022",
"kubernetes":
{
    "pod_name":"counter-2",
    "namespace_name":"ns-1",
    "pod_id":"05184f35-44ba-45ed-8f75-5016321619ce",
    "labels":{"environment":”production"},
    ...

All of the following example filter settings for condition, key, and value match the example JSON-structured log entry above:

Match against error condition:
- condition: key_value_matches
- key: log
- value: ^/[ERROR/]
Match against pod_id:
- condition: key_value_equals
- key: $kubernetes[‘pod_id’]
- value: 05184f35-44ba-45ed-8f75-5016321619ce
Match against environment = production:
- condition: key_value_equals
- key: $kubernetes[‘labels’][‘environment’]
- value: production

The resulting filter would be written as:

  filters:
    include-logs: true
    filterSpec:
      - filter:
        - condition: key_value_matches
          key: log
          value: ^/[ERROR/]
      - filter:
        - condition: key_value_equals
          key: $kubernetes[‘pod_id’]
          value: 05184f35-44ba-45ed-8f75-5016321619ce
      - filter:
        - condition: key_value_equals
          key: $kubernetes[‘labels’][‘environment’]
          value: production

Chose a Filter Condition Type

The following are the supported filter condition types supported by TKGI:

Condition	Key	Value	Description
`key_exists`	string:key	none	`true` if key exists
`key_does_not_exist`	string:key	none	`true` if key does not exist
`a_key_matches`	regexp:key	none	`true` if a key matches regex
`no_key_matches`	regexp:key	none	`true` if not keys match regex
`key_value_equals`	string:key	string:value	`true` if key exists and its value equals value
`key_value_does_not_equal`	string:key	string:value	`true` if key exists and its value does not equal value
`key_value_matches`	string:key	regexp:value	`true` if key exists and its value matches value
`key_value_does_not_match`	string:key	regexp:value	`true` if key exists and its value does not match value
`matching_keys_have_matching_values`	regexp:key	regexp:value	`true` if all keys matching key have values matching value
`matching_keys_do_not_have_matching_values`	regexp:key	regexp:value	`true` if all keys matching key do not have values matching value

Create a Filter Key

The filter key follows the pattern:

          key: $ROOTNODE-NAME[OBJECT-NAME][PARAMETER-NAME]`

Where:

ROOTNODE-NAME is the name of the root node in the JSON log entry.
OBJECT-NAME is name of a child node in the root node.
PARAMETER-NAME is name of a parameter in the child node.

For an examples of how to create a Filter Key, see Define a Filter Condition above.

(Optional) Unsecured ClusterLogSink and LogSink Log Forwarding

By default, TKGI uses a secure connection for log forwarding when using ClusterLogSink and LogSink resources of type syslog or webhook.

For debugging purposes on a local machine, you might want to temporarily forward logs using an unsecured connection. To do this, you must:

Deactivate sink forwarding validation by running the following command:
```
kubectl delete validatingwebhookconfigurations validator.pksapi.io
```
Set enable_tls to false in your log destination YAML file.

Warning: Deactivating secure log forwarding is not recommended.

Create ClusterMetricSink and MetricSink Resources

ClusterMetricSink and MetricSink resources collect metrics from different sources:

ClusterMetricSink resources collect metrics from a cluster. If you want to collect pod usage metrics, you have to use a ClusterMetricSink.
MetricSink resources collect metrics from a namespace within a cluster. If you want to isolate a certain workload’s metrics to its own output, you have to use a namespaced MetricSink.

ClusterMetricSink Resources

How It Works

By default, a ClusterMetricSink resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs that you specify in your ClusterMetricSink configuration:

ClusterMetricSink DaemonSet collect pod usage metrics using Telegraf agents running on every node in a cluster.
The ClusterMetricSink is able to fetch pod metrics using the Kubernetes input plugin, which gives access to pod usage metrics for each kubernetes node. Pod metrics come from the underlying container runtime, which does not isolate metrics based on namespace.

Do not use a ClusterMetricSink to isolate a specific workload’s metrics to its own output. A ClusterMetricSink cannot isolate one input from all the others because all of a ClusterMetricSink’s configurations are located in a shared ConfigMap. This means that each ClusterMetricSink’s input will also go to all other ClusterMetricSinks’ outputs.

MetricSink Resources

A MetricSink resource runs in a single node only and collects metrics from a namespace within a cluster using prometheus.io/scrape annotations set to true. A MetricSink creates a unique Telegraf agent pod and ConfigMap in the namespace, then writes the metrics to one or more outputs that you specify in your MetricSink configuration.

Do not use MetricSinks to collect pod usage metrics. Pod metrics come from the underlying container runtime, which does not isolate metrics based on namespace, so only ClusterMetricSink is able to fetch pod metrics.

For a list of supported output plugins, see Output Plugins in the Telegraf GitHub repository.

When to Use MetricSink vs. ClusterMetricSink

Case 1. Isolating Output => MetricSink

If you want to isolate a certain workload’s metrics to its own output, you have to use a namespaced MetricSink. A MetricSink creates a unique TSelegraf agent pod and ConfigMap in the namespace.

ClusterMetricSinks cannot isolate one input from all the others because all ClusterMetricSinks’ configurations are located in a shared ConfigMap. This means that each ClusterMetricSink’s inputs will also go to all other ClusterMetricSinks’ outputs.

Note that MetricSinks do not have access to pod usage metrics, as the source of those metrics does not filter by namespace and gives usage for all pods on a node.

Case 2. Pod Usage Metrics (CPU, Memory) => ClusterMetricSink

If you want to get pod usage metrics, you have to use a ClusterMetricSink.

To get pod usage metrics, Telegraf agents need to run on every node in a cluster, and this is what the ClusterMetricSink DaemonSet does.

Pod metrics come from the underlying container runtime, which does not isolate metrics based on namespace, so only ClusterMetricSink is able to fetch pod metrics.

ClusterMetricSinks use the Kubernetes input plugin, which gives access to pod usage metrics for each kubernetes node. MetricSinks only run in a single node, so they do not have access to pod usage metrics.

Create a ClusterMetricSink or MetricSink Resource

To define a ClusterMetricSink or MetricSink resource, perform the following steps:

Create a YAML file in the following format:
```
apiVersion: pksapi.io/v1beta1
kind: YOUR-SINK-RESOURCE
metadata:
  name: YOUR-SINK
  namespace: YOUR-NAMESPACE
spec:
  inputs:
  outputs:
  - type: YOUR-OUTPUT-PLUGIN
telegraf_agent_config:
  interval:  INTERVAL-LENGTH
```
Where:
- YOUR-SINK-RESOURCE is the sink resource you want to create. This must be either ClusterMetricSink or MetricSink. For information about these sink resources, see Overview.
- YOUR-SINK is a name you choose for your sink.
- YOUR-NAMESPACE is the name of your namespace. Omit this line if you are creating ClusterMetricSink.
- YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.
- (Optional) INTERVAL-LENGTH is the interval, in seconds, separating sink collection of input data. The assigned INTERVAL-LENGTH must be a positive integer less than 9223372037. The default is 10.
  
  Note: You can leave the inputs field blank. For ClusterMetricSink, this field is configured to include metrics from the kubelet by default. For MetricSink, the field includes all prometheus.io/scrape annotations set to true by default.
For example:
```
apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
  name: http
spec:
  inputs:
  outputs:
  - type: http
    url: https://example.com
    method: POST
    data_format: json
```
This will send all cluster metrics provided by the kubernetes input plugin via json POST to https://example.com

Create a ClusterMetricSink Resource for Node Exporter Metrics

To define a ClusterMetricSink resource for collecting Node Exporter metrics, perform the following steps:

Activate Node Exporter on your cluster workers by selecting the Enable node exporter on workers check box in the Tanzu Kubernetes Grid Integrated Edition tile > In-Cluster Monitoring.

Create a YAML file in the following format:

apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
  name: YOUR-SINK
spec:
  inputs:
  - monitor_kubernetes_pods: true
    type: prometheus
  outputs:
  - type: YOUR-OUTPUT-PLUGIN

Where:

YOUR-SINK is a name you choose for your sink.
YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

For example:

apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
  name: http
spec:
  inputs:
  - monitor_kubernetes_pods: true
    type: prometheus
  outputs:
  - type: http
    url: https://example.com
    method: POST
    data_format: json

Save the YAML file with an appropriate filename. For example, my-cluster-metric-sink.yml.
Apply the ClusterMetricSink resource to your cluster by running the following command:
```
kubectl apply -f YOUR-SINK.yml
```
Where YOUR-SINK.yml is the name of your YAML file.

For example:
```
$ kubectl apply -f my-cluster-metric-sink.yml
```

List Sinks

To list sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink and LogSink Resources

To list cluster log sinks, run the following command:

kubectl get clusterlogsinks

To list namespace log sinks, run the following command:

kubectl -n YOUR-NAMESPACE get logsinks

Where YOUR-NAMESPACE is the name of your namespace.

ClusterMetricSink and MetricSink Resources

To list cluster metric sinks, run the following command:

kubectl get clustermetricsinks

To list namespace metric sinks, run the following command:

kubectl -n YOUR-NAMESPACE get metricsinks

Where YOUR-NAMESPACE is the name of your namespace.

Delete Sinks

To delete sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink and LogSink Resources

To delete a cluster log sink, run the following command:

kubectl delete clusterlogsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

To delete a namespace log sink, run the following command:

kubectl -n YOUR-NAMESPACE delete logsink YOUR-SINK

Where:

YOUR-NAMESPACE is the name of your namespace.
YOUR-SINK is the name of your log sink.

ClusterMetricSink and MetricSink Resources

To delete a cluster metric sink, use the following command:

kubectl delete clustermetricsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

To delete a namespace metric sink, use the following command:

kubectl -n YOUR-NAMESPACE delete metricsink YOUR-SINK

Where:

YOUR-NAMESPACE is the name of your namespace.
YOUR-SINK is the name of your metric sink.