This topic describes the log sink resources you can use to monitor Kubernetes clusters provisioned by VMware Tanzu Kubernetes Grid Integrated Edition and their workloads.

Overview

You can use the following sink resources to collect logs from your Kubernetes clusters.

Sink Resource Sink Type Description
ClusterLogSink Log sink

Forwards logs from a cluster to a log destination. Logs are transported using one of the following:

  • The Syslog Protocol defined in RFC 5424
  • WebHook
  • Fluent Bit output plugins
LogSink Log sink

Forwards logs from a namespace within a cluster to a log destination. Logs are transported using one of the following:

  • The Syslog Protocol defined in RFC 5424
  • WebHook
  • Fluent Bit output plugins

Log Sinks

ClusterLogSink and LogSink resources collect pod logs and events from the Kubernetes API in your Kubernetes clusters. For more information, see:

Log Format

In Tanzu Kubernetes Grid Integrated Edition, you can create ClusterLogSink and LogSink resources of the following types:

  • Syslog
  • WebHook
  • Fluent Bit output plugins

Your log format depends on the type of ClusterLogSink or LogSink you want to use. For example, if you use a ClusterLogSink or LogSink resource of type syslog, Tanzu Kubernetes Grid Integrated Edition formats your logs as described in the sections below.

Syslog Format

All log entries collected by ClusterLogSink and LogSink resources of type syslog include a prefix in the following format:

APP-NAME/NAMESPACE/POD-ID

Where:

  • APP-NAME is pod.log or k8s.event.
  • NAMESPACE is the namespace associated with the pod log or Kubernetes event.
  • POD-ID is the ID of the pod associated with the pod log or Kubernetes event.

Pod Logs

Pod logs are distinguished by the string pod.log in the APP-NAME field.

The following is a sample pod log entry:

 36 <14>1 2018-11-26T18:51:41.647825+00:00 cluster-name pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450 app=“logspewer” pod-template-hash=“2614622458” namespace_name=“rocky-raccoon” object_name=“logspewer-6b58b6689d-dhddj” container_name=“logspewer”] 2018/11/26 18:51:41 Log Message 589910

Where:

  • cluster-name is the human-readable cluster name used when creating the cluster.
  • pod.log is the APP-NAME.
  • rocky-raccoon is the NAMESPACE.
  • logspewer-6b58b6689d-dhddj is the POD-ID.

Kubernetes API Events

Kubernetes API events are distinguished by the string k8s.event in the APP-NAME field.

The following is an example Kubernetes API event log entry:

 Nov 14 16:01:49 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n: Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7 to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1

Where:

  • cluster-name is the human-readable cluster name used when creating the cluster.
  • k8s.event is the APP-NAME.
  • rocky-raccoon is the NAMESPACE.
  • logspewer-6b58b6689d-j9n is the POD-ID.

Notable Kubernetes API Events

The following section lists Kubernetes API events that can help assess Kubernetes scheduling problems in Tanzu Kubernetes Grid Integrated Edition.

To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.

Failure to Retrieve Containers from Registry


ImagePullBackOff
Description Image pull back offs occur when the Kubernetes API cannot reach a registry to retrieve a container or the container does not exist in the registry. The scheduler might be trying to access a registry that is not available on the network. For example, Docker Hub is blocked by a firewall. Other reasons might include the registry is experiencing an outage or a specified container has been deleted or was never uploaded.
Identifying String Error:ErrImagePull
Example Sink Log Entry Jan 25 10:18:58 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test-669d4d66b9-zd9h4/: Error: ErrImagePull

Malfunctioning Containers


CrashLoopBackOff
Description Crash loop back offs imply that the container is not functioning as intended. There are several potential causes of crash loop back offs, which depend on the related workload. To investigate further, examine the logs for that workload.
Identifying String Back-off restarting failed container
Example Sink Log Entry Jan 25 09:26:44 cluster-name k8s.event/monitoring/cost-analyzer-prometheus-se: Back-off restarting failed container

Successful Scheduling of Containers


ContainerCreated
Description Operators can monitor the creation and successful start of containers to keep track of platform usage at a high level. Cluster users can track this event to monitor the usage of their cluster.
Identifying String Started container
Example Sink Log Entries Jan 25 09:14:55 cluster-name 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d/: Created pod: logspewer-6b58b6689d-sr96t
Jan 25 09:14:55 cluster-name 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Successfully assigned rocky-raccoon/ logspewer-6b58b6689d-sr96t to vm-efe48928-be8e-4db5-772c-426ee7aa52f2
Jan 25 09:14:55 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-mkd: Killing container with id docker://logspewer:Need to kill Pod
Jan 25 09:14:56 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Container image “oratos/logspewer:v0.1” already present on machine
Jan 25 09:14:56 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Created container
Jan 25 09:14:56 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Started container

Failure to Schedule Containers


FailedScheduling
Description This event occurs when a container cannot be scheduled. For instance, this may occur due to lack of node resources.
Identifying String Insufficient RESOURCE
where RESOURCE is a specific type of resource. For example, cpu.
Example Sink Log Entries Jan 25 10:51:48 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test2-5c87bf4b65-7fdtd/: 0/1 nodes are available: 1 Insufficient cpu.

Related Links

For more information about log sinks, see:

check-circle-line exclamation-circle-line close-line
Scroll to top icon