This topic describes the log sink resources you can use to monitor Kubernetes clusters provisioned by VMware Tanzu Kubernetes Grid Integrated Edition and their workloads.
Overview
You can use the following sink resources to collect logs from your Kubernetes clusters.
| Sink Resource | Sink Type | Description |
|---|---|---|
ClusterLogSink |
Log sink | Forwards logs from a cluster to a log destination. Logs are transported using one of the following:
|
LogSink |
Log sink | Forwards logs from a namespace within a cluster to a log destination. Logs are transported using one of the following:
|
Log Sinks
ClusterLogSink and LogSink resources collect pod logs and events from the Kubernetes API in your Kubernetes clusters. For more information, see:
Log Format
In Tanzu Kubernetes Grid Integrated Edition, you can create ClusterLogSink and LogSink resources of the following types:
- Syslog
- WebHook
- Fluent Bit output plugins
Your log format depends on the type of ClusterLogSink or LogSink you want to use. For example, if you use a ClusterLogSink or LogSink resource of type syslog, Tanzu Kubernetes Grid Integrated Edition formats your logs as described in the sections below.
Syslog Format
All log entries collected by ClusterLogSink and LogSink resources of type syslog include a prefix in the following format:
APP-NAME/NAMESPACE/POD-ID
Where:
APP-NAMEispod.logork8s.event.NAMESPACEis the namespace associated with the pod log or Kubernetes event.POD-IDis the ID of the pod associated with the pod log or Kubernetes event.
Pod Logs
Pod logs are distinguished by the string pod.log in the APP-NAME field.
The following is a sample pod log entry:
36 <14>1 2018-11-26T18:51:41.647825+00:00 cluster-name pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450 app=“logspewer” pod-template-hash=“2614622458” namespace_name=“rocky-raccoon” object_name=“logspewer-6b58b6689d-dhddj” container_name=“logspewer”] 2018/11/26 18:51:41 Log Message 589910
Where:
cluster-nameis the human-readable cluster name used when creating the cluster.pod.logis theAPP-NAME.rocky-raccoonis theNAMESPACE.logspewer-6b58b6689d-dhddjis thePOD-ID.
Kubernetes API Events
Kubernetes API events are distinguished by the string k8s.event in the APP-NAME field.
The following is an example Kubernetes API event log entry:
Nov 14 16:01:49 cluster-name k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n: Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7 to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1
Where:
cluster-nameis the human-readable cluster name used when creating the cluster.k8s.eventis theAPP-NAME.rocky-raccoonis theNAMESPACE.logspewer-6b58b6689d-j9nis thePOD-ID.
Notable Kubernetes API Events
The following section lists Kubernetes API events that can help assess Kubernetes scheduling problems in Tanzu Kubernetes Grid Integrated Edition.
To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.
Failure to Retrieve Containers from Registry
ImagePullBackOff |
|
|---|---|
| Description | Image pull back offs occur when the Kubernetes API cannot reach a registry to retrieve a container or the container does not exist in the registry. The scheduler might be trying to access a registry that is not available on the network. For example, Docker Hub is blocked by a firewall. Other reasons might include the registry is experiencing an outage or a specified container has been deleted or was never uploaded. |
| Identifying String | Error:ErrImagePull |
| Example Sink Log Entry | Jan 25 10:18:58 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test-669d4d66b9-zd9h4/: Error: ErrImagePull |
Malfunctioning Containers
CrashLoopBackOff |
|
|---|---|
| Description | Crash loop back offs imply that the container is not functioning as intended. There are several potential causes of crash loop back offs, which depend on the related workload. To investigate further, examine the logs for that workload. |
| Identifying String | Back-off restarting failed container |
| Example Sink Log Entry | Jan 25 09:26:44 cluster-name k8s.event/monitoring/cost-analyzer-prometheus-se: Back-off restarting failed container |
Successful Scheduling of Containers
ContainerCreated |
|
|---|---|
| Description | Operators can monitor the creation and successful start of containers to keep track of platform usage at a high level. Cluster users can track this event to monitor the usage of their cluster. |
| Identifying String | Started container |
| Example Sink Log Entries | Jan 25 09:14:55 cluster-name 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d/: Created pod: logspewer-6b58b6689d-sr96t |
Failure to Schedule Containers
FailedScheduling |
|
|---|---|
| Description | This event occurs when a container cannot be scheduled. For instance, this may occur due to lack of node resources. |
| Identifying String | Insufficient RESOURCEwhere RESOURCE is a specific type of resource. For example, cpu. |
| Example Sink Log Entries | Jan 25 10:51:48 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test2-5c87bf4b65-7fdtd/: 0/1 nodes are available: 1 Insufficient cpu. |
Related Links
For more information about log sinks, see:
-
Creating and Managing Sink Resources.
Follow these instructions to create
ClusterLogSinkandLogSinkresources, described in Overview above. -
Sink Architecture in Tanzu Kubernetes Grid Integrated Edition.
See this topic for conceptual information about sinks.
