This topic describes the log sink resources you can use to monitor VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) provisioned Kubernetes clusters and their workloads.
You can use the following sink resources to collect logs from your Kubernetes clusters.
Sink Resource | Sink Type | Description |
---|---|---|
ClusterLogSink |
Log sink | Forwards logs from a cluster to a log destination. Logs are transported using one of the following:
|
LogSink |
Log sink | Forwards logs from a namespace within a cluster to a log destination. Logs are transported using one of the following:
|
ClusterLogSink
and LogSink
resources collect pod logs and events from the Kubernetes API in your Kubernetes clusters. For more information, see:
In Tanzu Kubernetes Grid Integrated Edition, you can create ClusterLogSink
and LogSink
resources of the following types:
Your log format depends on the type of ClusterLogSink
or LogSink
you want to use. For example, if you use a ClusterLogSink
or LogSink
resource of type syslog
, Tanzu Kubernetes Grid Integrated Edition formats your logs as described in the sections below.
All log entries collected by ClusterLogSink
and LogSink
resources of type syslog
include a prefix in the following format:
APP-NAME/NAMESPACE/POD-ID
Where:
APP-NAME
is pod.log
or k8s.event
.NAMESPACE
is the namespace associated with the pod log or Kubernetes event.POD-ID
is the ID of the pod associated with the pod log or Kubernetes event.Pod logs are distinguished by the string pod.log
in the APP-NAME
field.
The following is a sample pod log entry:
36 <14>1 2018-11-26T18:51:41.647825+00:00 cluster-name
pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450
app="logspewer" pod-template-hash="2614622458" namespace_name="rocky-raccoon"
object_name="logspewer-6b58b6689d-dhddj" container_name="logspewer"]
2018/11/26 18:51:41 Log Message 589910
Where:
cluster-name
is the human-readable cluster name used when creating the cluster.pod.log
is the APP-NAME
.rocky-raccoon
is the NAMESPACE
.logspewer-6b58b6689d-dhddj
is the POD-ID
.Kubernetes API events are distinguished by the string k8s.event
in the APP-NAME
field.
The following is an example Kubernetes API event log entry:
Nov 14 16:01:49 cluster-name
k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n:
Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7
to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1
Where:
cluster-name
is the human-readable cluster name used when creating the cluster.k8s.event
is the APP-NAME
.rocky-raccoon
is the NAMESPACE
.logspewer-6b58b6689d-j9n
is the POD-ID
.The following section lists Kubernetes API events that can help assess Kubernetes scheduling problems in Tanzu Kubernetes Grid Integrated Edition.
To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.
ImagePullBackOff |
|
---|---|
Description | Image pull back offs occur when the Kubernetes API cannot reach a registry to retrieve a container or the container does not exist in the registry. The scheduler might be trying to access a registry that is not available on the network. For example, Docker Hub is blocked by a firewall. Other reasons might include the registry is experiencing an outage or a specified container has been deleted or was never uploaded. |
Identifying String | Error:ErrImagePull |
Example Sink Log Entry | Jan 25 10:18:58 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test-669d4d66b9-zd9h4/: Error: ErrImagePull |
CrashLoopBackOff |
|
---|---|
Description | Crash loop back offs imply that the container is not functioning as intended. There are several potential causes of crash loop back offs, which depend on the related workload. To investigate further, examine the logs for that workload. |
Identifying String | Back-off restarting failed container |
Example Sink Log Entry | Jan 25 09:26:44 cluster-name k8s.event/monitoring/cost-analyzer-prometheus-se: Back-off restarting failed container |
ContainerCreated |
|
---|---|
Description | Operators can monitor the creation and successful start of containers to keep track of platform usage at a high level. Cluster users can track this event to monitor the usage of their cluster. |
Identifying String | Started container |
Example Sink Log Entries | Jan 25 09:14:55 cluster-name 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d/: Created pod: logspewer-6b58b6689d-sr96t |
FailedScheduling |
|
---|---|
Description | This event occurs when a container cannot be scheduled. For instance, this might occur due to lack of node resources. |
Identifying String | Insufficient RESOURCE where RESOURCE is a specific type of resource. For example, cpu . |
Example Sink Log Entries | Jan 25 10:51:48 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test2-5c87bf4b65-7fdtd/: 0/1 nodes are available: 1 Insufficient cpu. |
For more information about log sinks, see:
Creating and Managing Sink Resources.
Follow these instructions to create ClusterLogSink
and LogSink
resources, described in Overview above.
Sink Architecture in Tanzu Kubernetes Grid Integrated Edition.
See this topic for conceptual information about sinks.