Checkout how to configure the streaming of logs from the Supervisor Cluster control plane VMs to a remote rsyslog receiver so that to avoid loosing valuable logging data.

Logs generated by the components in the Supervisor Cluster control plane VMs are stored locally in the file systems of the VMs. When a large amount of logs is accumulated, the logs are rotated at high rate, which leads to losing valuable messages that might help with identifying the root cause of different issues. vCenter Server and the Supervisor Cluster control plane VMs support streaming their local logs to a remote rsyslog receiver. This feature helps capture logs for the following services and components:

  • On vCenter Server: Workload Control Plane service, ESX Agent Manager service, Certificate Authority service, and all other services running in vCenter Server.

  • Supervisor Cluster control plane components and Supervisor Cluster embedded services, such as the VM service, and Tanzu Kubernetes Grid Service.

You can configure the vCenter Server appliance to collect and stream local log data to a remote rsyslog receiver. Once this configuration is applied to vCenter Server, the rsyslog sender running inside vCenter Server starts sending logs generated by services inside that vCenter Server system.

Supervisor Cluster uses the same mechanism as vCenter Server to offload local logs to reduce configuration management overhead. The Workload Control Plane service monitors the vCenter Server rsyslog configuration by polling logs periodically. If the Workload Control Plane service detects that the remote vCenter Server rsyslog configuration is not empty, the service propagates this configuration to each control plane VM in all Supervisor Clusters. This can generate a very large amount of rsyslog message traffic that can overwhelm the remote rsyslog receiver. Therefore, the receiver machine must have sufficient storage capacity to sustain large amounts of rsyslog messages.

Removing the rsyslog configuration from vCenter Server stops rsyslog messages from vCenter Server. The Workload Control Plane service detects the change and propagates it to each control plane VM in every Supervisor Cluster, eventually stopping the control plane VM streams too.

Configuration Steps

Take the following steps to configure rsyslog streaming for Supervisor Cluster control plane VMs:

  1. Configure an rsyslog receiver by provisioning a machine that:
    • Runs the rsyslog service in receiver mode. See the Receiving massive amounts of messages with high performance example from the rsyslog documentation.
    • Has sufficient storage space to accommodate large amounts of log data.
    • Has network connectivity to receive data from vCenter Server and the Supervisor Cluster control plane VMs.
  2. Log in to the vCenter Server appliance management interface at https://<vcenter server address>:5480 as an root.
  3. Configure vCenter Server to stream to rsyslog receiver through the vCenter Server appliance management interface. See Forward vCenter Server Log Files to Remote Syslog Server.

It might takes a few minutes for the rsyslog configuration of vCenter Server to be applied to the Supervisor Cluster control plane VMs. The Workload Control Plane service on the vCenter Server appliance polls the appliance configuration every 5 minutes and propagates it to all available Supervisor Clusters. The amount of time needed for the propagation to complete depends on the number of Supervisor Clusters in your environment. In case some of the control plane VMs on the Supervisor Clusters are unhealthy or performing some other operation, the Workload Control Plane service retries applying the rsyslog configuration until it succeeds.

Inspecting Logs of the Control Plane VM Components

The rsyslog of the Supervisor Cluster control plane VMs embeds tags in the log messages that indicate the source component of these log messages.

Log tags Description
vns-control-plane-pods <pod_name>/<instance_number>.log Logs originating from Kubernetes pods in control plane VMs. For example:

vns-control-plane-pods etcd/0.log

or

vns-control-plane-pods nsx-ncp/573.log
vns-control-plane-imc Initial configuration logs from control plane VMs.
vns-control-plane-boostrap Bootstrap logs from control plane deployment of Kubernetes nodes.
vns-control-plane-upgrade-logs Logs from control plane node patches and minor version upgrades.
vns-control-plane-svchost-logs Control plane VM system level service host or agent logs.
vns-control-plane-update-controller Control plane desired state synchronizer and realizer log.
vns-control-plane-compact-etcd-logs Logs for keeping control plane etcd service storage compaction.