Fine-tune Syslog on ESXi Hosts

How to Specify Log File Retention

By default, log files cannot expand past a configured size. Once a log file reaches the configured size, logging is routed to a new log file and the oldest log file is deleted.

Note: Best practice is to balance the rotate and size settings. Increasing the rotate setting ensures that syslog files are generated often enough to prevent any potential corruption or destruction from the other log files. Increasing the size setting reduces the time for switching to another log file. Optimal size settings are a multiple of 1024 KiB.

Use the Syslog.global.defaultSize setting to specify the log file maximum size in KiB, and Syslog.global.defaultRotate to set the maximum number of old log files to keep before rotating to a new log file. To change the log file retention parameters associated with a specific program, use the Syslog.loggers.<progName>.rotate and Syslog.loggers.<progName>.size settings, where <progName> is the name of the program whose parameters you want to adjust.

Manage Settings that Affect the Virtual Machine Log File

You can configure some settings that affect the virtual machine log file, vmware.log, either in the vmx file or in the /etc/vmware/config file. You must power off a virtual machine to edit the vmx file and edits take effect only on that virtual machine. If you use the /etc/vmware/config file, you must add the prefix "vmx" to the setting, for example vmx.log.keepOld = "20", and edits affect all virtual machines on the ESXi host.

Table 1. Configurable settings for the vmware.log file
Parameter	Description	Example	Notes
logging	Disables all virtual machine logging.	The default value is `logging = "TRUE"` To disable virtual machine logging: `logging = "FALSE"`	Do not use this setting, because disabling virtual machine logging makes it difficult or impossible to get support for virtual machine problems. If you need to use this setting for some reason, you can only place it in the vmx file of a virtual machine.
log.throttleBytesPerSec	Controls when a log file throttles. Log file throttling occurs when writes to the vmware.log exceed the specified rate for a significant amount of time. This occurs when code within the VMX process, which controls a virtual machine, creates excessive log messages. The default value for this setting is 1 KB/sec. In case of log thottling, you see `<<< Log Throttled >>>` in the vmware.log file.	`log.throttleBytesPerSec = "1500"` To disable log throttling, use `log.throttleBytesPerSec = "0xFFFFFFFF"`	Log file thottling might obscure information necessary to diagnose problems with the affected virtual machine. If you need to disable log throttling, place the line in the example in the vmx file of the affected virtual machine. Remove the line after the debugging session ends.
log.keepOld	Controls the number of older vmware.log file to retain.	`log.keepOld = "20"`	Do not put the value of this setting below the default value (10). If virtual machines are frequently modified or moved, consider raising this setting to 20 or more.
log.rotateSize	Controls the maximum size of a vmware.log file in bytes.	`log.rotateSize = "2500000"` To disable limiting the maximum size of a vmware.log file, use `log.rotateSize = "0"`	A value of this setting below 100,000 can cause a loss of critical log messages and affect virtual machine performance. In ESXi 7.x and earlier, the default value of this setting places no limit on the size of a vmware.log file. In ESXi 8.x and later, the default value of this setting is 2,048,000.
log.fileName	Controls the name and location of virtual machine log files.	`log.fileName = "myVMLog"` This setting changes the name of the virtual machine log files from vmware.log to `myVMlog`. `log.fileName = "/vmfs/volumes/vol1/myVM/myVM.log"` This setting directs virtual machine log files to a directory on a different VMFS volume (vol1) by using `myVM` for a file name.	Do not place a log file outside the virtual machine directory to make sure that the collection of host support bundles picks up the log file, which can be critical to debug virtual machine problems.
log.fileLevel	Controls the minimum level at which messages are written to vmware.log. Every log message has a level associated with it. Levels below the specified setting are not added to a virtual machine log file. The virtual machine message log levels (from most to least restricted) are: error warning notice info (default) trivia debug debug1 debug2 debug3 debug4 debug5 debug6 debug7 debug8 debug9 debug10	`log.fileLevel = "debug1"`	Do not set a more restrictive level than "info" to avoid filtering out messages that are necessary for debugging virtual machine problems. Lower the level below "info" only upon request by licensed support. Restore the setting to "info" after debugging ends.
log.filter.minLogLevel.<groupName>	Controls the output of specialized debugging messages.	`log.filter.minLogLevel.disklib = "debug5"`	Use this setting only upon request by licensed support, who should provide one or more <groupName> parameters. Remove the setting after debugging ends.
log.sysogID	Activates the sending of virtual machine log messages to the system logger of an ESXi host, such as the syslog.	`log.syslogID = "vmx"`	Use "vmx" as value for this setting to allow the ESXi syslog daemon, vmsyslogd, to send these messages to a separate log file.
log.syslogLevel	Controls the minimum level at which messages are output to the system logger of an ESXi host, such as the syslog.	`log.syslogLevel = "debug"`	The levels and functioning of this setting are identical to those for the log.fileLevel setting.

How to Specify Message Transmission to Remote Hosts

Optionally, you can configure ESXi to send syslog messages to one or more remote hosts, called syslog collectors, such as VMware Aria Operations for Logs (formerly VMware vRealize Log Insight and vCenter Log Insight), to collect syslog messages.

Note: Best practice is that you configure each ESXi host to send syslog messages to at least one syslog collector. This helps ensure that the messages are preserved in case of a catastrophic system event and that you can process syslog messages in various ways, such as real-time categorization and analysis (for example, by type, time span, or machine), or archive messages.

Use the Syslog.global.logHost setting to define remote host specifications. Separate multiple remote host specifications with a comma (,). After setting Syslog.global.logHost, ESXi hosts open and maintain connections to the syslog collectors, and the transmission of messages begins immediately. When ESXi generates a syslog message, it writes it to the appropriate log file on the ESXi host and also forwards it to all configured syslog collectors.

In addition to syslog messages, audit messages can also be transmitted to syslog collectors for security purposes. Audit records track security-related activity on the ESXi host. For more information about audit records, see Audit Records.

Note: Consult with your company security response team if and how to set audit records. Certified configurations usually require audit records to be activated.

What follows is the syntax for Syslog.global.logHost remote host specifications:

protocol://target[:port]


Parameter	Description	Notes
protocol	Specifies the networking protocol. Valid values are udp, tcp, and ssl.	The ssl protocol specifies that transmission of syslog messages is encrypted. The tcp and udp protocols do not encrypt the transmission. Note: If capturing syslog messages or audit messages is critical to your system, avoid using the udp protocol because the networking infrastructure external to ESXi might drop UDP messages.
target	Specifies the remote host. You can use either an IPV4 or IPV6 address, or a host name.	When you use an IPV6 address, you must embed it in square brackets [xxx], where xxx is the IPV6 address.
port	(Optional) Specifies the remote host port to use. If you use UDP or TCP, the default port is 514. If you use SSL, the default port is 1514. If you choose to use different ports from 514 or 1514, you must adjust the ESXi firewall to open the port.	For details how to open the ESXi firewall for the port specified in each remote host specification, see Configuring the ESXi Firewall.

Еxamples of remote machine specifications:


Syslog.global.logHost string example	Notes
tcp://10.176.130.7:12345	Transmits syslog messages to 10.176.130.7 using TCP/IP and port 12345.
tcp://[2001:db8:85a3:8d3:1319:8a2e:370:7348]	Transmits syslog messages to an IPV6 address using port 1514.
tcp://[2001:db8:85a3:8d3:1319:8a2e:370:7348]:5432	Transmit syslog messages to an IPV6 address using port 54321.
udp://company.com	Transmits syslog messages to company.com using UDP and port 514.
udp://company.com,tcp://10.20.30.40:1050	Transmits syslog messages to two remote hosts. The first remote host uses UDP to communicate with company.com using port 514. The second remote host uses TCP to communicate with the IPV4 address 10.20.30.40 using port 1050.
ssl://company.com	Transmits syslog messages to company.com using SSL (TLS) and port 514.

Maximum Message Transmission Length

If you use UDP, the maximum syslog message transmission length is 480 bytes for IPV4 and 1180 bytes for IPV6.

For TCP or SSL, the default maximum syslog message transmission length is 1 kibibyte (KiB). You can increase this length by using the Syslog.global.remoteHost.maxMsgLen parameter. The maximum value is 16 KiB. Messages longer than 16 KiB are truncated.

Note: If increasing the maximum transmission length is necessary, best practice is to increase the length only as much as specifically necessary.

Increasing the maximum syslog message length can cause problems if the networking and syslog infrastructure external to ESXi is unable to handle messages longer than 1 KiB.

Note: Best practice is that you do not use UDP to transmit syslog messages due to the packet length constraints and the possibility that the external networking infrastructure might drop the messages.

Considerations for Certificates When Configuring SSL Transmissions to Remote Hosts

When configuring ESXi to transmit syslog messages to remote hosts using SSL, you must add an SSL certificate for each remote host to the ESXi host CA store. For more information, see Certificate Management for ESXi Hosts and Manage CA Certificates with ESXCLI.

Note: Consult with your syslog collector documentation on how to configure the collector for secure receipt of syslog messages using SSL and a private key.

Additional SSL Transmission Parameters

An ESXi system complying with security certification requirements might require activating X509 CRL checks. You turn on the advanced settings Syslog.global.certificate.strictX509Compliance and Syslog.global.certificate.checkCRL by changing the default value of false to true. Due to implementation limitations, if you activate CRL checks by using the setting Syslog.global.certificate.checkSSLCerts, then all certificates in a certificate chain must provide a CRL link. By default, the setting is active. You can deactivate SSL certificate checks by changing the setting to false, but this is not a best practice. You might need to turn off SSL certificate checks when troubleshooting communications with a remote host, but do this only for a limited time.

Where to Find Syslog Daemon Error and Status Information

The ESXi syslog daemon uses the log file /var/run/log/vmsyslogd.log to store status and error information, including dropped messages. If audit record transmission is active, the syslog daemon also emits audit records related to its operation, such as daemon start, stop, and error conditions, which allows you to verify that the syslog daemon runs properly.

How to Change the Default Syslog Log File Storage Area

The default syslog log file storage area is /var/run/log, local to each ESXi host. Use the Syslog.global.logDir syslog configuration variable to change the default syslog log file storage area, as long as the location resides on persistent storage. If Syslog.global.logDir is configured to a persistent store shared by multiple ESXi hosts to store their syslog log files, change the Syslog.global.logDirUnique setting to true to prevent mixing logs. The Syslog.global.logDirUnique setting makes sure that each ESXi machine gets a unique name added to the Syslog.global.logDir path, separating the log files from other hosts.

Syslog Message Queueing for Remote Hosts and Message Drops

Once syslog emissions start, they never stop except for ESXi reboots and failures, or a syslog reconfiguration to stop.

To avoid dropping messages, ESXi uses an in-memory queue that allows the vmsyslogd service to handle the following conditions for a short time:

ESXi generates log messages at a faster rate than the vmsyslogd service can process and transmit
Network connectivity between ESXi and the remote host fails

If either of these conditions continues for a long period, the capacity of the in-memory queue might not be sufficient and vmsyslog might stop transmitting messages and audit logs to remote hosts. No data is lost, because dropped messages are written to the /var/run/log/vmsyslogd-dropped.log file.

To minimize the chance that message dropping occurs, place the vmsyslogd log files on the fastest available storage and configure vmsyslogd and its syslog collectors on a network with end-to-end bandwidth greater than 1 GigE.

Note: Consider an end-to-end bandwidth of 2.5 GigE or more, preferably 10 GigE, to optimize performance and prevent message drops.

If you see excessive logs from an application, file a support service request for analysis and correction.

You can see statistics about message drops in the syslog daemon log file.

You can see dropped messages at /var/run/log/vmsyslogd-dropped.log. This log file has retention settings specific to it, similar to those for the program-specific retention parameters. The dropped messages log file retention parameters are: Syslog.global.droppedMsgs.fileRotate and Syslog.global.droppedMsgs.fileSize.