The Guest Introspection thin agent is installed with VMware Tools™ on each guest virtual machine.
Troubleshooting the Thin Agent on Linux
If a virtual machine is slow in reading and writing operations, and unzipping or saving files then there might be problems with the thin agent.
- Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
- Ensure that File Introspection is installed on the system.
- Verify that the thin agent is running by with the systemctl status vsepd.service command.
- If you suspect that the thin agent is causing a performance problem with the system, stop the service by running the service vsepd stop command.
- Then perform a test to get a baseline. You can then start the vsep service and perform another test by running the service vsepd start command.
- For deployments consuming network events,
vmw_conn_notify
also needs to be checked by running the systemctcl status vmw_conn_notifyd.service. - Enable debugging for the Linux thin agent:
- Run /etc/vsep/vsep
refresh-logging
. - Usage:
/etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>
Where, <dest>: [1-2] 1 - log to the VM and 2 - log to the ESX host.
<level>: [1-7] where 4 is for logging level INFO, 7 is for logging level DEBUG.
<sub-component-name>: one or more of transport, timer, file, network, process, system
When logging to host is enabled, logs are stored in vmware.log of respective vmfs directory of VMs on ESXi hosts.
Note: Enabling full logging might result in heavy log activity flooding the vmware.log file. Disable full logging as soon as possible.
- Run /etc/vsep/vsep
Enable debugs based on context (file, process, network or system)
Enhanced logging support allowed thin agent to log module debug level information of specific feature/functionality to vmware.log on host or syslog in the VM.
One needs to restart Thin Agent service in case debug logs are not generated in respective files. Note that logging to vmware.log on host may get throttled if logging is heavy. The refresh-logging input parameter has been added to /etc/vsep/vsepd. Its usage can be displayed by running:
Debugging:
# /etc/vsep/vsepd refresh-loggingUsage: /etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>
where, <dest>: [1-2]: 1 is log to the VM and 2 is log to the ESX host. When logging to VM is enabled, the logs will be stored at the following location based on the Linux distribution software.
On Ubuntu VMs : /var/log/syslog
On CentOS, RHEL and SLES: /var/log/messages
When logging to host is enabled, the logs will be stored in vmware.log of respective vmfs directory of VMs on ESXi host.
<level>: [1-7], where 4 is for logging level INFO, 7 is for logging level DEBUG.
<sub-component-name>: one or more of transport, timer, file, network, process, system
Example:
Enabling the following commands only print logs from that context.
Debug logging for network introspection can be enabled using the following command.
/etc/vsep/vsepd refresh-logging 1 7 networkDebug logging for process introspection:
/etc/vsep/vsepd refresh-logging 1 7 process
Debug logging for anti-virus use case:
/etc/vsep/vsepd refresh-logging 1 7 fileFor command processing in timer context (all use cases):
/etc/vsep/vsepd refresh-logging 1 7 timer
For user monitoring:
/etc/vsep/vsepd refresh-logging 1 7 system
For framework communication between SVM and Context Mux (all use cases):
/etc/vsep/vsepd refresh-logging 1 7 transportTroubleshooting Thin Agent Crashes on Linux
Thin agent dumps core when it crashes. However, it is dependent on the operating system configuration for core dumps. Each Linux distro has different ways and configurations of generating core dump when a system crashes.
For example, you can use apport for applications to dump core on a crash, where as in Red Hat you use abrtd. However, thin agent dumps backtrace in /var/log/syslog (Ubuntu) or /var/log/messages (CentOS, RHEL and SLES) in VM or vmware.log if logging is enabled on host, depending on the logging destination.
A sample backtrace:
localhost systemd: Started Session 4 of user root. localhost vsep: EMERG: 0: sig_handler(): Received signal: 11 localhost vsep: EMERG: 0: sig_handler(): backtrace returned 7 pointers localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x1d35e) [0x7fa2e4c9135e] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(+0x35a00) [0x7fa2e3d76a00] localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x3f789) [0x7fa2e4cb3789] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6e0fc) [0x7fa2e47960fc] localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6d745) [0x7fa2e4795745] localhost vsep: EMERG: 0: sig_handler(): /lib64/libpthread.so.0(+0x7df3) [0x7fa2e4109df3] localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(clone+0x6d) [0x7fa2e3e373dd] localhost vsep: EMERG: 0: sig_handler(): Unmarking all fanotify marked mount points