The Guest Introspection thin agent is installed with VMware Tools™ on each guest virtual machine.

Troubleshooting the Thin Agent on Linux

If a virtual machine is slow in reading and writing operations, and unzipping or saving files then there might be problems with the thin agent.

  1. Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
  2. Ensure that File Introspection is installed on the system.
  3. Verify that the thin agent is running by with the systemctl status vsepd.service command.
  4. If you suspect that the thin agent is causing a performance problem with the system, stop the service by running the service vsepd stop command.
  5. Then perform a test to get a baseline. You can then start the vsep service and perform another test by running the service vsepd start command.
  6. For deployments consuming network events, vmw_conn_notify also needs to be checked by running the systemctcl status vmw_conn_notifyd.service.
  7. Enable debugging for the Linux thin agent:
    1. Run /etc/vsep/vsep refresh-logging.
    2. Usage: /etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>

      Where, <dest>: [1-2] 1 - log to the VM and 2 - log to the ESX host.

      <level>: [1-7] where 4 is for logging level INFO, 7 is for logging level DEBUG.

      <sub-component-name>: one or more of transport, timer, file, network, process, system

      When logging to host is enabled, logs are stored in vmware.log of respective vmfs directory of VMs on ESXi hosts.

      Note: Enabling full logging might result in heavy log activity flooding the vmware.log file. Disable full logging as soon as possible.

Enable debugs based on context (file, process, network or system)

Enhanced logging support allowed thin agent to log module debug level information of specific feature/functionality to vmware.log on host or syslog in the VM.

One needs to restart Thin Agent service in case debug logs are not generated in respective files. Note that logging to vmware.log on host may get throttled if logging is heavy. The refresh-logging input parameter has been added to /etc/vsep/vsepd. Its usage can be displayed by running:

Debugging:

# /etc/vsep/vsepd refresh-logging

Usage: /etc/vsep/vsepd refresh-logging <dest> <<level> sub-component-name>

where, <dest>: [1-2]: 1 is log to the VM and 2 is log to the ESX host. When logging to VM is enabled, the logs will be stored at the following location based on the Linux distribution software.

On Ubuntu VMs : /var/log/syslog

On CentOS, RHEL and SLES: /var/log/messages

When logging to host is enabled, the logs will be stored in vmware.log of respective vmfs directory of VMs on ESXi host.

<level>: [1-7], where 4 is for logging level INFO, 7 is for logging level DEBUG.

<sub-component-name>: one or more of transport, timer, file, network, process, system

Example:

Enabling the following commands only print logs from that context.

Debug logging for network introspection can be enabled using the following command.

/etc/vsep/vsepd refresh-logging 1 7 network

Debug logging for process introspection:

/etc/vsep/vsepd refresh-logging 1 7 process

Debug logging for anti-virus use case:

/etc/vsep/vsepd refresh-logging 1 7 file

For command processing in timer context (all use cases):

/etc/vsep/vsepd refresh-logging 1 7 timer

For user monitoring:

/etc/vsep/vsepd refresh-logging 1 7 system

For framework communication between SVM and Context Mux (all use cases):

/etc/vsep/vsepd refresh-logging 1 7 transport

Troubleshooting Thin Agent Crashes on Linux

Thin agent dumps core when it crashes. However, it is dependent on the operating system configuration for core dumps. Each Linux distro has different ways and configurations of generating core dump when a system crashes.

For example, you can use apport for applications to dump core on a crash, where as in Red Hat you use abrtd. However, thin agent dumps backtrace in /var/log/syslog (Ubuntu) or /var/log/messages (CentOS, RHEL and SLES) in VM or vmware.log if logging is enabled on host, depending on the logging destination.

A sample backtrace:

localhost systemd: Started Session 4 of user root.
localhost vsep: EMERG: 0: sig_handler(): Received signal: 11
localhost vsep: EMERG: 0: sig_handler(): backtrace returned 7 pointers
localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x1d35e) [0x7fa2e4c9135e]
localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(+0x35a00) [0x7fa2e3d76a00]
localhost vsep: EMERG: 0: sig_handler(): /usr/sbin/vsep(+0x3f789) [0x7fa2e4cb3789]
localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6e0fc) [0x7fa2e47960fc]
localhost vsep: EMERG: 0: sig_handler(): /lib64/libglib-2.0.so.0(+0x6d745) [0x7fa2e4795745]
localhost vsep: EMERG: 0: sig_handler(): /lib64/libpthread.so.0(+0x7df3) [0x7fa2e4109df3]
localhost vsep: EMERG: 0: sig_handler(): /lib64/libc.so.6(clone+0x6d) [0x7fa2e3e373dd]
localhost vsep: EMERG: 0: sig_handler(): Unmarking all fanotify marked mount points