Troubleshooting the Thin Agent on Linux or Windows

The Guest Introspection thin agent is installed with VMware Tools™ on each guest virtual machine.

Troubleshooting the Thin Agent on Linux

If a virtual machine is slow in reading and writing operations, and unzipping or saving files then there might be problems with the thin agent.

Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
Ensure that File Introspection is installed on the system.
Verify that the thin agent is running by with the service vsepd status command.
If you believe that the thin agent is causing a performance problem with the system, stop the service by running the service vsepd stop command.
Then perform a test to get a baseline. You can then start the vsep service and perform another test by running the service vsepd start command.
Enable debugging for the Linux thin agent:
1. Edit the /etc/vsep/vsep.conf file.
2. Change DEBUG_LEVEL=4 to DEBUG_LEVEL=7 for all logs, or set to DEBUG_LEVEL=6 for moderate logs.
3. The default log destination(DEBUG_DEST=2) is vmware.log (on host). To change it to the guest VM (/var/log/message or /var/log/syslog) set DEBUG_DEST=1.
  Note: Enabling full logging might result in heavy log activity flooding the vmware.log file. Disable full logging as soon as possible.

Troubleshooting the Thin Agent on Windows

Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data is collected, you can compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
Ensure that VMware Tools™ is up-to-date. If you see that only a particular virtual machine is affected, see Installing and upgrading VMware Tools in vSphere (2004754).
Verify that the thin agent is loaded by running the PowerShell command fltmc.
Verify that vsepflt is included in the list of drivers. If the driver is not loaded, try loading the driver with the fltmc load vsepflt command.
If the thin agent is causing a performance problem with the system, unload the driver with this command: fltmc unload vsepflt.

Next, perform a test to get a baseline. You can then load the driver and perform another test by running this command:

fltmc load vsepflt.

If you do verify that there is a performance problem with the Thin agent, see Slow VMs after upgrading VMware tools in NSX and vCloud Networking and Security (2144236).
If you are not using Network Introspection, remove or disable this driver.
Network Introspection can also be removed through the Modify VMware Tools installer:
1. Mount the VMware Tools installer.
2. Navigate to Control Panel > Programs and Features.
3. Right-click VMware Tools > Modify.
4. Select Complete install.
5. Find NSX File Introspection. This contains a subfolder for Network Introspection.
6. Disable Network Introspection.
7. Reboot the VM to finish the uninstallation of the driver.
Enable debug logging for the thin agent. All debugging information is configured to log to the vmware.log file for that virtual machine.
Review the file scans of the thin agent by reviewing the procmon logs. For more information, see Troubleshooting vShield Endpoint performance issues with anti-virus software (2094239).

Collect Environment and Workload Details

Determine if Guest Introspection is used in your environment. If it is not, remove the Guest Introspection service for the virtual machine, and confirm that the problem is resolved. Troubleshoot a Guest Introspection problem only if Guest Inspection is required.
Collect environment details:
1. To collect the ESXi build version, run the command uname –a on the ESXi host or select a host in the vSphere Web Client and look for the build number at the top of the right pane.
2. Linux product version and build number.
3. /usr/sbin/vsep -v returns the production version:
```
Build number
------------------
Ubuntu 
dpkg -l | grep vmware-nsx-gi-file
SLES12 and RHEL7
rpm -qa | grep vmware-nsx-gi-file
```
Collect the NSX for vSphere version, and the following:
- Partner solution name and version number
- EPSec Library version number used by the partner solution: Log into the SVM and run strings <path to EPSec library>/libEPSec.so | grep BUILD
- Guest operating system in the virtual machine
- Any other third-party applications or file system drivers
ESX GI Module (MUX) version - run the command esxcli software vib list | grep epsec-mux.
Collect workload details, such as the type of server.
Collect ESXi host logs. For more information, see Collecting diagnostic information for VMware ESX/ESXi (653).
Collect service virtual machine (SVM) logs from the partner solution. Contact your partner for more details on SVM log collection.
Collect a suspend state file while the problem is occurring, see Suspending a virtual machine on ESX/ESX (2005831) to collect diagnostic information.

Troubleshooting Thin Agent Crashes

If the Thin Agent crashes, the core file is generated in the /directory. Collect the core dump file (core) from location / directory. Use the file command to check if the core file is generated by vsep. For example:

# file core
core: ELF 64-bit LSB  core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/vsep'

Virtual Machine Hangs or Freezes

Collect the VMware vmss file of the virtual machine in a suspended state, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831), or crash the virtual machine and collect the full memory dump file. VMware offers a utility to convert an ESXi vmss file to a core dump file. See Vmss2core fling for more information.