The Guest Introspection thin agent is installed withVMware Tools ™ on each guest virtual machine.

Troubleshooting the Thin Agent on Linux

If a virtual machine is slow in reading and writing operations, and unzipping or saving files then there may be issues with the thin agent.

  1. Check the compatibility of all the components involved. Compatibility is one of the main issues with Endpoint. You need the build numbers for ESXi, vCenter Server, NSX Manager, and which ever Security solution you have chosen (Trend Micro, McAfee, Kaspersky, Symantec etc). Once this data has been collected, compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.

  2. Ensure that File Introspection is installed on the system.

  3. Verify that the thin agent is running by with the service vsepd status command. Once this command is executed you should see the vsep service in running state.

  4. If you believe that the thin agent is causing a performance issue with the system, stop the service by running the service vsepd stop command.

  5. Then perform a test to get a baseline. You can then start the vsep service and perform another test by running the service vsepd start command.

  6. Enable debugging for the Linux thin agent:

    1. Open the /etc/vsep/vsep.conf file

    2. Change DEBUG_LEVEL=4 to DEBUG_LEVEL=7 for all logs

    3. This can be set to DEBUG_LEVEL=6 for moderate logs

    4. The default log destination(DEBUG_DEST=2) is vmware.log (on host) to change it to guest (i.e /var/log/message or /var/log/syslog) set DEBUG_DEST=1

      Note:

      Enabling full logging may result in heavy log activity flooding the vmware.log file, causing it to potentially grow to be very large. Disable full logging as soon as possible.

Troubleshooting the Thin Agent on Windows

  1. Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and which ever Security solution you have chosen (Trend Micro, McAfee, Kaspersky, Symantec etc). Once all of this data has been collected, you can compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.

  2. Ensure that VMware Tools ™ is up-to-date. If you see that only a particular virtual machine is affected, see Installing and upgrading VMware Tools in vSphere (2004754).

  3. Verify that the thin agent is loaded by running the Powershell command fltmc.

    Once this command is executed, You should see the name vsepflt on the list of drivers. If the driver is not loaded, you should be able to load the driver with the fltmc load vsepflt command.

  4. If t the thin agent is causing a performance issue with the system, unload the driver with this command: fltmc unload vsepflt.

    Next, perform a test to get a baseline. You can then load the driver and perform another test by running this command:

    fltmc load vsepflt.

    If you do verify that there is a performance problem with the Thin agent, see Slow VMs after upgrading VMware tools in NSX and vCloud Networking and Security (2144236).

  5. If you are not using Network Introspection, remove or disable this driver.

    Network Introspection can also be removed through the Modify VMware Tools installer:

    1. Mount the VMware Tools installer.

    2. Navigate to Control Panel > Programs and Features.

    3. Right-click VMware Tools > Modify.

    4. Select Complete install.

    5. Find NSX File Introspection. There should be a sub folder just for Network Introspection.

    6. Disable Network Introspection.

    7. Reboot the VM to complete the uninstallation of the driver.

  6. Enable debug logging for the thin agent. For more information, see Guest Introspection Logs. All debugging information is configured to log to the vmware.log file for that virtual machine.

  7. Review the file scans of the thin agent by reviewing the procmon logs. For more information, see Troubleshooting vShield Endpoint performance issues with anti-virus software (2094239).

Collect Environment and Workload Details

  1. Determine if NSX Guest Introspection is used in the customer environment. If it is not, remove the Guest Introspection service for the virtual machine, and confirm the issue is resolved. Troubleshoot a Guest Introspection issue only if Guest Inspection is required.

  2. Collect environment details:

    1. ESXi build version - Run the command uname –a on the ESXi host or click on a host in the vSphere Web Client and look for the build number at top of the right-hand pane.

    2. Linux product version and build number

    3. /usr/sbin/vsep -v will give the production version

      Build number
      ------------------
      Ubuntu 
      dpkg -l | grep vmware-nsx-gi-file
      SLES12 and RHEL7
      rpm -qa | grep vmware-nsx-gi-file
  3. VMware NSX ® for vSphere ® version, and the following:

    • Partner solution name and version number

    • EPSec Library version number used by the partner solution: Log into the SVM and run #strings path to EPSec library/libEPSec.so | grep BUILD

    • Guest operating system in the virtual machine

    • Any other third-party applications or file system drivers

  4. ESX GI Module (MUX) version - run the command esxcli software vib list | grep epsec-mux.

  5. Collect workload details, such as the type of server.

  6. Collect ESXi host logs. For more information, see Collecting diagnostic information for VMware ESX/ESXi (653).

  7. Collect service virtual machine (SVM) logs from the partner solution. Reach out to your partner for more details on SVM log collection.

  8. Collect a suspend state file while the problem is occurring, see Suspending a virtual machine on ESX/ESX (2005831) to collect diagnostic information.

Troubleshooting Thin Agent crash

If the Thin Agent crashes, the core file is generated in the /directory. Collect the core dump file (core) from location / directory. Use the file command to check if core is generated by vsep. For example:

# file core
core: ELF 64-bit LSB  core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/sbin/vsep'

Virtual machine hangs or freezes

Collect the VMware vmss file of the virtual machine in a suspended state, see Suspending a virtual machine on ESX/ESXi to collect diagnostic information (2005831) or crash the virtual machine and collect the full memory dump file. VMware offers a utility to convert an ESXi vmss file to a core dump file. See Vmss2core fling for more information.