NSX Guest Introspection Platform Thin Agent Logs on Windows

The thin agent is installed on the VM Guest OS and intercepts various types of IO activity that include file, network, processes and so on.

Log Path and Sample Message

The thin agent consists of NSX Guest Introspection drivers – vsepflt.sys, vnetwfp.sys.

The thin agent logs are pushed on the ESXi host, as part of the vCenter Log Bundle. The log path is /vmfs/volumes/<datastore>/<vmname>/vmware.log For example: /vmfs/volumes/5978d759-56c31014-53b6-1866abaace386/Windows10-(64-bit)/vmware.log

Thin agent messages follow the format of <timestamp> <VM Name><Process Name><[PID]>: <message>.

In the log example below Guest: vnet or Guest:vsep, indicate log messages related to the respective GI drivers, followed by debug messages.

For example:

2017-10-17T14:25:19.877Z| vcpu-0| I125: Guest: vnet: AUDIT: DriverEntry :
 vnetFilter build-4325502 loaded
2017-10-17T14:25:20.282Z| vcpu-0| I125: Guest: vsep: 
AUDIT: VFileSocketMgrConnectHelper : Mux is connected
2017-10-17T14:25:20.375Z| vcpu-0| I125: 
Guest: vsep: AUDIT: DriverEntry : vfileFilter build-4286645 loaded
 
2017-10-17T18:22:35.924Z| vcpu-0| I125: Guest: vsep: AUDIT: 
VFileSocketMgrConnectHelper : Mux is connected
2017-10-17T18:24:05.258Z| vcpu-0| I125: Guest: vsep: AUDIT: 
VFileFltPostOpCreate : File (\Windows\System32\Tasks\Microsoft\Windows\
SoftwareProtectionPlatform\SvcRestartTask) in a transaction, ignore

Enabling NSX File Introspection driver logs

Because the debug setting can flood the vmware.log file to the point that it throttles, we recommend you disable the debug mode as soon as you have collected all the required information.

This procedure requires you to modify the Windows registry. Before you modify the registry, ensure to take a backup of the registry. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article 136393.

Click Start > Run. Enter regedit, and click OK. The Registry Editor window opens. For more information seen the Microsoft Knowledge Base article 256986.
Create this key using the registry editor: HKEY_LOCAL_Machine\SYSTEM\CurrentControlSet\services\vsepflt\parameters.
Under the newly created parameters key, create these DWORDs. Ensure that hexadecimal is selected when putting in these values:
```
Name: log_dest
Type: DWORD
Value: 0x2

Name: log_level
Type: DWORD
Value: 0x10
```
Other values for log level parameter key:
```
Audit 0x1
Error 0x2
Warn 0x4
Info 0x8
Debug 0x10
```
If you need to restart the File Introspection driver, open a command prompt as an administrator. Run these commands to unload and reload the NSX Endpoint filesystem mini driver:
- fltmc unload vsepflt
- fltmc load vsepflt
You can find the log entries in the vmware.log file located in the virtual machine.

Enabling NSX Network Introspection Driver Logs

Because the debug setting can flood the vmware.log file to the point that it can make it to throttle, we recommend you disable the debug mode as soon as you have collected all the required information.

Click Start > Run. Enter regedit, and click OK. The Registry Editor window opens. For more information seen the Microsoft Knowledge Base article 256986.

Edit the registry:

Windows Registry Editor Version 5.0 
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vnetwfp\Parameters] 
"log_level" = DWORD: 0x0000001F
"log_dest"  = DWORD: 0x00000001

With the log_dest registry settings DWORD: 0x00000001, the endpoint thin agent driver logs that forwards the logs to the debugger. Run the debugger (DbgView from SysInternals or windbg) to capture the debug output.

Alternatively, you can set the log_dest registry setting to DWORD:0x000000002, in which case the driver logs will be printed to vmware.log file, which is located in the corresponding virtual machine folder on the ESXi Host.

Enabling UMC logging

The Endpoint Protection user-mode component (UMC) runs within the VMware Tools service in the protected virtual machine.

On a Windows VM, create a tools config file if it doesn’t exist in the following path: C:\ProgramData\VMWare\VMware Tools\tools.conf
Add these lines in the tools.conf file to enable UMC component logging.
```
[logging]
log = true
vsep.level = debug
vsep.handler = vmx
```
With the vsep.handler = vmx setting, the UMC component logs into the vmware.log file, which is located in the corresponding virtual machine folder on the ESXi host.
With the following setting logs, the UMC component logs will be printed in the specified log file.
```
vsep.handler = file
vsep.data = c:/path/to/vsep.log
```

Troubleshooting the Thin Agent on Windows

Check the compatibility of all the components involved. You need the build numbers for ESXi, vCenter Server, NSX Manager, and the Security solution you have selected (for example, Trend Micro, McAfee, Kaspersky, or Symantec). After this data is collected, you can compare the compatibility of the vSphere components. For more information, see the VMware Product Interoperability Matrices.
Ensure that VMware Tools™ is up-to-date. If you see that only a particular virtual machine is affected, see Installing and upgrading VMware Tools in vSphere (2004754).
Verify that the thin agent is loaded by running the PowerShell command fltmc.
Verify that vsepflt is included in the list of drivers. If the driver is not loaded, try loading the driver with the fltmc load vsepflt command.
If the thin agent is causing a performance problem with the system, unload the driver with this command: fltmc unload vsepflt.
If you are not using Network Introspection, remove or disable this driver.
Network Introspection can also be removed through the Modify VMware Tools installer:
1. Mount the VMware Tools installer.
2. Navigate to Control Panel > Programs and Features.
3. Right-click VMware Tools > Modify.
4. Select Complete install.
5. Find NSX File Introspection. This contains a subfolder for Network Introspection.
6. Disable Network Introspection.
7. Reboot the VM to finish the uninstallation of the driver.
Enable debug logging for the thin agent. All debugging information is configured to log to the vmware.log file for that virtual machine.
Review the file scans of the thin agent by reviewing the procmon logs. For more information, see Troubleshooting vShield Endpoint performance issues with anti-virus software (2094239).

Troubleshooting Thin Agent Crashes on Windows

If the thin agent kernel mode components crashes, the the memory dump is generated in the /%systemroot%\MEMORY.DM.