The objective of NSX Intrusion Detection and Prevention Service (IDS/IPS) is to monitor network traffic on the hosts and edges for malicious activity by comparing the traffic against a known set of signatures. The objective of NSX Malware Prevention is to extract files from the network traffic on the hosts and edges and analyze these files for malicious behavior.

Overview of NSX Intrusion Detection and Prevention Service

NSX IDS/IPS monitors network traffic on a host for suspicious activity by comparing the traffic against signatures. A signature specifies a pattern for a type of network intrusion that needs to be detected and reported. Whenever a matching traffic pattern to a signature is found, a predefined action is taken, such as generating an alert or blocking the traffic from reaching its destination.

Implementation of IDS is carried through the following methods:
  • Knowledge-based signatures: Knowledge-based signatures incorporate specific knowledge or pattern that corresponds to a known type of attack. In this approach, IDS attempts to detect intrusions based on already known malicious instruction sequences specified in signatures. Thus, knowledge-based signatures are limited to attacks that are already known and cannot cover targeted or zero-day threats.
  • Behavior-based detection: Behaviour-based detection attempts to identify anomalous behavior by pinpointing interesting events that are different or unusual compared to a baseline or normal traffic.

    These events are called informational or info and consists of events that pinpoint unusual activities in a network that are not necessarily malicious but can provide valuable information when investigating a breach. Signatures are bundled together with custom detection logic that can be updated without having to recompile or modify the IDS engine. Behavior-based detection introduces a new IDS intrusion severity level as 'suspicious'.

NSX supports IDS/IPS capability on both Distributed and Gateway firewall. NSX IDS/IPS on Gateway Firewall feature only on tier-1 gateways.

Overview of NSX Malware Prevention

NSX Malware Prevention can detect and prevent known malicious files and unknown malicious files. Unknown malicious files are also referred to as zero-day threats. To detect malware, NSX Malware Prevention uses a combination of the following techniques :
  • Hash-based detection of known malicious files
  • Local analysis of unknown files
  • Cloud analysis of unknown files

In all the versions of NSX 4.x, the supported maximum file size limit for malware analysis is 64 MB.

NSX Distributed Malware Prevention
  • In NSX 4.0, malware detection and prevention on the Distributed Firewall is supported only for Windows guest endpoints (VMs), which are running on vSphere host clusters that are prepared for NSX.

    Only Windows Portable Executable (PE) files are supported for local analysis and cloud analysis. Other file categories are not supported by NSX Distributed Malware Prevention

  • Starting in NSX 4.0.1.1, malware detection and prevention on the Distributed Firewall is supported for both Windows and Linux guest endpoints (VMs), which are running on vSpherehost clusters that are prepared for NSX.

    Local analysis and cloud analysis of all categories of malware files are supported. To view the list of supported file categories, see File Categories Supported for NSX Malware Prevention.

NSX Malware Prevention on Gateway Firewall

In NSX 4.0 or later, only detection of malware is supported on the Gateway Firewall. Local analysis and cloud analysis of all categories of malware files are supported. To view the list of supported file categories, see the hyperlinked topic that is mentioned in the NSX Distributed Malware Prevention section.

Figure 1. Concept diagram of NSX Malware Prevention

Concept diagram of NSX Malware Prevention.

On the north-south traffic, the NSX Malware Prevention feature uses the IDS/IPS engine on the NSX Edges to extract or intercept the files that are entering the data center. On the east-west traffic, this feature uses the capabilities of the NSX Guest Introspection (GI) platform. If the file bypasses scrutiny on the NSX Edge and reaches the host, the file is extracted by the GI thin agent on guest VMs.

To detect and prevent malware on guest VMs, you must install the NSX Guest Introspection thin agent on guest VMs, and deploy the NSX Distributed Malware Prevention service on vSphere host clusters that are prepared for NSX. When this service is deployed, a service virtual machine (SVM) is installed on each host of the vSphere cluster and NSX Malware Prevention is enabled on the host cluster.

NSX Guest Introspection thin agent drivers for Windows are included with VMware Tools. To view the VMware Tools versions that are supported for your version of NSX, see the VMware Product Interoperability Matrix. To view the list of supported Windows guest operating systems for a specific VMware Tools version, see the release notes for that version in the VMware Tools documentation.

Note: Windows 11 and Windows 2022 guest OS versions are supported with VMware Tools 12.0.6 or later.

The Guest Introspection thin agent for Linux is available as part of the operating system specific packages (OSPs). The packages are hosted on VMware packages portal. Installing open-vm-tools or VM Tools is not required for Linux. To view the list of supported Linux guest operating system versions, see the Prerequisites section in Install the Guest Introspection Thin Agent for Anti-virus on Linux Virtual Machines.

NSX Malware Prevention File Events

File events are generated when files are extracted by the IDS engine on the NSX Edges in the north-south traffic and by the NSX Guest Introspection agent on the virtual machine endpoints in the distributed east-west traffic.

NSX Malware Prevention feature inspects the extracted files to determine whether they are benign, malicious, or suspicious. Each unique inspection of a file is counted as a single file event in NSX. In other words, a file event refers to a unique file inspection.

For information about monitoring the NSX Malware Prevention file events by using the UI, see Monitoring File Events.

For information about monitoring the file events by using the NSX Malware Prevention file event APIs, see the documentation on the VMware Developer Documentation portal.