Recognizing Duplication of Forwarded Data

This topic describes how you can recognize duplication of different types of forwarded data.

Alert Duplication

In the most trivial case, you will notice that alerts are duplicated when you see two records that have the same alert_id. However, the Carbon Black Cloud Alerts Service intentionally updates certain alerts (for example, Carbon Black Analytics alerts) anytime the Carbon Black Cloud observes new, suspicious endpoint activities related to the alert's events within a short period of time. This necessitates re-forwarding that updated alert to make sure that the updated alert attributes are available to customers who need that updated state. Under the most extreme circumstances, the Alerts Service updates specific alerts up to 60 times. This updating behaviour is not true for other kinds of alerts, such as Watchlist alerts or Device Control alerts.

When the Alerts Service deliberately issues updated copies of the alert, the last_update_time field will always be incremented for the same alert_id, and one or more other fields in that alert record will have been updated as well.

When the Data Forwarder creates a duplicate of the alert, all data fields will be identical, including both the alert_id and last_update_time fields.

Event Duplication: NGAV origin

All NGAV events emitted by the Carbon Black Cloud Data Forwarder include a unique identifier event_id.

Event Duplication: EDR origin

The EDR events emitted by the Carbon Black Cloud Data Forwarder do not include a unique identifier.

Event duplication: NGAV + EDR Side-by-Side

By design, the NGAV and EDR features of the sensor independently instrument the events they deem to be reportable. This generally means that (for those events instrumented by both features — for example, excluding modloads and fileless scriptloads, which are exclusive to Carbon Black Cloud Enterprise EDR — that there can occasionally be separately-reported events for the same activity.

Further, because of the lack of an EDR event_id, there is no definitive single field by which an NGAV event can be correlated to its EDR equivalent. A combination of childproc_guid and type can suffice for type=endpoint.event.procstart, but for other event types, process_guid and type still means multiple events. In such cases, correlating by device_timestamp can help, but two threads in the sensor rarely generate exactly the same timestamp down to millisecond precision.

For example, the sensor reports four filemods for a process: one NGAV filemod and three EDR filemods. The NGAV event and one of the three EDR events are for ACTION_FILE_CREATE. EDR reports ACTION_FILE_CREATE | ACTION_FILE_MOD_OPEN | ACTION_FILE_OPEN_WRITE. In the case of FILE_CREATE, the filemod_name matches unambiguously between the "same" event, but the same is not true of many regmod events.