This topic describes how you can recognize duplication of different types of forwarded data.

Alert Duplication

In some instances, alerts are duplicated and this is apparent when you see two records that have the same alert_id. The Carbon Black Cloud Alerts Service intentionally updates certain alerts (for example, Carbon Black Analytics alerts) anytime the Carbon Black Cloud observes new, suspicious endpoint activities related to the alert's events within a short period of time. This necessitates re-forwarding that updated alert to make sure that the updated alert attributes are available to customers who need that updated state. Under the most extreme circumstances, the Alerts Service updates specific alerts up to 60 times. This updating behaviour is not true for other kinds of alerts, such as Watchlist alerts or Device Control alerts.

When the Alerts Service deliberately issues updated copies of the alert, the last_update_time field will always be incremented for the same alert_id, and one or more other fields in that alert record will have been updated as well.

When the Data Forwarder creates a duplicate of the alert, all data fields will be identical, including both the alert_id and last_update_time fields.

Finally, there are several, specific instances when updated versions of existing Alerts are sent or are purposely not sent. Use the following table to identify those instances.

Carbon Black Cloud Data Forwarder issues updated versions of existing Alerts when the following occur: Carbon Black Cloud Data Forwarder does NOT issue updated versions of existing Alerts in the following instances:
  • You specify an alert Determination, such as False Positive, True Positive.
  • Managed Detection & Response team specifies an alert Determination, such as False Positive, True Positive.
  • Carbon Black Cloud determines that the “threat cause” of a CB_ANALYTICS alert has changed.
  • Carbon Black Cloud determines that the “threat score” has increased.
  • If you update the Workflow of the Alert from Open to In Progress or Closed.

    Reasoning: you should triage and act on Alerts either in your SIEM or in the Carbon Black Cloud, not both.

  • If the Managed Detection & Response team updates the Alert Workflow value.
  • If you add, update, or delete the Notes on an Alert.

    Reasoning: the Notes data is not stored with Alerts, but as a separate data stream. In addition, updating the Threat ID Notes would cause hundreds or thousands of Alerts to be updated all at once.

Event Duplication: NGAV origin

All NGAV events emitted by the Carbon Black Cloud Data Forwarder include a unique identifier event_id.

Event Duplication: EDR origin

The EDR events emitted by the Carbon Black Cloud Data Forwarder do not include a unique identifier.

Event duplication: NGAV + EDR Side-by-Side

By design, the NGAV and EDR features of the sensor independently instrument the events they deem to be reportable. This generally means that (for those events instrumented by both features — for example, excluding modloads and fileless scriptloads, which are exclusive to Carbon Black Cloud Enterprise EDR — that there can occasionally be separately-reported events for the same activity.

Further, because of the lack of an EDR event_id, there is no definitive single field by which an NGAV event can be correlated to its EDR equivalent. A combination of childproc_guid and type can suffice for type=endpoint.event.procstart, but for other event types, process_guid and type still means multiple events. In such cases, correlating by device_timestamp can help, but two threads in the sensor rarely generate exactly the same timestamp down to millisecond precision.

For example, the sensor reports four filemods for a process: one NGAV filemod and three EDR filemods. The NGAV event and one of the three EDR events are for ACTION_FILE_CREATE. EDR reports ACTION_FILE_CREATE | ACTION_FILE_MOD_OPEN | ACTION_FILE_OPEN_WRITE. In the case of FILE_CREATE, the filemod_name matches unambiguously between the "same" event, but the same is not true of many regmod events.