This section discusses partitioning and purging related to new data directories.

Partitioning

New data directories are used when the next partition occurs (every 3 days by default), or sooner if the current data disk is at risk of becoming full.

The server uses simple heuristics in calculating when to partition and where to place the new event partition. It is important that only Carbon Black EDR event data is stored on these partitions. Do not store Carbon Black EDR binary data or any other data that is not associated with Carbon Black EDR event data on these partitions.

  1. A new partition is created in the cbevents* directory that has the most free space at the time of partitioning.

  2. If the current data volume is more than 95% full and additional partitions exist that have more than 5% free space available, the server immediately partitions.

    You can control this threshold using the following configuration parameter:

    SolrTimePartitioningFreeSpaceThresholdPerc

    • Rule 1 ensures that new volumes are used in a balanced fashion. As old data is aging out (being purged), some partitions free up. This ensures optimal use of free space.

    • Rule 2 ensures that the system uses fragmented disk space efficiently in case many cbevents* directories exist. For example, assume you have five volumes, and each has 20% free space. This could result in none of the volumes fitting into the three-day partition. The system continues to try to use one of the partitions (up to its maximum available space) before moving to the next one. As a result, the server might end up with smaller partitions. However, this scenario should be rare.

Active and Read-Only Directories

Any cbevents directories prefixed with cbevents* are used to create new cbevent partitions. Any cbevents directories prefixed with _cbevents* are used as read-only.

A read-only partition can load existing partitions, but new partitions are not created on it. You can use this approach when retiring old volumes. Old partitions will eventually be purged based on time. A second use for _cbevents prefix is for directories that are used only for "cold" partitions (old partitions that are only loaded on demand).

Partition Purging

The system purges partitions based on disk space, time, or the maximum number of allowed partitions.

When purging based on disk space, a purging algorithm considers the overall amount of free disk space. For example, three 100 GB volumes exist, each with 30 GB of free space, giving you a total of 90 GB of free space and a total disk space of 300 GB. The total event data size is the sum of index sizes on all three volumes. (This could be less than 210 GB since the main data volume can also contain store files and other data.)

The following shows how the current purging thresholds (in cb.conf ) are interpreted when multiple volumes exist:

  • MaxEventStoreSizeInPercent - Purge the oldest partition when the total sum of all event core sizes exceeds the given percentage of a total disk space (on all volumes).
  • MaxEventStoreSizeInMB - Purge the oldest partition when the total event store size (on all volumes) exceeds the given threshold.
  • MinAvailableSizeInMB - Purge the oldest partition when the total free disk space (on all volumes) falls below the given threshold.

Extending Disk Space on the Fly

You can add disk space on the fly without having to restart the Carbon Black EDR server.

New directories are automatically used when a new partition occurs, avoiding any server downtime. Make sure that the user has the correct read+write permissions to the directory.