A Carbon Black EDR server stores each instance of a process execution and all event data with which it is associated (for example, module loads, registry or file modifications, and network connections) in process documents.
Process documents from multiple sensors are stored in database structures known as shards. To provide optimum storage and search performance, shards are periodically rotated when they reach a certain disk size.
For best performance, the recommended maximum size of a single shard is 500 GB, with up to 12 searchable shards per server for a total process event retention of 6 TB. However, for increased retention, shard size and count can be increased to 750 GB and 14 respectively with additional hardware. The latter configuration can extend total event retention of a single Carbon Black EDR server to 10.5 TB. It is important to note that, while more and bigger shards result in increased retention, fewer and smaller shards result in an improved search performance. You can determine the desired configuration (and retention) for your needs as follows.
Compute an estimate of the event data that the server will store (given your retention needs). Use the data provided in Endpoint activity level percentiles and On-disk size percentiles to calculate this value:
Total Event Data Volume = (Number of Endpoints * Number of Process Documents per Endpoint per Day * Size of Process Document on Disk) * Retention Days |
If the estimated event data volume is less than 10.5 TB, see Server sizing chart based on event data volume to select the appropriate server size. You should choose the smallest shard count and size combination that meets your total event data. A single Carbon Black EDR server is sufficient to support your deployment. If the estimated value is larger than 10.5 TB, see Cluster Sizing to determine the clustered deployment that is necessary to support your environment.
If OS or endpoint breakdown is known or can be estimated, this information can be incorporated into the previous calculation to improve planning accuracy. For example, a deployment includes 1,000 Windows workstations (50-percentile) and 100 Linux servers (75-percentile). A reasonable estimate would be:
Required Event Data Storage = (1000 * 7800 + 100 * 125000) * 3.6KB * 30 days ~ 2.1TB
Because this value is less than 10.5 TB, a single server can accommodate this deployment. The next step is to determine the appropriate server size based on the desired search performance.
| Event Data Volume | Shard Size | Number of Shards | Disk Space for Event Data* | Memory | CPU | Example Configuration |
|---|---|---|---|---|---|---|
| < 500GB | 50GB | 10 | 1TB | 16 GB | 2 cores | 2.3 GHz Intel Xeon E5-2686 v4 Processor or similar. SSD disks with 16000 IOPS (250 MiB/s throughput) |
| < 2TB | 200GB | 10 | 3TB | 32 GB | 4 cores | |
| < 3.6TB | 300GB | 12 | 5TB | 64 GB | 8 cores | |
| < 6TB | 500GB | 12 | 8TB | 128 GB | 16 cores | |
| < 10.5TB | 750GB | 14** | 13TB | 256 GB | 32 cores |
*Disk size is the space that is required to store Carbon Black EDR event datastore, including necessary overhead for database optimization and binary storage. Additional disk space is required for storing non-event-data OS files.
**Complex queries take more time to execute, resulting in slower console response times.
After the server sizing is completed, configure the Carbon Black EDR application with the following settings in cb.conf:
SolrTimePartitioningMinutes=8640 (for 6 days)SolrTimePartitioningActivePartitions= [Number of Shards]SolrTimePartitioningMaxSizeMB= [Shard Size in MB]#MaxEventStoreDays(comment out, partition size and count determines days)#MaxEventStoreSizeInMB(comment out, partition size and count determines size)MaxEventStoreSizeInPercent=90 (a safeguard to avoid running out of disk)
See the Carbon Black EDR Server Configuration Guide for instructions on how to configure these parameters in cb.conf. See also Cb Enterprise Response - Managing Retention.
