To accommodate all log data from the products in the SDDC, you must size the compute resources and storage for the Log Insight nodes properly.

By default, the vRealize Log Insight virtual appliance uses the predefined values for small configurations, which have 4 vCPUs, 8 GB of virtual memory, and 530.5 GB of disk space provisioned. vRealize Log Insight uses 100 GB of the disk space to store raw data, index, metadata, and other information.

Sizing Nodes

Select a size for the vRealize Log Insight nodes so as to collect and store log data from the SDDC management components and tenant workloads according to the objectives of this design.

Table 1. Compute Resources for a vRealize Log Insight Small-Size Node

Attribute

Specification

Appliance size

Small

Number of CPUs

4

Memory

8 GB

Disk Capacity

530.5 GB (490 GB for event storage)

IOPS

500 IOPS

Amount of processed log data when using log ingestion

30 GB/day of processing per node

Number of processed log messages

2,000 event/second of processing per node

Environment

Up to 100 syslog connections per node

Sizing Storage

Sizing is based on IT organization requirements, but this design provides calculations based on a single-region implementation, and is implemented on a per-region basis. This sizing is calculated according to the following node configuration per region:

Table 2. Management Systems Whose Log Data Is Stored by vRealize Log Insight

Category

Logging Sources

Quantity

Consolidated cluster

Platform Services Controller

1

vCenter Server

1

ESXi Hosts

64

NSX for vSphere for the consolidated cluster

NSX Manager

1

NSX Controller Instances

3

NSX Edge services gateway instances:

  • Two ESGs for north-south routing

  • Universal distributed logical router

  • Load balancer for vRealize Automation and vRealize Operations Manager

  • Load balancer for Platform Services Controllers

4

vRealize Automation

vRealize Automation Appliance with embedded vRealize Orchestrator

1

vRealize IaaS Web Server

1

vRealize IaaS Manager Server, DEM 1 and Agent Server

1

Microsoft SQL Server

1

vRealize Business for Cloud

vRealize Business server appliance

1

vRealize Business data collector

1

vRealize Operations Manager

Analytics nodes

1

Remote collector node

1

These components aggregate to approximately 85 syslog and vRealize Log Insight Agent sources.

Assuming that you want to retain 7 days of data, apply the following calculation:

vRealize Log Insight receives approximately 150 MB to 190 MB of log data per-day per-source as follows.

  • The rate of 150 MB of logs per day is valid for Linux where 170 bytes per message is the default message size.

  • The rate of 190 MB of logs per day is valid for Windows where 220 bytes per message is the default message size.

170 bytes per message * 10 messages per second * 86400 seconds per day = 150 MB of logs per-day per-source (Linux)
220 bytes per message * 10 messages per second * 86400 seconds per day = 190 MB of logs per-day per-source (Windows)

In this validated design, to simplify calculation, all calculations have been done using the large 220 byte size which results in 190 MB of log data expected per-day per-source.

For 220 logging sources, at a basal rate of approximately 190 MB of logs that are ingested per-day per-source over 7 days, you need the following storage space:

Calculate the storage space required for a single day for log data using the following calculation:

85 sources * 190 MB of logs per-day per-source * 1e-9 GB per byte ≈ 16 GB disk space per-day

Based on the amount of data stored in a day, to size the appliance for 7 days of log retention, use the following calculation:

(112 GB * 7 days) / 1 аppliance ≈ 112 GB log data per vRealize Log Insight node

112 GB * 1.7 indexing overhead ≈ 190 GB log data per vRealize Log Insight Node 

Based on this example, the storage space that is allocated per vRealize Log Insight virtual appliance is enough to monitor the SDDC.

Consider the following approaches when you must increase the Log Insight capacity:

  • If you must maintain a log data retention for more than 7 days in your SDDC, you might add more storage per node by adding a new virtual hard disk. vRealize Log Insight supports virtual hard disks of up to 2 TB. If you must add more than 2 TB to a virtual appliance, add another virtual hard disk.

    When you add storage to increase the retention period, extend the storage for all virtual appliances.

    When you add storage so that you can increase the retention period, extend the storage for all virtual appliances. To increase the storage, add new virtual hard disks only. Do not extend existing retention virtual disks. Once provisioned, do not reduce the size or remove virtual disks to avoid data loss.

  • If you must monitor more components by using log ingestion and exceed the number of syslog connections or ingestion limits defined in this design, you can do the following:

    • Increase the size of the vRealize Log Insight node, to a medium or large deployment size as defined in the vRealize Log Insight documentation.

    • Deploy more vRealize Log Insight virtual appliances to scale your environment out. vRealize Log Insight can scale up to 12 nodes in an HA cluster.

Table 3. Design Decisions About the Compute Resources for the vRealize Log Insight Nodes

Decision ID

Design Decision

Design Justification

Design Implication

CSDDC-OPS-LOG-002

Deploy vRealize Log Insight nodes of small size.

Accommodates the number of expected syslog and vRealize Log Insight Agent connections from the following sources:

  • Consolidated vCenter Server and connected Platform Services Controller

  • ESXi hosts in the consolidated cluster

  • NSX for vSphere components in the consolidated cluster

  • vRealize Automation components

  • vRealize Business components

  • vRealize Operations Manager components

These sources approximately generate about 85 syslog and vRealize Log Insight Agent sources.

Using a small-size appliance ensures that the storage space for the vRealize Log Insight cluster is sufficient for 7 days of data retention.

You must increase the size of the nodes if you configure Log Insight to monitor additional syslog sources.