To accommodate all log data from the products in the SDDC, you must size the compute resources and storage for the Log Insight nodes properly.

By default, the vRealize Log Insight virtual appliance uses the predefined values for small configurations, which have 4 vCPUs, 8 GB of virtual memory, and 530.5 GB of disk space. vRealize Log Insight uses 100 GB of the disk space to store raw data, index, metadata, and other information.

Sizing Nodes

Select a size for the vRealize Log Insight nodes so as to collect and store log data from the SDDC management components and tenant workloads according to the objectives of this design.

Table 1. Compute Resources for a vRealize Log Insight Medium-Size Node

Attribute

Specification

Appliance size

Medium

Number of CPUs

8

Memory

16 GB

Disk Capacity

530.5 GB (490 GB for event storage)

IOPS

1,000 IOPS

Amount of processed log data when using log ingestion

75 GB/day of processing per node

Number of processed log messages

5,000 event/second of processing per node

Environment

Up to 250 syslog connections per node

Sizing Storage

Storage sizing depends on IT organization requirements, but this design provides calculations based on a single-region implementation, and is implemented on a per-region basis. This sizing is calculated according to the following node configuration per region:

Table 2. Management Systems Whose Log Data Is Stored by vRealize Log Insight

Category

Logging Sources

Quantity

Management pod

Platform Services Controller

1

vCenter Server

1

ESXi Hosts

4

Shared edge and compute pod

Platform Services Controller

1

vCenter Server

1

ESXi Hosts

64

NSX for vSphere for the management pod

NSX Manager

1

NSX Controller Instances

3

NSX Edge services gateway instances:

  • Two ESGs for north-south routing

  • Universal distributed logical router

  • Load balancer for vRealize Automation and vRealize Operations Manager

  • Load balancer for Platform Services Controllers

5

NSX for vSphere for the shared edge and compute pod

NSX Manager

1

NSX Controller Instances

3

NSX Edge services gateway instances:

  • Universal distributed logical router

  • Distributed logical router

  • Two ESGs for north-south routing

4

Cross-region event forwarding

Total * 2

These components aggregate to approximately 108 syslog and vRealize Log Insight Agent sources per region, or 220 sources with a cross-region configuration. Assuming that you want to retain 7 days of data, apply the following calculation:

vRealize Log Insight receives approximately 150 MB to 190 MB of log data per-day per-source as follows.

  • The rate of 150 MB of logs per day is valid for Linux where 170 bytes per message is the default message size.

  • The rate of 190 MB of logs per day is valid for Windows where 220 bytes per message is the default message size.

170 bytes per message * 10 messages per second * 86400 seconds per day = 150 MB of logs per-day per-source (Linux)
220 bytes per message * 10 messages per second * 86400 seconds per day = 190 MB of logs per-day per-source (Windows)

In this validated design, to simplify calculation, all calculations have been done using the large 220 byte size which results in 190 MB of log data expected per-day per-source.

For 220 logging sources, at a basal rate of approximately 190 MB of logs that are ingested per-day per-source over 7 days, you need the following storage space:

Calculate the storage space required for a single day for log data using the following calculation:

220 sources * 190 MB of logs per-day per-source * 1e-9 GB per byte ≈ 42 GB disk space per-day

Based on the amount of data stored in a day, to size the appliance for 7 days of log retention, use the following calculation:

(42 GB * 7 days) / 3 аppliances ≈ 100 GB log data per vRealize Log Insight node

100 GB * 1.7 indexing overhead ≈ 170 GB log data per vRealize Log Insight Node 

Based on this example, the storage space that is allocated per medium-size vRealize Log Insight virtual appliance is enough to monitor the SDDC.

Consider the following approaches when you must increase the Log Insight capacity:

  • If you must maintain a log data retention for more than 7 days in your SDDC, you might add more storage per node by adding a new virtual hard disk. vRealize Log Insight supports virtual hard disks of up to 2 TB. If you must add more than 2 TB to a virtual appliance, add another virtual hard disk.

    When you add storage to increase the retention period, extend the storage for all virtual appliances.

    When you add storage so that you can increase the retention period, extend the storage for all virtual appliances. To increase the storage, add new virtual hard disks only. Do not extend existing retention virtual disks. Once provisioned, do not reduce the size or remove virtual disks to avoid data loss.

  • If you must monitor more components by using log ingestion and exceed the number of syslog connections or ingestion limits defined in this design, you can do the following:

    • Increase the size of the vRealize Log Insight node, to a medium or large deployment size as defined in the vRealize Log Insight documentation.

    • Deploy more vRealize Log Insight virtual appliances to scale your environment out. vRealize Log Insight can scale up to 12 nodes in an HA cluster.

Table 3. Design Decisions about the Compute Resources for the vRealize Log Insight Nodes

Decision ID

Design Decision

Design Justification

Design Implication

SDDC-OPS-LOG-003

Deploy vRealize Log Insight nodes of medium size.

Accommodates the number of expected syslog and vRealize Log Insight Agent connections from the following sources:

  • Management vCenter Server and Compute vCenter Server, and connected Platform Services Controller pair

  • Management ESXi hosts, and shared edge and compute ESXi hosts

  • Management and compute components for NSX for vSphere

  • Cross-vRealize Log Insight cluster event forwarding.

These source approximately generate about 220 syslog and vRealize Log Insight Agent sources.

Using a medium-size appliances ensures that the storage space for the vRealize Log Insight cluster is sufficient for 7 days of data retention.

You must increase the size of the nodes if you configure Log Insight to monitor additional syslog sources.