Aria Operations for Logs gathers and organizes logs from different parts of the Aria Suite. It allows for easy searching and analysis of logs in almost real-time. By collecting, importing, and studying logs, it helps find solutions to issues with systems, services, and applications, and uncovers valuable insights.
Design Considerations for Edge Monitoring with Aria Operations
Aria Operations for Logs does not support the same distributed design of Aria Operations. The concept of cloud proxies does not exist for Aria Operations for Logs. Logs from hosts, VMs, apps and clusters can be directly configured to central Aria Operations for Logs cluster via syslog and other ingestion API.
At present, the interim solution for implementing a distributed model of Aria Operations for Logs involves setting up an additional instance of Aria Operations specifically for Logs at the Edge site and configuring it as a Forwarder. While this approach offers certain advantages, it also presents its own set of benefits and challenges, which we will explore in detail shortly.
Architecture
The image below shows there are two ways we can integrate Aria Operations for Logs.
Aria Operations for Logs supports:
Direct ingestion
Using Aria Operations for Logs Forwarder
Direct ingestion
This design will make the endpoints to directly integrate with Aria Operations for Logs without needing any Forwarder to relay the logs to the central datacenter.
Host Logs : Host logs are transmitted as Syslog to the Aria Operations for Logs Server. Hardware manufacturers supply consoles that allow us to activate remote syslog capabilities. The Aria Operations for Logs functions as a syslog server, actively listening on UDP/TCP port 514.
ESXi Logs: In Direct Ingestion the Host logs are sent to Aria Operations for Logs using Syslog. You need to configure syslog settings in the ESXi hosts to send the logs to Aria Operations for Logs Server or You can integrate the vCenter in the Aria Operations for Logs Server, which will automatically configure the syslog in the ESXi hosts. The Port number used for syslog is 514 UDP.
vCenter Logs : vCenter generates a variety of useful objects such as Metrics, Task, and Alarm reports. If you require vCenter logs, it's advisable to integrate syslog from the VAMI (vCenter Appliance Management Interface)
VM Logs : All the VM logs can easily integrate with Syslog. This syslog then can forward the Logs to the Aria Operations for Logs Server. As another option Aria provides Log insight agents which can be installed in the VM which then can relay the logs to Aria Operations for Logs Server.
Cluster Logs : To effectively monitor and integrate a TKG Kubernetes cluster, FluentBit with Aria Operations for Logs is required to be deployed. Fluent Bit works as a log collector within the cluster. Configuring Fluent Bit to forward logs to Aria Operations for Logs by specifying the destination IP within the Fluent Bit configuration. It is recommended to deploy FluentBit as a DaemonSet to ensure it runs on every node, guaranteeing comprehensive log collection across the cluster. By default, Aria Operations for Logs communicates over TCP port 514.
Firewall and Ports – Direct Ingestion
Aria Operations for Logs offers the capability to gather logs from diverse endpoints, utilizing different ports and protocols.When employing syslog for log collection, it establishes TCP or UDP connections on port 514. Additionally, for endpoints utilizing the Ingestion API, Aria Operations for Logs utilizes port 9000 without SSL encryption and port 9543 with SSL encryption for data ingestion.
Aria Operations for Logs Forwarder
Aria Operations for Logs server is the collection server deployed in the central datacenter, A Forwarder is another instance of a general “Aria Operations for Logs server” but configured as a Forwarder to aggregate and forward the logs to the central datacenter. It is common to deploy a Forwarder at edge datacenter which forwards its logs to a central Aria Operations for Logs instance.
Benefit of using a Forwarder
Forwarders using the ingestion API support sending aggregated logs. encrypted events
Forwarders supports log archiving by allowing selective logging.
Forwarders can serve as a backup for logs in case of a network failure.
Forwarders make it possible to support variable retention periods
Forwarders can be used as an alternative to archiving
Forwarders using the ingestion API can add metadata to events
Firewall and Ports
The Communication Port that a Forwarder uses to send the traffic to the central site is 9000/9543 which are ports used by Aria operations internal CFAPI Protocol.
Sizing and Capacity Planning:
Properly sizing Aria Operations for Logs is critical for optimal performance. It presents various characteristics, including log ingest rate, IOPS, syslog connections, and event per second.
Log Ingest Rate: This denotes the number of logs processed per second.
IOPS: This reflects the storage performance necessary to write the log ingest rate effectively.
Syslog Connections: These signify the active syslog connections to Aria Operations for Logs, with each connection originating from a source device or application.
Events Per Second: This measures the rate at which events or log entries are generated by monitored systems and applications.
Accurately estimating these metrics is crucial for determining the appropriate hardware resources, such as CPU, memory, and storage required to support the anticipated workload and scale of the log management and analysis infrastructure.Below we will discuss sizing and capacity considerations for both the Methods i.e. Direct Ingestion and with Aria Operations for Log Forwarder.
Direct Ingestion
In Direct Ingestion, we directly ingest logs using syslog connections. Each edge site may have multiple syslog connections.Properly handling these connections in the central Aria Operations for Log cluster requires appropriate sizing.This can be accomplished using the vrlisizer tool. By providing inventory details as input, the tool offers suggestions on the ideal size of data nodes to be utilized in the Aria Operations for Logs cluster.
Sizing and Capacity calculation for central Aria cluster:Aria Operations for Logs Cluster supports a minimum of 3 and maximum of 18 worker nodes. The number of events per second and connections increases linearly with the number of nodes. For example the ingestion in an 18-node cluster is 13500 syslog connections, 270,000 events per second (EPS), or 4 TB of events per day. Based on the suggestion provided by the vrlisizer, preset sizes can be selected.
Forwarder
The Forwarder installed at the edge site has several advantages, as mentioned before. It also works as a syslog aggregator, which means it decreases the number of syslog connections needed for the uplink. This is really useful when there are many syslog endpoints at an edge site, which could otherwise consume a lot of resources at the central site. By using a Forwarder, we can cut down on the number of syslog connections and manage the resource usage more efficiently.
Sizing and Capacity calculation for a Forwarder
Preset Size |
Log Ingest Rate |
Virtual CPUs |
Memory |
IOPS |
Syslog Connections |
Events per Second |
|---|---|---|---|---|---|---|
Small |
30 GB/day |
4 |
8 GB |
500 |
100 |
2000 |
Medium |
75 GB/day |
8 |
16 GB |
1000 |
250 |
5000 |
large |
225 GB/day |
16 |
32 GB |
1500 |
750 |
15000 |
The Aria Operations for Logs Forwarder comes in preset sizes designed to handle different levels of log ingestion.Each size requires at least 530 GB of storage space. However, because it needs a lot of disk space, which might be limited on edge sites, setting up a Forwarder can be difficult.
It's recommended to use an Aria Operations for Logs Forwarder in places where memory limit is not a problem. It would be advisable to utilize single Aria Operations for Logs Forwarder deployed on a site to be consumed by multiple sites. An Aria Operations for Logs Forwarder can also be deployed in a cluster fashion for high availability.
ID |
Design Decision |
Design Justification |
Design Implication |
|---|---|---|---|
1 |
Directly Ingest the infra and other logs to the server deployed in central datacenter |
Send the logs to the Central server without any extra hops. |
This saves the extra resource that might be needed in case of Forwarder. |
2 |
Deploy a Forwarder in the Edge Site where capabilities like Syslog aggregation, backup, buffering and archiving required |
A Forwarder is needed with large deployments where during a Wan outage it can provide high amount buffering and backup, |
Forwarder will consume 530Gb of Storage resources on Edge |
Logging Sources and Log types
Aria Operations for Logs supports various logging sources and log types, enabling organizations to centralize and analyse logs from diverse environments. Here are some common logging sources and log types that can be consumed by Aria Operations for Logs
Infrastructure Logs:
VMware vSphere: includes ESXi hosts, vCenter Server, and virtual machines etc
Networking Devices: includes Logs from network devices such as routers, switches, and firewalls.
Storage Systems: Logs from storage arrays and storage management systems.
Guest Operating System Logs:
Linux: System logs, application logs, and kernel logs
Windows: Event logs, application logs, and system logs
Application Logs: -
Databases: includes logs from MySQL, PostgreSQL, Microsoft SQL Server, and Oracle.
Web Servers: includes Logs from Apache HTTP Server, Nginx, Microsoft IIS, and Tomcat.
Middleware: includes Logs from Apache Kafka, RabbitMQ, and Redis.
Custom Applications: Includes Logs generated by custom applications
TKG Cluster Logs:
Control Plane Logs: covers logs like Kubernetes API server, Controller, Scheduler logs
Worker Node Logs: covers kubelet, kubeproxy logs
Application Logs: Pods and application logs
Cluster Networking Logs : includes logs from CNI, ingress etc
ID |
Design Decision |
Design Justification |
Design Implication |
|---|---|---|---|
3 |
Install Dell - iDRAC |
provides functionality that helps you deploy, update, monitor and maintain Dell PowerEdge servers with or without a systems management software agent |
reduces or eliminates the need for administrators to physically visit the server. |
4 |
Install VMware - vSphere |
insight into your vCenter Server and ESXi logs |
None |
5 |
Install VMware - vSAN |
insight into your vSAN logs, |
None |
6 |
Install VMware Aria Operations for Logs Content Pack |
enhance the presentation of log data and facilitate analysis of all logs redirected from the Aria Operations for Logs |
None |
7 |
Install VMware Tanzu Kubernetes Grid Multicloud |
provides content for API server logs, controller logs, scheduler logs, pod related logs, and so on. This content pack supports Fluent Bit integration. |
None |
8 |
Install General |
Basic content pack providing capabilities for Aria Operations for Logs |
None |
Collection Agent Syslog and LI agent
Logs from endpoints are gathered via syslog and then transmitted to the server. On the other hand, the LI Agent employs the proprietary CFAPI, a VMware-specific protocol, for log collection. Additionally, the LI Agent offers support for syslog, enabling both endpoint log collection and relay to syslog servers following the syslog protocol. the Aria LI Agent, designed specifically for Aria environments, provides some additional customization options, log parsing capabilities, filtering, and security features.
Both the LI Agent and syslog are extensively utilized in content packs for log collection, monitoring, and management through default user interfaces provided within the content pack ecosystem. Due to its VMware proprietary nature, the LI Agent comprehensively understands VMware properties and offers enhanced log parsing capabilities.
In contrast, syslog, being a generic protocol, is straightforward to use and configure, requiring minimal setup. Furthermore, syslog deployment does not necessitate additional agent installations, simplifying the deployment process. For ECS 3.0, we recommend utilizing syslog due to its ease of deployment and versatility.
ID |
Design Decision |
Design Justification |
Design Implication |
|---|---|---|---|
9 |
Use Syslog for the VM and Edge Applications |
Easy to integrate syslog can be directly configured with Aria Operations for Logs cluster. |
None |
10 |
If li agent based content pack for Edge Application is already available in Marketplace. Use Content Packs |
Content packs which are built on Li agent has additional tagging which are useful. |
reduces or eliminates the need for administrators to physically visit the server. |
11 |
By Default Windows doesn’t support syslog and needs additional application to convert it. Use Li agent where Window VM is required. |
Window VM does not support syslog it uses event log to report the logs. |
Require a li agent to be configured in the windows machine |
Wan Outage
When WAN outage occurs, in the context of direct data ingestion, which involves transferring data directly from a source to a destination without intermediate storage or processing, it can lead to potential data loss. Li agent provides 2GB of buffer space, traditional syslog supports 3Mb of buffering, tools such as rsyslog or syslog-ng come with advanced buffering capabilities.
Roles and Permissions in Aria Operations for Logs
Permissions and access levels are associated with Edge Personas, and control the allowed actions in Aria Operations for Logs. Permissions apply to particular administrative or user tasks. The predefined roles have a fixed set of permissions. You can modify these permissions for all predefined roles except the Super Admin role.
Additionally, you can also create custom roles according to different Edge Persona and assign permissions with access levels according to your requirement. More on Roles and Permissions can be found on Aria Operations for Logs Roles.
