Configuring logging in TAS for VMs

VMware Tanzu Application Service for VMs (TAS for VMs) generates many logs. Learn about the log types and how to forward system logs to an external aggregator service, how to scale Loggregator component VMs to keep up with app log volume, and how to manage app traffic logging.

For more information about Loggregator components, see Loggregator architecture.

System logs, app logs, app traffic logs

TAS for VMs generates two types of logs, system logs from TAS for VMs components and app logs from hosted apps, as differentiated in the table below:

Log type	Originate from	Follow format	Stream from	Can stream out to (configurable)	Visible to
System logs	Platform components	Syslog standard	rsyslog agent	Component syslog drain	Operators
App logs	Hosted apps	Format is up to the developer	Firehose1	External data platform, optionally by using nozzles	Developers and Operators
		Converted to syslog standard	Syslog Agent	External syslog drain

¹The Loggregator Firehose also streams component metrics.

App traffic logs are system logs. When app containers communicate, or attempt to communicate, their host Diego Cells generate app traffic logs. App traffic logs are system logs, not app logs. These logs come from host Diego Cells, not apps, and they carry no information from within the app. App traffic logs only show app communication behavior, as detected from outside by the host Diego Cell.

Log cache

Log Cache is a Loggregator feature that you can filter and query app logs through a CLI plug-in or API endpoints. Cached app logs are available on demand. You do not need to stream them continuously.

Example uses of the Log Cache CLI plug-in

To access cached logs with the Log Cache CLI plug-in, you must first download and install the plug-in.

To download the Log Cache CLI plug-in, see the cf CLI plug-ins page on the Cloud Foundry website.

After you have installed the plug-in, the basic command to access cached app logs is:

cf tail OPTIONS APP-NAME-OR-ID

Where:

• OPTIONS are the flags you use to filter app logs.
• APP-NAME-OR-ID is the name or source ID of your app.

Some flags you can use Log Cache to filter app logs are:

• --start-time: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with --end-time to view logs within a time period.

• --end-time: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with --start-time to view logs within a time period.

• --json: Output logs in JSON.

• --follow: Append exported logs to stdout.

For more information about using the Log Cache CLI, see Log Cache CLI: Usage on GitHub.

Example uses of the Log Cache API

The Log Cache API is hosted on TAS for VMs, and references your system domain to return responses. The root URL for API calls is https://log-cache.SYSTEM-DOMAIN, where SYSTEM-DOMAIN is your system domain.

The basic call to access and filter cached app logs is:

GET https://log-cache.SYSTEM-DOMAIN/v1/read/APP-ID

Where:

• SYSTEM-DOMAIN is your system domain.
• APP-ID is the source ID of your app.

Append the following parameters to your GET call to customize app logs:

• start_time: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with end_time to view logs within a time period.

• end_time: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with start_time to view logs within a time period.

• envelope_types: Filters by Envelope Type. The available filters are: LOG, COUNTER, GAUGE, TIMER, and EVENT. Set an envelope type filter to emit logs of only that type. Specify this parameter multiple times to include more types.

• limit: Sets a maximum number of envelopes to request. The max limit is 1000. This value defaults to 100.

More API parameters are available to customize retrieved app logs. For more information, see Log Cache: RESTful API Gateway.

Activate syslog forwarding

In the System Logging pane, you can configure system logging in TAS for VMs to forward log messages from TAS for VMs component VMs to an external service. VMware recommends forwarding logs to an external service for use in troubleshooting. If you do not fill these fields, platform logs are not forwarded but remain available on the component VMs and for download through Tanzu Operations Manager.

Note This procedure explains how to configure system logging for TAS for VMs component VMs. To forward logs from Operations Manager tiles to an external service, you must also configure system logging in each tile. For more information about configuring system logging, see the documentation for the given tiles.

To configure the System Logging pane:

1. Select System Logging.

2. For Syslog server address, enter the hostname or IP address of the syslog server.

3. For Syslog server port, enter the port of the syslog server. The default port for a syslog server is 514.

   Important The host must be reachable from the TAS for VMs network and accept UDP or TCP connections. Ensure that the syslog server listens on external interfaces.

4. For Transport protocol, select a transport protocol for log forwarding.

5. (Optional) For Environment identifier, enter a custom label (e.g. the name of your foundation) to include in the structured data of forwarded syslog messages with the parameter name environment.

6. For TLS encryption, select one of the following options:

   • Use: Configures TAS for VMs to use TLS encryption when forwarding logs to a remote server. If you select this option, configure the following fields:
   • For Permitted peer, enter either the name or SHA1 fingerprint of the remote peer.
   • For CA certificate, enter the CA certificate for the remote server.
   • Do not use: TAS for VMs does not use TLS encryption when forwarding logs to a remote server.

7. (Optional) To include security events in the log stream, select the Log Cloud Controller security events check box. When this check box is selected, TAS for VMs logs all API requests in the Common Event Format (CEF), including the endpoint, user, source IP address, and request result.

8. (Optional) To transmit logs over TCP, select the Use TCP for file forwarding local transport check box. This prevents log truncation, but might cause performance issues.

9. The Do not forward debug logs check box is selected by default. To forward DEBUG syslog messages to an external service, deselect the check box.

   Note Some TAS for VMs components generate a high volume of DEBUG syslog messages. Selecting the Do not forward debug logs check box prevents TAS for VMs components from forwarding the DEBUG syslog messages to external services. However, TAS for VMs still writes the messages to the local disk.

10. For Custom rsyslog configuration, enter a custom syslog rule. For more information about adding custom syslog rules, see Customizing platform log forwarding.

11. Configure how TAS for VMs emits app logs and app metrics for ingestion in your deployment. The options include:

    • Use existing Firehose integrations for app metric and app log ingestion.
    • Preserve existing Firehose integrations for app metrics, but use an alternate method for app log ingestion.
    • Deactivate all Firehose integrations and use alternate methods for both app log and app metric ingestion.
      
      The following table provides the configuration procedures for each option. For more information about each field, see the Field Descriptions table.

    Option	Configuration Procedure
    Use existing Firehose app log and metrics integrations	Activate Enable V1 Firehose. Activate Enable V2 Firehose. Deactivate Do not forward app logs to the Firehose. (Optional) Configure Aggregate syslog drain destinations.
    Preserve existing Firehose integrations for app metrics, but use an alternate method for app log ingestion	Caution Do not use this option if your deployment depends on partner log integrations. Activate Enable V1 Firehose. Activate Enable V2 Firehose. Activate Do not forward app logs to the Firehose. Configure Aggregate syslog drain destinations.
    Deactivate all Firehose integrations and use alternate methods for both app log and app metric ingestion	Caution Do not use this option if your deployment depends on any of these: Service tile metrics Healthwatch or App Metrics Partner log or metric integrations Deactivate Enable V1 Firehose. Deactivate Enable V2 Firehose. Deactivate Do not forward app logs to the Firehose. Configure Aggregate syslog drain destinations. Deactivate the Smoke Test errand. Otherwise, the TAS for VMs deployment fails. To deactivate errands, see Configure Errands below.

     
    Field Descriptions:
    
    The following table provides more details on field values:

    Field Name	Description
    Enable V1 Firehose	Activated by default. When this checkbox is activated, logs and metrics flow to the Loggregator V1 Firehose.
    Enable V2 Firehose	Activated by default. When this checkbox is activated, logs and metrics flow to the Loggregator V2 Firehose.
    Send default Loggregator drain metadata	Activated by default. When this checkbox is activated, TAS for VMs sends all metadata in app and aggregate syslog drains. Deactivating this checkbox can reduce logging to external databases by up to 50 percent.
    Do not forward app logs to the Firehose	Deactivated by default. When this checkbox is activated, TAS for VMs prevents the Firehose from emitting app logs, but still allows the Firehose to emit app metrics. Deactivating logs in Firehose helps reduce the load on TAS for VMs by allowing you to scale down Doppler and Traffic Controller VMs.
    Aggregate syslog drain destinations	Specify zero or more aggregate syslog drains. Aggregate syslog drains forward all app logs on your foundation to the endpoints that you provide in this field: Syslog Drain URL: Enter the endpoint in the format: syslog://HOSTNAME:PORT. To use TLS for sending logs, specify syslog-tls://HOSTNAME:PORT. Certificate Authority: (Optional) Enter the Certificate Authority (CA) certificate to trust for the endpoint. Client Certificate: (Optional) Enter the client certificate and private key to be used for mTLS communication with the endpoint. Caution In earlier versions of TAS for VMs, multiple aggregate syslog drains were entered by using a comma-separated list. This is no longer supported and endpoints should be added individually.

12. (Optional) For System metrics scrape interval, the default value is 1m, which configures TAS for VMs to send BOSH system metrics to your logging endpoint once per minute. To configure TAS for VMs to send metrics more or less frequently, modify the value in this field. For example, enter 2m to send metrics every two minutes, or 10s to send metrics every ten seconds. VMware recommends configuring a minimum interval of five seconds, or 5s.

13. (Optional) For OpenTelemetry Collector Metric Exporters (beta), the default value is empty, which deactivates the beta OpenTelemetry Aggregate Metric Egress support. To configure TAS for VMs to send metrics over the OpenTelemetry protocol, enter valid OpenTelemetry Collector Exporter YAML configuration in this text box. See Configuring the OpenTelemetry Collector for examples of how to configure exporters. Currently TAS for VMs provides support for a limited number of OpenTelemetry Collector Exporters, including the OTLP exporter. Note that this feature is in beta and may still change in significant ways.

    Caution If configuring a metric exporter that listens on a port, ensure that the port is not claimed by a TAS for VMs component on any of the VMs in your deployment.

14. Click Save.

To configure Tanzu Operations Manager for system logging, see Settings page in Using the Tanzu Operations Manager interface.

Scale Loggregator Firehose system

Apps constantly generate app logs and TAS for VMs platform components constantly generate component metrics. The Loggregator Firehose system combines these data streams and handles them as follows. For more information, see Loggregator Firehose Architecture.

• The Loggregator agent running on each component or app VM collects and sends this data out to Doppler components.

• Doppler components temporarily buffer the data before periodically forwarding it to the Traffic Controller. When the log and metrics data input to a Doppler exceeds its buffer size for a given interval, data can be lost.

• The Traffic Controller / Reverse Log Proxy serve the aggregated data stream to clients.

Use the following instructions to scale the Loggregator Firehose system. For guidance on monitoring and capacity planning, see Monitoring TAS for VMs.

Add component VM instances

To add component VM instances for Loggregator components:

1. Go to the Tanzu Operations Manager Installation Dashboard.

2. Click the TAS for VMs tile.

3. Select Resource Config.

4. Increase the number in the Instances column of the component you want to scale. You can add instances for the following Loggregator components:

   • Loggregator Trafficcontroller
   • Doppler Server
   • Log Cache

5. Click Save.

6. Return to the Tanzu Operations Manager Installation Dashboard.

7. Click Review Pending Changes.

8. Click Apply Changes.

Scale Loggregator Shared Nothing system

VMware recommends the newer Shared Nothing logging and metrics architecture as an alternative to the Loggregator Firehose architecture.

The Shared Nothing architecture egresses logs and metrics from the platform directly from each VM as opposed to the more centralized approach of the Loggregator Firehose architecture. Egressing directly from the VMs where logs and metrics are generated has better scaling characteristics, may reduce costs and should be preferred for new deployments.

For more information, see Shared Nothing Architecture.

• The Syslog agent running on each component or app VM collects and sends this data out to Log Cache and any configured external syslog destination.

• The OpenTelemetry Collector (beta) running on each component or app VM egresses metrics via the configured exporters.

When using the Shared Nothing architecture you can potentially scale down the VMs associated with the Loggregator Firehose architecture. See Scaling down Logging and Metrics infrastructure for more information.

App traffic logging

App traffic logging generates logs when app containers communicate with each other directly, or attempt to communicate, by container-to-container networking (C2C) policies and App Security Groups (ASGs). For more information about C2C policies, see Container-to-Container Networking versus ASGs in Container-to-Container Networking. For more information about ASGs, see App Security Groups.

With App traffic logging, network security teams audit C2C traffic, by seeing allowed and denied packets, without needing access to the Cloud Controller or the apps themselves.

Activate app traffic logging

To activate app traffic logging:

1. Go to the Tanzu Operations Manager Installation Dashboard.

2. Click the TAS for VMs tile.

3. Select the Networking pane.

4. Activate the Log traffic for all accepted and denied app packets check box.

5. Click Save.

6. Return to the Tanzu Operations Manager Installation Dashboard.

7. Click Review Pending Changes.

8. Click Apply Changes.

App logging behavior

App traffic logging generates log messages as follows:

• TCP traffic: Logs the first packet of every new TCP connection.

• UDP traffic: Logs UDP packets sent and received, up to a maximum per-second rate for each container. Set this rate limit in the UDP logging interval field in the Networking pane of the TAS for VMs tile. The default rate limit is 100 packets per second.

• Packets denied: Logs packets blocked by either a container-specific networking policy or by ASG rules applied across the space, org, or deployment. Logs packet denials up to a maximum per-second rate for each container, set in the Denied logging interval field in the Networking pane of the TAS for VMs tile. The default rate is 1 (packet per second). For more information about container-specific networking policies, see Policies in Container-to-Container Networking.

App traffic log format and contents

App traffic logs are formatted as described in the silk-release Traffic logging documentation, following the iptables-logger format but without line breaks.

For example, the first part of an app traffic log line looks like:

{"timestamp": "2020-06-23T11:36:01.710452019Z", "source": "cfnetworking.iptables", "message": "cfnetworking.iptables.ingress-allowed", "log_level": 1, "data": { "destination": { "container_id": "d5978989-1401-49ff-46cd-33e5","app_guid": "bc6f229d-5e4a-4c41-a63f-e8795496c283",.

Each log message includes:

• A timestamp, either in RFC3339 format or in epoch format. The format depends upon the configuration of the Timestamp format for component logs in the System Logging pane of the TAS for VMs tile

• The GUID for the source or destination app that sent or was designated to receive the packet

• The protocol of the communication, TCP or UDP

• GUIDs for the container, space, and org running the source or destination app

• IP addresses and ports for both source and destination apps

• A message field recording whether the packet was allowed or denied, with one of the following four possibilities:

  • ASG allowed packet to exit source app container
  • C2C policy allowed packet to enter destination app container
  • ASG prevented packet from exiting source app container
  • C2C policy prevented packet from entering destination app container

• Additional information described in the silk-release documentation

Causes of denied packet

You can determine whether a denied-packet log results from a container networking policy or an ASG rule as follows:

• Container networking policy: Log message string includes ingress-denied and packet direction is ingress.

• ASG rule: Log message string includes egress-denied and packet direction is egress.

Global vs. ASG-level app traffic logging

TAS for VMs supports two mechanisms for enabling app traffic logging. Enabling the Log traffic for all accepted and denied app packets check box in the Networking pane of the TAS for VMs tile activates app traffic logging globally for all ASGs and container policies. Setting the log property of an ASG to true activates app traffic logging at the individual ASG level. For more information about setting the log property of an ASG to true, see The Structure and Attributes of ASGs in App Security Groups.

Because these two mechanisms operate independently, TAS for VMs generates duplicate logs when app traffic logging is activated globally and an ASG’s log property is set to true. To avoid duplicate logs, VMware recommends setting the log property to false for all ASGs, or leaving it out entirely, when app traffic logging is activated globally. For more information, see App Traffic Logging.

To focus on specific ASGs for log analysis, VMware recommends enabling app traffic logs globally and using a logging platform to filter traffic logs by ASG, rather than setting log at the individual ASG level.