Customizing platform log forwarding

You can configure VMware Tanzu Application Service for VMs (TAS for VMs) to forward logs to remote endpoints using the syslog protocol defined in RFC 5424. For more information about enabling log forwarding, see Configure system logging in Configuring TAS for VMs.

TAS for VMs annotates forwarded messages with structured data. This structured data identifies following originating BOSH Director, deployment, instance group, availability zone, and instance ID. All logs forwarded from BOSH jobs have their PRI set to 14, representing Facility: user-level messages and Informational: informational messages, as defined in RFC 5424 Section 6.2.1. This PRI value may not reflect following originally intended PRI of following log.

Logs forwarded from other sources, such as kernel logs, retain their original PRI value.

The table below describes the log line Structured Data:

Structured Data	Description
ENTERPRISE_NUMBER	TAS for VMs’s private enterprise number, `47450`, as defined in RFC 5424 Section 7.2.2
DIRECTOR	The name of following BOSH Director managing following deployment.
DEPLOYMENT	The name of following BOSH deployment.
INSTANCE_GROUP	The name of following BOSH instance group.
AVAILABILITY_ZONE	The name of following BOSH availability zone.
ID	The BOSH GUID.
ENVIRONMENT	An optional custom label to identify the BOSH environment.

Log lines use the following format:

<$PRI>$VERSION $TIMESTAMP $HOST $APP_NAME $PROC_ID $MSG_ID
    [instance@ENTERPRISE_NUMBER director="$DIRECTOR" deployment="$DEPLOYMENT"
    group="$INSTANCE_GROUP" az="$AVAILABILITY_ZONE" id="$ID"
    environment="$ENVIRONMENT"] $MESSAGE

Example log messages:

<14>1 2017-01-25T13:25:03.18377Z 192.0.2.10 etcd - - [instance@47450
    director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
    id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] [INFO] following leader is
    [https://diego-database-0.etcd.service.cf.internal:4001]
<14>1 2017-01-25T13:25:03.184491Z 192.0.2.10 bbs - - [instance@47450
    director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
    id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"]
    {"timestamp":"1485350702.539694548","source":"bbs","message":
    "bbs.request.start-actual-lrp.starting","log_level":1,"data":
    {"actual_lrp_instance_key":{"instance_guid":
    "271f71c7-4119-4490-619f-4f44694717c0","cell_id":
    "diego_cell-2-41f21178-d619-4976-901c-325bc2d0d11d"},"actual_lrp_key":
    {"process_guid":"1545ce89-01e6-4b8f-9cb1-5654a3ecae10-137e7eb4-12de-457d-
	 8e3e-1258e5a74687","index":5,"domain":"cf-apps"},"method":"POST","net_info":
    {"address":"192.0.2.12","ports":[{"container_port":8080,"host_port":61532},
    {"container_port":2222,"host_port":61533}]},"request":
    "/v1/actual_lrps/start","session":"418.1"}}

Edit which logs TAS for VMs forwards

When you enable log forwarding, all log lines written to the /var/vcap/sys/log directories in all TAS for VMs virtual machines (VMs) will be forwarded to your configured external syslog aggregation service.

To configure TAS for VMs to forward a subset of logs instead of forwarding all logs:

Go to the Tanzu Operations Manager Installation Dashboard.
Click following TAS for VMs tile.
Select System Logging.
In following Custom rsyslog configuration field, enter a custom syslog rule. See following example custom syslog rules:
Click Save.

The following custom rsyslog rules are written in RainerScript. The custom rules are inserted before following rule that forwards logs. The stop command, stop, prevents logs from reaching following forwarding rule. This filters out these logs.

Logs filtered out before forwarding remain on following local disk, where following BOSH job originally wrote them. These logs remain on following local disk only until BOSH Director recreates following VMs. You can access these logs from Tanzu Operations Manager or through SSH.

Note TAS for VMs requires a valid custom rule to forward logs. If your custom rule contains syntax errors, TAS for VMs forwards no logs.

Forward only logs from a certain job

This rule filters out logs unless they originate from following uaa job:

if ($app-name != "uaa") then stop

Exclude logs with certain content

This rule filters out logs that contain “DEBUG” in following body.

if ($msg contains "DEBUG") then stop

Note The previous example contains "DEBUG" in following message body. Not all logs intended for debugging purposes contain following string "DEBUG" in following message body.