You can configure VMware Tanzu Application Service for VMs (TAS for VMs) to forward logs to remote endpoints using the syslog protocol defined in RFC 5424. For more information about enabling log forwarding, see Configure system logging in Configuring TAS for VMs.
TAS for VMs annotates forwarded messages with structured data. This structured data identifies the originating BOSH Director, deployment, instance group, availability zone, and instance ID. All logs forwarded from BOSH jobs have their PRI set to 14, representing Facility: user-level messages and Informational: informational messages, as defined in RFC 5424 Section 6.2.1. This PRI value may not reflect the originally intended PRI of the log.
Logs forwarded from other sources, such as kernel logs, retain their original PRI value.
The table below describes the log line Structured Data:
| Structured Data | Description |
|---|---|
| ENTERPRISE_NUMBER | TAS for VMs’s private enterprise number, 47450, as defined in RFC 5424 Section 7.2.2 |
| DIRECTOR | The name of the BOSH Director managing the deployment. |
| DEPLOYMENT | The name of the BOSH deployment. |
| INSTANCE_GROUP | The name of the BOSH instance group. |
| AVAILABILITY_ZONE | The name of the BOSH availability zone. |
| ID | The BOSH GUID. |
Log lines use the following format:
<$PRI>$VERSION $TIMESTAMP $HOST $APP_NAME $PROC_ID $MSG_ID
[instance@ENTERPRISE_NUMBER director="$DIRECTOR" deployment="$DEPLOYMENT"
group="$INSTANCE_GROUP" az="$AVAILABILITY_ZONE" id="$ID"] $MESSAGE
Example log messages:
<14>1 2017-01-25T13:25:03.18377Z 192.0.2.10 etcd - - [instance@47450
director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] [INFO] the leader is
[https://diego-database-0.etcd.service.cf.internal:4001]
<14>1 2017-01-25T13:25:03.184491Z 192.0.2.10 bbs - - [instance@47450
director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"]
{"timestamp":"1485350702.539694548","source":"bbs","message":
"bbs.request.start-actual-lrp.starting","log_level":1,"data":
{"actual_lrp_instance_key":{"instance_guid":
"271f71c7-4119-4490-619f-4f44694717c0","cell_id":
"diego_cell-2-41f21178-d619-4976-901c-325bc2d0d11d"},"actual_lrp_key":
{"process_guid":"1545ce89-01e6-4b8f-9cb1-5654a3ecae10-137e7eb4-12de-457d-
8e3e-1258e5a74687","index":5,"domain":"cf-apps"},"method":"POST","net_info":
{"address":"192.0.2.12","ports":[{"container_port":8080,"host_port":61532},
{"container_port":2222,"host_port":61533}]},"request":
"/v1/actual_lrps/start","session":"418.1"}}
Modify Which Logs TAS for VMs Forwards
When you enable log forwarding, TAS for VMs forwards all log lines written to the /var/vcap/sys/log directories on all Ops Manager virtual machines (VMs) to your configured External Syslog Aggregator endpoint by default.
To configure TAS for VMs to forward a subset of logs instead of forwarding all logs:
-
Go to the Ops Manager Installation Dashboard.
-
Click the TAS for VMs tile.
-
Select System Logging.
-
In the Custom rsyslog configuration field, enter a custom syslog rule. See the example custom syslog rules below.
-
Click Save.
The custom rsyslog rules shown below are written in RainerScript. The custom rules are inserted before the rule that forwards logs. The stop command, stop, prevents logs from reaching the forwarding rule. This filters out these logs.
Logs filtered out before forwarding remain on the local disk, where the BOSH job originally wrote them. These logs remain on the local disk only until BOSH Director recreates the VMs. You can access these logs from Ops Manager or through SSH.
Note: TAS for VMs requires a valid custom rule to forward logs. If your custom rule contains syntax errors, TAS for VMs forwards no logs.
Forward only logs from a certain job
This rule filters out logs unless they originate from the uaa job:
if ($app-name != "uaa") then stop
Exclude logs with certain content
This rule filters out logs that contain “DEBUG” in the body.
if ($msg contains "DEBUG") then stop
Note: The above example contains "DEBUG" in the message body. Not all logs intended for debugging purposes contain the string "DEBUG" in the message body.
