You can configure VMware Tanzu Application Service for VMs (TAS for VMs) to forward logs to remote endpoints using the syslog protocol defined in RFC 5424. For more information about enabling log forwarding, see Configure system logging in Configuring TAS for VMs.
TAS for VMs annotates forwarded messages with structured data. This structured data identifies the originating BOSH Director, deployment, instance group, availability zone, and instance ID. All logs forwarded from BOSH jobs have their PRI set to 14, representing Facility: user-level messages and Informational: informational messages, as defined in RFC 5424 Section 6.2.1. This PRI value might not reflect the originally intended PRI of the log.
Logs forwarded from other sources, such as kernel logs, retain their original PRI value.
The following table describes the log line Structured Data:
| Structured Data | Description |
|---|---|
| ENTERPRISE_NUMBER | TAS for VMs’s private enterprise number, 47450, as defined in RFC 5424 Section 7.2.2 |
| DIRECTOR | The name of the BOSH Director managing the deployment. |
| DEPLOYMENT | The name of the BOSH deployment. |
| INSTANCE_GROUP | The name of the BOSH instance group. |
| AVAILABILITY_ZONE | The name of the BOSH availability zone. |
| ID | The BOSH GUID. |
| ENVIRONMENT | An optional custom label to identify the BOSH environment. |
Log lines use the following format:
<$PRI>$VERSION $TIMESTAMP $HOST $APP_NAME $PROC_ID $MSG_ID
[instance@ENTERPRISE_NUMBER director="$DIRECTOR" deployment="$DEPLOYMENT"
group="$INSTANCE_GROUP" az="$AVAILABILITY_ZONE" id="$ID"
environment="$ENVIRONMENT"] $MESSAGE
Example log messages:
<14>1 2017-01-25T13:25:03.18377Z 192.0.2.10 etcd - - [instance@47450
director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] [INFO] the leader is
[https://diego-database-0.etcd.service.cf.internal:4001]
<14>1 2017-01-25T13:25:03.184491Z 192.0.2.10 bbs - - [instance@47450
director="test-env" deployment="cf-c42ae2c4dfb6f67b6c27" group="diego_database" az="us-west1-a"
id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"]
{"timestamp":"1485350702.539694548","source":"bbs","message":
"bbs.request.start-actual-lrp.starting","log_level":1,"data":
{"actual_lrp_instance_key":{"instance_guid":
"271f71c7-4119-4490-619f-4f44694717c0","cell_id":
"diego_cell-2-41f21178-d619-4976-901c-325bc2d0d11d"},"actual_lrp_key":
{"process_guid":"1545ce89-01e6-4b8f-9cb1-5654a3ecae10-137e7eb4-12de-457d-
8e3e-1258e5a74687","index":5,"domain":"cf-apps"},"method":"POST","net_info":
{"address":"192.0.2.12","ports":[{"container_port":8080,"host_port":61532},
{"container_port":2222,"host_port":61533}]},"request":
"/v1/actual_lrps/start","session":"418.1"}}
Edit which logs TAS for VMs forwards
When you enable log forwarding, all log lines written to the /var/vcap/sys/log directories in all TAS for VMs virtual machines (VMs) will be forwarded to your configured external syslog aggregation service.
To configure TAS for VMs to forward a subset of logs instead of forwarding all logs:
-
Go to the Tanzu Operations Manager Installation Dashboard.
-
Click the TAS for VMs tile.
-
Select System Logging.
-
In the Custom rsyslog configuration field, enter a custom syslog rule. See the following custom syslog rules example.
-
Click Save.
The custom rsyslog rules shown here are written in RainerScript. The custom rules are inserted before the rule that forwards logs. The stop command, stop, prevents logs from reaching the forwarding rule. This filters out these logs.
Logs filtered out before forwarding remain on the local disk, where the BOSH job originally wrote them. These logs remain on the local disk only until BOSH Director recreates the VMs. You can access these logs from Tanzu Operations Manager or through SSH.
TAS for VMs requires a valid custom rule to forward logs. If your custom rule contains syntax errors, TAS for VMs forwards no logs.
Forward only logs from a certain job
This rule filters out logs unless they originate from the uaa job:
if ($app-name != "uaa") then stop
Exclude logs with certain content
This rule filters out logs that contain “DEBUG” in the body.
if ($msg contains "DEBUG") then stop
The previous example contains “DEBUG” in the message body. Not all logs intended for debugging purposes contain the string “DEBUG” in the message body.
