Streaming app logs to Azure OMS Log Analytics

Here are instructions for integrating your VMware Tanzu Application Service for VMs (TAS for VMs) apps with OMS Log Analytics in the Microsoft documentation.

Operations Management Suite (OMS) Log Analytics is a monitoring service for Microsoft Azure. The OMS Log Analytics Firehose Nozzle is a TAS for VMs component that forwards metrics from the Loggregator Firehose to OMS Log Analytics.

This topic assumes you are using the latest version of the Cloud Foundry Command Line Interface (cf CLI) and a working TAS for VMs deployment on Azure.

Step 1: Create an OMS workspace in Azure

To create an OMS workspace, see the Microsoft documentation in the Microsoft Azure documentation.

Step 2: Deploy the nozzle to TAS for VMs

To deploy the OMS Log Analytics Firehose nozzle to TAS for VMs:

Authenticate to your TAS for VMs instance. For more information, see Creating and Managing Users with the UAA CLI (UAAC) and Orgs, spaces, roles, and permissions. Run:
```
cf login -a https://api.YOUR-DOMAIN -u YOUR-USERNAME --skip-ssl-validation
```
Where:
- YOUR-DOMAIN is your domain.
- YOUR-USERNAME is your TAS for VMs user name.
To create a new TAS for VMs user and grant it access to the Loggregator Firehose using the UAA CLI (UAAC):
1. Target your UAA server by running:
```
uaac target uaa.YOUR-DOMAIN --skip-ssl-validation
```
  Where YOUR-DOMAIN is your domain.
2. Obtain an access token for the admin client by running:
```
uaac token client get admin
```
3. Create a new user by running:
```
uaac user add USERNAME -p PASSWORD  --email EMAIL
```
  Where:
  - USERNAME is a new user name.
  - PASSWORD is a password.
  - EMAIL is an email address.
4. Grant the new user admin permissions by running:
```
uaac member add cloud_controller.admin USERNAME
```
  Where USERNAME is the user name you set in an earlier step.
5. Grant the new user permission to read logs from the Loggregator Firehose endpoint by running:
```
uaac member add doppler.firehose USERNAME
```
  Where USERNAME is the user name you set.
Download the OMS Log Analytics Firehose Nozzle BOSH release from GitHub. Clone the repository and navigate to the oms-log-analytics-firehose-nozzle directory by running:
```
git clone https://github.com/Azure/oms-log-analytics-firehose-nozzle.git
cd oms-log-analytics-firehose-nozzle
```

Set the environment variables in the OMS Log Analytics Firehose Nozzle manifest:

Environment variable	Description
applications: - name: oms_nozzle ... env: OMS_WORKSPACE: YOUR-WORKSPACE-ID OMS_KEY: YOUR-OMS-KEY	Enter the ID and key value for your OMS workspace.
`OMS_POST_TIMEOUT: 10s`	(Optional) Set the HTTP post timeout for sending events to OMS LogmAnalytics. The default value is 10 seconds.
`OMS_BATCH_TIME: 10s`	(Optional) Set the interval for posting a batch to OMS. The default value is 10 seconds. For more information, see Configure Additional Logging.
`OMS_MAX_MSG_NUM_PER_BATCH: 1000`	(Optional) Set the maximum number of messages to include in an OMS batch. The default amount is 1000. For more information, see Configure Additional Logging.
FIREHOSE_USER: YOUR-FIREHOSE-USER FIREHOSE_USER_PASSWORD: YOUR-FIREHOSE-PASSWORD	Enter the user name and password for the Firehose user you created in Step 2c.
`API_ADDR: https://api.YOUR-DOMAIN`	Enter the URL of your API endpoint.
`DOPPLER_ADDR: wss://doppler.YOUR-DOMAIN:443`	Enter the URL of your Loggregator Traffic Controller endpoint.
`EVENT_FILTER: YOUR-LIST`	(Optional) Enter the event types you want to filter out in a comma-separated list. The valid event types are `METRIC`, `LOG`, and `HTTP`.
`IDLE_TIMEOUT: 60s`	(Optional) Set the duration for the Firehose keep-alive connection. The default time is 60 seconds.
`SKIP_SSL_VALIDATION: TRUE-OR-FALSE`	Set this value to `TRUE` to allow insecure connections to the UAA and the Traffic Controller. To block insecure connections to the UAA and Traffic Controller, set this value to `FALSE`.
`LOG_LEVEL: INFO`	(Optional) Change this value to increase or decrease the amount of logs. Valid log levels in increasing order include `INFO`, `ERROR`, and `DEBUG`. The default value is `INFO`.
`LOG_EVENT_COUNT: TRUE-OR-FALSE`	Set this value to `TRUE` to log the total count of events that the nozzle has sent, received, and lost. OMS logs this value as `CounterEvents`. For more information, see Configure Additional Logging.
`LOG_EVENT_COUNT_INTERVAL: 60s`	(Optional) Set the time interval for logging the event count to OMS. The default interval is 60 seconds. For more information, see Configure Additional Logging.

Push the app by running:
```
cf push
```

Step 3: View logs in OMS Portal

Import the TAS for VMs OMS view to your OMS Portal to view visualized logs and metrics. You can also create alert rules for specific events.

The OMS view of TAS for VMs is not available in the OMS Solutions Gallery. You can add it manually to view your logs in OMS Portal.

Import the OMS view

To import the OMS view:

From the main OMS Overview page, go to View Designer.
Click Import.
Click Browse.
Select the Cloud Foundry (Preview).omsview file.
Save the view. The main OMS Overview page displays the Tile.
Click the Tile to view visualized metrics.

For more information, see the Microsoft documentation in the Azure documentation.

Create alert rules

For more information about OMS Log Analytics alerts, see the Microsoft documentation in the Azure documentation.

Set alert queries

This section includes example queries that operators can set in the OMS Portal.

This query alerts the operator when the nozzle sends a slowConsumerAlert to OMS:
```
Type=CF_ValueMetric_CL Name_s=slowConsumerAlert
```
This query alerts the operator when Loggregator sends an LGR to indicate problems with logging:
```
Type=CF_LogMessage_CL SourceType_s=LGR MessageType_s=ERR
```
This query alerts the operator when the number of lost events reaches a certain threshold, specified in the OMS Portal:
```
Type=CF_CounterEvent_CL Job_s=nozzle Name_s=eventsLost
```
This query alerts the operator when the nozzle receives the TruncatingBuffer.DroppedMessages CounterEvent:
```
Type=CF_CounterEvent_CL Name_s="TruncatingBuffer.DroppedMessages"
```

Step 4: Configure additional logging (optional)

OMS Log Analytics Firehose Nozzle forwards metrics from the Loggregator Firehose to OMS with minimal processing, but the nozzle can push additional metrics to OMS.

Logs sent and received, events lost

If you set the LOG_EVENT_COUNT environment variable to TRUE in the manifest, the nozzle periodically sends the count of sent, received, and lost events to OMS. The value you set for the LOG_EVENT_COUNT_INTERVAL specifies how frequently the nozzle sends the count.

The nozzle does not count CounterEvents in the sent, received, or lost event count.

The nozzle sends the count as a CounterEvent with one of these values for CounterKey:

CounterEvent	CounterKey
nozzle.stats.eventsReceived	The number of events the Firehose has received during the interval
nozzle.stats.eventsSent	The number of events the nozzle has sent to OMS during the interval
nozzle.stats.eventsLost	The number of events the nozzle has tried to send to OMS during the interval, but failed to send after 4 attempts

In most cases, the total count of eventsSent plus eventsLost is less than the total eventsReceived at the same time. The nozzle buffers some messages and posts them in a batch to OMS. Operators can adjust the buffer size by adjusting the OMS_BATCH_TIME and OMS_MAX_MSG_NUM_PER_BATCH environment variables in the manifest.

Log slow consumer alerts

Loggregator sends the nozzle a slowConsumerAlert when:

WebSocket sends the error code ClosePolicyViolation (1008).
The nozzle receives a CounterEvent with the value TruncatingBuffer.DroppedMessages.

In either case, the nozzle sends the slowConsumerAlert event to OMS as the following ValueMetric:

ValueMetric	MetricKey
nozzle.alert.slowConsumerAlert	1

The nozzle does not count ValueMetrics in the sent, received, or lost event count.

For more information, see the Slow Nozzle Alerts section of the Loggregator Guide for TAS for VMs Operators topic.

Step 5: Scale the deployment (optional)

Scale the nozzle

If the nozzle is unable to keep up with processing logs from the Firehose, Loggregator alerts the nozzle. When the nozzle receives the alert, it sends a slowConsumerAlert to OMS. If this happens, scaling up the nozzle minimizes data loss.

If an operator scales up their deployment, the Firehose evenly distributes events across all instances of the nozzle. For more information, see the Scaling Nozzles section of the Loggregator Guide for TAS for VMs Operators topic.

Operators can create an alert rule for the slowConsumerAlert message. For more information, see Create alert rules.

Scale Loggregator

Loggregator sends LGR log entries to indicate problems with logging. For more information, see Scaling Loggregator in Loggregator Guide for TAS for VMs Operators.

Operators can create an alert rule for the LGR message. For more information, see Create alert rules.