Logging Windows Worker Workloads

This topic describes how to install and configure components and integrations to capture VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows Worker Kubernetes cluster and worker node logs.

Prerequisites

Before starting the tasks in this topic:

Your environment must be a Windows 2019 data center with vRealize Log Insight (vRLI).
You must have an accessible container image registry.
Docker must be installed on the local machine you will be working from. For more information, see Prepare the Working Environment below.

Overview

This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.

In addition to deploying Fluent Bit, you must also deploy the syslog and Kubernetes filter plugins. The Kubernetes filter plugin is a natively supported Fluent Bit filter which improves logging by adding Kubernetes metadata, such as namespace and Pod name to your logs.

To send Windows worker logs to vRLI using Fluent Bit:

Install Fluent Bit
Validate the Fluent Bit Deployment Using a Sample App

Prepare the Working Environment

To prepare your working environment:

Install docker on the local machine you will be working from. If you are working from a TKGI-provisioned Windows worker node docker is already installed.
To configure Docker to create Windows containers:
1. To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
2. If the Switch to Windows containers… option is present on the Docker Desktop menu, click it. The Docker Desktop menu should now display the Switch to Linux containers… option.
For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.

Install Fluent Bit

To install Fluent Bit:

Build a Windows Fluent Bit Docker Image
Build a Windows Fluent Bit Syslog Plugin Docker Image
Configure Fluent Bit
Deploy Fluent Bit on the Windows Cluster

Build a Windows Fluent Bit Docker Image

To build a Fluent Bit Windows Docker image:

Log in to the Windows 2019 machine where you will do your work.
Download the Fluent Bit source code from the fluent/fluent-bit repository on GitHub.
Configure Docker for creating Windows containers. For more information, see Prepare the Working Environment above.
To build a Fluent Bit container image, run:
```
docker.exe build  -f Dockerfile.windows -t fluent-bit .
```
Note: Docker downloads the Microsoft Visual C++ Redistributable Update while building the Fluent Bit container and installs vc_redist.x64.exe in the new container. If this process fails, see Troubleshooting below.
Push the Fluent Bit container image to your registry.

Build a Windows Fluent Bit Syslog Plugin Docker Image

Fluent Bit does not officially support syslog output. To send TKGI Windows worker logs to vRLI, you must use a syslog plugin.

To build a Fluent Bit syslog plugin image:

Open the benmoss/fluent-bit-out-syslog GitHub repository.
Ensure that you have checked out the windows branch.
Download the syslog plugin.
To remove a conflict between this syslog plugin and the default Fluent Bit syslog output:
1. Locate the main.go file you downloaded to .../windows/cmd/main.go.
2. Open the main.go file in a text editor.
3. Change theout_syslog plugin setting to syslogvrli.
  
  For example:
```
//export FLBPluginRegister
func FLBPluginRegister(def unsafe.Pointer) int {
    return output.FLBPluginRegister(
        def,
        "syslogvrli",
        "syslog output plugin that follows RFC 5424",
    )
}
```
4. Save the file.
Modify the Dockerfile.windows file:
1. Remove the following lines from the file:
```
ARG BASE_IMAGE=mcr.microsoft.com/windows/servercore:ltsc2019
FROM $BASE_IMAGE as builder
```
2. Replace those lines with the following:
```
FROM REGISTRY-IMAGE
```
  Where REGISTRY-IMAGE is the registry image location of the container you built in Build a Windows Fluent Bit Docker Image above.
  
  For example:
```
FROM example.com/k8s-services/windows-images/fluent-bit:latest
```
3. Save the file.
Configure Docker for creating Windows containers. For more information, see Prepare the Working Environment above.

To build the container, use:

docker.exe build  -f Dockerfile.windows -t FLUENT-BIT-IMAGE .

Where FLUENT-BIT-IMAGE is the name for the Fluent Bit image to store in your registry.

For example:

docker.exe build  -f Dockerfile.windows -t fluent-bit-syslog .

Push the image to your registry.

Configure Fluent Bit

The Fluent Bit Service Account, Cluster Role, and Cluster Role Binding objects ensure that the Fluent Bit Kubernetes filter can access and read metadata from the Kubernetes API server kubernetes.default.svc.cluster.local:443.

Your Fluent Bit configuration is a YAML file that defines the Fluent Bit Service Account, Cluster Role, Cluster Role Binding and other objects.

To create a Fluent Bit deployment configuration file:

Create a file named fluent-bit.yml.

Populate the file with the following:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit-win
  namespace: pks-system
--- 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit-win
  namespace: pks-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-win
  labels:
    app: fluent-bit
  namespace: pks-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     debug
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              C:\var\log\containers\*.log
        Parser            docker
        DB                /var/log/flb_kube1.db
        Skip_Long_Lines   On
        Refresh_Interval  60

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        DNS_Retries         10
        Kube_Tag_Prefix     kube.c.var.log.containers.
        K8S-Logging.Parser  On

    [OUTPUT]
        Name           PLUGIN-ID                              #Use the plugin id you changed before you use the image.
        Match          *
        InstanceName  plaintext-cluster-sink
        Addr          OUTPUT-ADDRESS:OUTPUT-PORT               #If you enable TLS, please use port 1514.
        Cluster       true
        #TLSConfig     {"insecure_skip_verify":true}           #If you enable TLS, place TLS-related configurations here.
  parsers.conf: |
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: fluent-bit
  name: fluent-bit-windows
  namespace: pks-system
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      nodeSelector:
        beta.kubernetes.io/os: windows
      tolerations:
      - key: "windows"
        operator: "Equal"
        value: "2019"
        effect: "NoSchedule"
      containers:
      - image: FLUENT-BIT-IMAGE:latest                                          #Change image to point to the image in your registry.
        imagePullPolicy: IfNotPresent
        name: fluent-bit
        command: [fluent-bit]
        args:
        - --plugin
        - /syslog-plugin/out_syslog.so
        - --config
        - /fluent-bit/etc/fluent-bit.conf
        workingDir: /fluent-bit/etc
        volumeMounts:
        - mountPath: /fluent-bit/etc
          name: fluent-bit-config
        - mountPath: /var/log
          name: varlog
          readOnly: false
        - mountPath: /ProgramData/docker/containers
          name: dockercontainers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: dockercontainers
        hostPath:
          path: /ProgramData/docker/containers
      - configMap:
          defaultMode: 420
          name: fluent-bit-win                                        
        name: fluent-bit-config
      serviceAccountName: fluent-bit-win
  updateStrategy:
    type: RollingUpdate

Where:

PLUGIN-ID is the plug in ID for your plug in. For example, syslogvrli.
OUTPUT-ADDRESS is the IP address of your vRealize Log Insight installation.
OUTPUT-PORT is the port to use to communicate with your vRealize Log Insight installation. Use port 514 for most installations. Use port 1514 if you enable TLS.
FLUENT-BIT-IMAGE is the name of the Fluent Bit image in your registry. For example, fluent-bit-syslog.

Save the file.

Deploy Fluent Bit on the Windows Cluster

To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.

To deploy Fluent Bit:

Deploy Fluent Bit using kubectl:
```
kubectl create -f CONFIG-FILE
```
Where CONFIG-FILE is the filename of your Fluent Bit deployment configuration file.

For example:
```
kubectl create -f fluent-bit.yml
```

Validate the Fluent Bit Deployment Using a Sample App

Configure a Sample App
Deploy the Sample App
Validate the Fluent Bit Deployment

Configure a Sample App

To confirm that logging is working correctly, use a sample app that outputs log entries frequently. The logspewer sample app defined below outputs a log every 10 seconds when running.

To configure a logspewer sample app for testing:

Create a file named sample.yml.

Populate the file with the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logspewer
  namespace: pks-system
  labels:
    app: logspewer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logspewer
  template:
    metadata:
      labels:
        app: logspewer
    spec:
      containers:
      - name: logspewer
        image: pivotalgreenhouse/logspewer:latest
        env:
        - name: INTERVAL_IN_SECONDS
          value: "10"
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
      - key: "windows"
        operator: "Equal"
        value: "2019"
        effect: "NoSchedule"

Save the file.

Deploy the Sample App

Deploy the test app using kubectl:

To deploy the test app:
```
kubectl create -f CONFIG-FILE
```
Where CONFIG-FILE is the filename of your sample app deployment configuration file.

For example:
```
kubectl create -f sample.yml
```

Validate the Fluent Bit Deployment

Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.

To confirm the sample app’s logs are being written to vRLI:

Open vRLI.
Open the vRLI > Interactive Analytics tab.
Search the list for “logspewer”. The “logspewer” items are the log entries generated by the sample app.

Troubleshooting

‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container

Symptom

The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll' is displayed while Docker installs the Microsoft Visual C++ Redistributable Update.

Description

When Docker builds the Fluent Bit container, it installs the Microsoft Visual C++ Redistributable Update to the container. To do this, it downloads the Redistributable Update as vc_redist.x64.exe, installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/ directory.

If a Copy-Item : Cannot find path error is returned for either msvcp140.dll, vccorlib140.dll, or vcruntime140.dll the installation of vc_redist.x64.exe has failed.

Workaround

To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:

Download the Microsoft Visual C++ Redistributable Update, vc_redist.x64.exe, from Microsoft.
Install vc_redist.x64.exe on your local Windows 2019 machine.
Copy the following files from C:\Windows\System32\ to the directory containing dockerfile.windows: msvcp140.dll, vccorlib140.dll, and vcruntime140.dll.

Modify your Dockerfile.windows file:

Remove the following lines from the file:

RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
    Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; `
    Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; `
    Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
    Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;

Replace those lines with the following:

RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
    Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; 

COPY msvcp140.dll  /fluent-bit/bin/; 
COPY vccorlib140.dll /fluent-bit/bin/; 
COPY vcruntime140.dll /fluent-bit/bin/;

Save the file.

Re-run the docker build command as shown in Build a Windows Fluent Bit Docker Image above.