Logging Windows Worker Workloads

This topic describes how to install and configure components and integrations to capture logs from VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) provisioned Windows Worker Kubernetes clusters and worker nodes.

Prerequisites

Before starting the tasks in this topic:

Your environment must be a Windows 2019 data center with vRealize Log Insight (vRLI).
You must have an accessible container image registry.
Docker must be installed on the local machine you are working from. For more information, see Prepare the Working Environment below.

Overview

This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.

To use Fluent Bit to send Windows worker logs to vRLI:

Prepare a Fluent Bit Image
Install Fluent Bit

Prepare the Working Environment

To prepare your working environment:

Install docker on the local machine you are working from. If you are working from a TKGI-provisioned Windows worker node, Docker is already installed.
Follow the steps in Configure Docker for Creating Windows Containers below.

Configure Docker for Creating Windows Containers

To configure Docker to create Windows containers:

To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
If the Switch to Windows containers… option is present on the Docker Desktop menu, click it. The Docker Desktop menu now displays the Switch to Linux containers… option.

For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.

Build a Windows Fluent Bit Docker Image

To build a Fluent Bit Windows Docker image:

Log in to the Windows 2019 machine where you are doing your work.
Download the Fluent Bit source code from the fluent/fluent-bit repository on GitHub.
Open the Windows Dockerfile located at fluent-bit/dockerfiles/Dockerfile.windows for editing.
Set the FLUENTBIT_VERSION parameter to 1.8. Fluent Bit v1.8 supports the native syslog plugin.

Change the last line in the Windows Dockerfile from:

ENTRYPOINT ["fluent-bit.exe", "-i", "dummy", "-o", "stdout"]

to instead read:

ENTRYPOINT ["fluent-bit.exe", "--config", "/fluent-bit/etc/fluent-bit.conf"]

Save the modified Windows Dockerfile.
Configure Docker for creating Windows containers. For more information, see Configure Docker for Creating Windows Containers above.
To build a Fluent Bit container image, run:
```
docker.exe build  -f Dockerfile.windows -t fluent-bit-1.8 .
```
While building the Fluent Bit container, Docker downloads the Fluent Bit Windows installation ZIP and Microsoft Visual C++ Redistributable Update and installs vc_redist.x64.exe in the new container:
* If the process fails while Docker downloads the Microsoft Visual C++ Redistributable Update or while installing vc_redist.x64.exe in the new container, see ‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.
* If Docker fails to invoke Invoke-WebRequest while downloading the Fluent Bit Windows installation ZIP from fluentbit.io, see ‘The remote name could not be resolved’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.
Push the Fluent Bit container image to your registry.

Configure Fluent Bit

A Fluent Bit configuration defines the ServiceAccount, ClusterRole, and ClusterRoleBinding objects that ensure that the Fluent Bit Kubernetes filter can access and read metadata from the Kubernetes API server kubernetes.default.svc.cluster.local:443.

You must configure Fluent Bit using a YAML configuration file. For more information on configuring the Syslog output plugin, see Syslog in the Fluent Bit documentation.

Note: Because of Docker container drive mounting limitations, the Fluent Bit configuration below mounts the entire E: disk to the container, allowing Fluent Bit to read content from the container log path E:\docker-root.

To create a Fluent Bit deployment configuration YAML file:

Create a file named fluent-bit.yml.

Populate the file with the following:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit-win
  namespace: pks-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit-win
  namespace: pks-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-windows-config
  labels:
    app: fluent-bit
  namespace: pks-system
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     debug
        Daemon        off
        Parsers_File  parsers.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              C:\var\log\containers\*.log
        Parser            docker
        DB                /var/log/flb_kube1.db
        Skip_Long_Lines   On
        Refresh_Interval  60

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Kube_Tag_Prefix     kube.c.var.log.containers.

    [FILTER]
        Name                nest
        Match               kube.*
        Operation           lift
        Nested_Under        kubernetes

    [OUTPUT]
        Name          syslog
        Match          *
        Host          OUTPUT-ADDRESS
        Port          OUTPUT-PORT  
        Mode          MODE  
        syslog_format rfc5424
        Syslog_Hostname_key  host
        Syslog_Appname_key   pod_name
        Syslog_Procid_key    container_name
        Syslog_Message_key   log
   #    tls           on    #only needed when use tls mode
   #    tls.verify    off   #only needed when use tls mode

  parsers.conf: |
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: fluent-bit
  name: fluent-bit-windows
  namespace: pks-system
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      nodeSelector:
        beta.kubernetes.io/os: windows
      tolerations:
      - key: "windows"
        operator: "Equal"
        value: "2019"
        effect: "NoSchedule"
      containers:
      - image: fluent-bit-1.8
        imagePullPolicy: IfNotPresent
        name: fluent-bit
        volumeMounts:
        - mountPath: /var/log
          name: varlog
          readOnly: false
        - mountPath: /fluent-bit/etc
          name: fluent-bit-windows-config
        - mountPath: "E:"
          name: rootcontainers
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: rootcontainers
        hostPath:
          path: E:/
      - configMap:
          defaultMode: 420
          name: fluent-bit-windows-config                                       
        name: fluent-bit-windows-config
      serviceAccountName: fluent-bit-win
  updateStrategy:
    type: RollingUpdate

Where:

OUTPUT-ADDRESS is the IP address of your vRealize Log Insight installation.
OUTPUT-PORT is the port to use to communicate with your vRealize Log Insight installation. If you enable TLS, set the Port parameter to 1514, otherwise set Port to 514 for most installations.
MODE is the desired transport type. The available options are tcp, tls, and udp. If you set MODE as tls, you must also include the required tls parameters.
FLUENT-BIT-IMAGE is the name of the Fluent Bit image in your registry. For example, fluent-bit-1.8.

Save the file.

Install Fluent Bit

To install Fluent Bit:

Deploy Fluent Bit on the Windows Cluster
Validate the Fluent Bit Deployment Using a Sample App

Deploy Fluent Bit on the Windows Cluster

To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.

To deploy Fluent Bit:

Deploy Fluent Bit using kubectl:
```
kubectl create -f CONFIG-FILE
```
Where CONFIG-FILE is the filename of your Fluent Bit deployment configuration file.

For example:
```
kubectl create -f fluent-bit.yml
```

Validate the Fluent Bit Deployment Using a Sample App

Configure a Sample App
Deploy the Sample App
Validate the Fluent Bit Deployment

Configure a Sample App

To confirm that logging is working correctly, use a sample app that outputs log entries frequently. The logspewer sample app defined below outputs a log every 10 seconds when running.

To configure a logspewer sample app for testing:

Create a file named sample.yml.

Populate the file with the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logspewer
  namespace: pks-system
  labels:
    app: logspewer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logspewer
  template:
    metadata:
      labels:
        app: logspewer
    spec:
      containers:
      - name: logspewer
        image: pivotalgreenhouse/logspewer:latest
        env:
        - name: INTERVAL_IN_SECONDS
          value: "10"
      nodeSelector:
        kubernetes.io/os: windows
      tolerations:
      - key: "windows"
        operator: "Equal"
        value: "2019"
        effect: "NoSchedule"

Save the file.

Deploy the Sample App

Deploy the test app using kubectl:

To deploy the test app:
```
kubectl create -f CONFIG-FILE
```
Where CONFIG-FILE is the filename of your sample app deployment configuration file.

For example:
```
kubectl create -f sample.yml
```

Validate the Fluent Bit Deployment

Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.

To confirm the sample app’s logs are being written to vRLI:

Open vRLI.
Open the vRLI > Interactive Analytics tab.
Search the list for “logspewer”. The “logspewer” items are the log entries generated by the sample app.

Troubleshooting

‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container

Symptom

The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll' is displayed while Docker installs the Microsoft Visual C++ Redistributable Update.

Description

When Docker builds the Fluent Bit container, it installs the Microsoft Visual C++ Redistributable Update to the container. To do this, it downloads the Redistributable Update as vc_redist.x64.exe, installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/ directory.

If a Copy-Item : Cannot find path error is returned for either msvcp140.dll, vccorlib140.dll, or vcruntime140.dll the installation of vc_redist.x64.exe has failed.

Workaround

To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:

Download the Microsoft Visual C++ Redistributable Update, vc_redist.x64.exe, from Microsoft.
Install vc_redist.x64.exe on your local Windows 2019 machine.
Copy the following files from C:\Windows\System32\ to the directory containing dockerfile.windows: msvcp140.dll, vccorlib140.dll, and vcruntime140.dll.

Modify your Dockerfile.windows file:

Remove the following lines from the file:

RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
    Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; `
    Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; `
    Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
    Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;

Replace those lines with the following:

RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
    Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; 

COPY msvcp140.dll  /fluent-bit/bin/; 
COPY vccorlib140.dll /fluent-bit/bin/; 
COPY vcruntime140.dll /fluent-bit/bin/;

Save the file.

Re-run the Docker build command. For more information, see Build a Windows Fluent Bit Docker Image above.

‘The remote name could not be resolved’ Error When Creating the Fluent Bit Docker Container

Symptom

At the Installing Fluent Bit step, you see the error Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'.

For example:

Step 10/16 : RUN Write-Host ('Installing Fluent Bit');     
             $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);     
             Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;     
             Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;     
             Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
 
 ---> Running in f56e62f47fb5
 
Installing Fluent Bit
 
Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'
 
At line:1 char:184
 
+ ... oString(2); Invoke-WebRequest -Uri https://fluentbit.io/releases/$($m ...
 
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
 
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
The command 
            'powershell -Command $ErrorActionPreference = 'Stop'; 
            $ProgressPreference = 'SilentlyContinue'; 
            Write-Host ('Installing Fluent Bit');     
            $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);     
            Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;     
            Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;     
            Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;' 
returned a non-zero code: 1

Description

To install Fluent Bit while building a Fluent Bit container, Docker uses Invoke-WebRequest to download the Fluent Bit Windows installation ZIP from fluentbit.io.

The PowerShell Invoke-WebRequest command is not consistently reliable within a docker container. For more information, see docker build exception: Invoke-WebRequest : The remote name could not be resolved in the Docker GitHub repository.

Workaround

Change the script to download the Fluent Bit Windows installation ZIP using DOCKER ADD.

To modify the file:

Back up the Windows Dockerfile located at fluent-bit/dockerfiles/Dockerfile.windows.
Open the Windows Dockerfile for editing.

Locate the following lines:

RUN Write-Host ('Installing Fluent Bit'); `

    $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2); `

    Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip; `

    Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `

Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;

Replace the lines with:

...
ARG FLUENTBIT_VERSION=1.8.0
ARG majorminor=1.8
...

ADD "https://fluentbit.io/releases/$majorminor/td-agent-bit-$FLUENTBIT_VERSION-win64.zip" /local/td-agent-bit.zip
RUN Write-Host ('Installing Fluent Bit'); `
Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `
Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;

Save the file.
Re-run the Docker build command. For more information, see Build a Windows Fluent Bit Docker Image above.