This topic describes how to install and configure components and integrations to capture VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows Worker Kubernetes cluster and worker node logs.

Prerequisites

Before starting the tasks in this topic:

  • Your environment must be a Windows 2019 data center with vRealize Log Insight (vRLI).
  • You must have an accessible container image registry.
  • Docker must be installed on the local machine you are working from. For more information, see Prepare the Working Environment below.

Overview

This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.

To use Fluent Bit to send Windows worker logs to vRLI:

  1. Prepare a Fluent Bit Image
  2. Install Fluent Bit

Prepare the Working Environment

To prepare your working environment:

  1. Install docker on the local machine you are working from. If you are working from a TKGI-provisioned Windows worker node, Docker is already installed.
  2. Follow the steps in Configure Docker for Creating Windows Containers below.

Configure Docker for Creating Windows Containers

To configure Docker to create Windows containers:

  1. To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
  2. If the Switch to Windows containers… option is present on the Docker Desktop menu, click it. The Docker Desktop menu should now display the Switch to Linux containers… option.

For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.

Prepare a Fluent Bit Image

  1. Build a Windows Fluent Bit Docker Image
  2. Configure Fluent Bit

Build a Windows Fluent Bit Docker Image

To build a Fluent Bit Windows Docker image:

  1. Log in to the Windows 2019 machine where you are doing your work.
  2. Download the Fluent Bit source code from the fluent/fluent-bit repository on GitHub.
  3. Open the Windows Dockerfile located at fluent-bit/dockerfiles/Dockerfile.windows for editing.
  4. Set the FLUENTBIT_VERSION parameter to 1.8. Fluent Bit v1.8 supports the native syslog plugin.
  5. Change the last line in the Windows Dockerfile from:

    ENTRYPOINT ["fluent-bit.exe", "-i", "dummy", "-o", "stdout"]
    

    to instead read:

    ENTRYPOINT ["fluent-bit.exe", "--config", "/fluent-bit/etc/fluent-bit.conf"]
    
  6. Save the modified Windows Dockerfile.
  7. Configure Docker for creating Windows containers. For more information, see Configure Docker for Creating Windows Containers above.
  8. To build a Fluent Bit container image, run:

    docker.exe build  -f Dockerfile.windows -t fluent-bit-1.8 .
    

    While building the Fluent Bit container, Docker downloads the Fluent Bit Windows installation ZIP and Microsoft Visual C++ Redistributable Update and installs vc_redist.x64.exe in the new container:
    * If the process fails while Docker downloads the Microsoft Visual C++ Redistributable Update or while installing vc_redist.x64.exe in the new container, see ‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.
    * If Docker fails to invoke Invoke-WebRequest while downloading the Fluent Bit Windows installation ZIP from fluentbit.io, see ‘The remote name could not be resolved’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.

  9. Push the Fluent Bit container image to your registry.

Configure Fluent Bit

A Fluent Bit configuration defines the ServiceAccount, ClusterRole, and ClusterRoleBinding objects that ensure that the Fluent Bit Kubernetes filter can access and read metadata from the Kubernetes API server kubernetes.default.svc.cluster.local:443.

You must configure Fluent Bit using a YAML configuration file. For more information on configuring the Syslog output plugin, see Syslog in the Fluent Bit documentation.

Note: Because of Docker container drive mounting limitations, the Fluent Bit configuration below mounts the entire E: disk to the container, allowing Fluent Bit to read content from the container log path E:\docker-root.

To create a Fluent Bit deployment configuration YAML file:

  1. Create a file named fluent-bit.yml.
  2. Populate the file with the following:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: fluent-bit-win
      namespace: pks-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: fluent-bit-read
    rules:
    - apiGroups: [""]
      resources:
      - namespaces
      - pods
      verbs: ["get", "list", "watch"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: fluent-bit-read
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: fluent-bit-read
    subjects:
    - kind: ServiceAccount
      name: fluent-bit-win
      namespace: pks-system
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fluent-bit-windows-config
      labels:
        app: fluent-bit
      namespace: pks-system
    data:
      fluent-bit.conf: |
        [SERVICE]
            Flush         5
            Log_Level     debug
            Daemon        off
            Parsers_File  parsers.conf
    
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              C:\var\log\containers\*.log
            Parser            docker
            DB                /var/log/flb_kube1.db
            Skip_Long_Lines   On
            Refresh_Interval  60
    
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc.cluster.local:443
            Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
            Merge_Log           On
            Merge_Log_Key       log_processed
            K8S-Logging.Parser  On
            K8S-Logging.Exclude Off
            Kube_Tag_Prefix     kube.c.var.log.containers.
    
        [FILTER]
            Name                nest
            Match               kube.*
            Operation           lift
            Nested_Under        kubernetes
    
        [OUTPUT]
            Name          syslog
            Match          *
            Host          OUTPUT-ADDRESS
            Port          OUTPUT-PORT  
            Mode          MODE  
            syslog_format rfc5424
            Syslog_Hostname_key  host
            Syslog_Appname_key   pod_name
            Syslog_Procid_key    container_name
            Syslog_Message_key   log
       #    tls           on    #only needed when use tls mode
       #    tls.verify    off   #only needed when use tls mode
    
      parsers.conf: |
        [PARSER]
            Name   json
            Format json
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z
    
        [PARSER]
            Name        docker
            Format      json
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L
            Time_Keep   On
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        app: fluent-bit
      name: fluent-bit-windows
      namespace: pks-system
    spec:
      selector:
        matchLabels:
          app: fluent-bit
      template:
        metadata:
          labels:
            app: fluent-bit
        spec:
          nodeSelector:
            beta.kubernetes.io/os: windows
          tolerations:
          - key: "windows"
            operator: "Equal"
            value: "2019"
            effect: "NoSchedule"
          containers:
          - image: fluent-bit-1.8
            imagePullPolicy: IfNotPresent
            name: fluent-bit
            volumeMounts:
            - mountPath: /var/log
              name: varlog
              readOnly: false
            - mountPath: /fluent-bit/etc
              name: fluent-bit-windows-config
            - mountPath: "E:"
              name: rootcontainers
          volumes:
          - name: varlog
            hostPath:
              path: /var/log
          - name: rootcontainers
            hostPath:
              path: E:/
          - configMap:
              defaultMode: 420
              name: fluent-bit-windows-config                                       
            name: fluent-bit-windows-config
          serviceAccountName: fluent-bit-win
      updateStrategy:
        type: RollingUpdate
    

    Where:

    • OUTPUT-ADDRESS is the IP address of your vRealize Log Insight installation.
    • OUTPUT-PORT is the port to use to communicate with your vRealize Log Insight installation. If you enable TLS, set the Port parameter to 1514, otherwise set Port to 514 for most installations.
    • MODE is the desired transport type. The available options are tcp, tls, and udp. If you set MODE as tls, you must also include the required tls parameters.
    • FLUENT-BIT-IMAGE is the name of the Fluent Bit image in your registry. For example, fluent-bit-1.8.
  3. Save the file.

Install Fluent Bit

To install Fluent Bit:

  1. Deploy Fluent Bit on the Windows Cluster
  2. Validate the Fluent Bit Deployment Using a Sample App

Deploy Fluent Bit on the Windows Cluster

To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.

To deploy Fluent Bit:

  1. Deploy Fluent Bit using kubectl:

    kubectl create -f CONFIG-FILE
    

    Where CONFIG-FILE is the filename of your Fluent Bit deployment configuration file.

    For example:

    kubectl create -f fluent-bit.yml
    

Validate the Fluent Bit Deployment Using a Sample App

  1. Configure a Sample App
  2. Deploy the Sample App
  3. Validate the Fluent Bit Deployment

Configure a Sample App

To confirm that logging is working correctly, use a sample app that outputs log entries frequently. The logspewer sample app defined below outputs a log every 10 seconds when running.

To configure a logspewer sample app for testing:

  1. Create a file named sample.yml.
  2. Populate the file with the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: logspewer
      namespace: pks-system
      labels:
        app: logspewer
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: logspewer
      template:
        metadata:
          labels:
            app: logspewer
        spec:
          containers:
          - name: logspewer
            image: pivotalgreenhouse/logspewer:latest
            env:
            - name: INTERVAL_IN_SECONDS
              value: "10"
          nodeSelector:
            kubernetes.io/os: windows
          tolerations:
          - key: "windows"
            operator: "Equal"
            value: "2019"
            effect: "NoSchedule"
    
  3. Save the file.

Deploy the Sample App

Deploy the test app using kubectl:

  1. To deploy the test app:

    kubectl create -f CONFIG-FILE
    

    Where CONFIG-FILE is the filename of your sample app deployment configuration file.

    For example:

    kubectl create -f sample.yml
    

Validate the Fluent Bit Deployment

Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.

To confirm the sample app’s logs are being written to vRLI:

  1. Open vRLI.
  2. Open the vRLI > Interactive Analytics tab.
  3. Search the list for “logspewer”. The “logspewer” items are the log entries generated by the sample app.

Troubleshooting


‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container

Symptom

The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll' is displayed while Docker installs the Microsoft Visual C++ Redistributable Update.

Description

When Docker builds the Fluent Bit container, it installs the Microsoft Visual C++ Redistributable Update to the container. To do this, it downloads the Redistributable Update as vc_redist.x64.exe, installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/ directory.

If a Copy-Item : Cannot find path error is returned for either msvcp140.dll, vccorlib140.dll, or vcruntime140.dll the installation of vc_redist.x64.exe has failed.

Workaround

To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:

  1. Download the Microsoft Visual C++ Redistributable Update, vc_redist.x64.exe, from Microsoft.
  2. Install vc_redist.x64.exe on your local Windows 2019 machine.
  3. Copy the following files from C:\Windows\System32\ to the directory containing dockerfile.windows: msvcp140.dll, vccorlib140.dll, and vcruntime140.dll.
  4. Modify your Dockerfile.windows file:

    1. Remove the following lines from the file:

      RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
          Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; `
          Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; `
          Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
          Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;
      
    2. Replace those lines with the following:

      RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
          Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; 
      
      COPY msvcp140.dll  /fluent-bit/bin/; 
      COPY vccorlib140.dll /fluent-bit/bin/; 
      COPY vcruntime140.dll /fluent-bit/bin/;
      
    3. Save the file.

  5. Re-run the Docker build command. For more information, see Build a Windows Fluent Bit Docker Image above.


‘The remote name could not be resolved’ Error When Creating the Fluent Bit Docker Container

Symptom

At the Installing Fluent Bit step, you see the error Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'.

For example:

Step 10/16 : RUN Write-Host ('Installing Fluent Bit');     
             $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);     
             Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;     
             Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;     
             Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
 
 ---> Running in f56e62f47fb5
 
Installing Fluent Bit
 
Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'
 
At line:1 char:184
 
+ ... oString(2); Invoke-WebRequest -Uri https://fluentbit.io/releases/$($m ...
 
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
 
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
 
The command 
            'powershell -Command $ErrorActionPreference = 'Stop'; 
            $ProgressPreference = 'SilentlyContinue'; 
            Write-Host ('Installing Fluent Bit');     
            $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);     
            Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;     
            Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;     
            Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;' 
returned a non-zero code: 1

Description

To install Fluent Bit while building a Fluent Bit container, Docker uses Invoke-WebRequest to download the Fluent Bit Windows installation ZIP from fluentbit.io.

The PowerShell Invoke-WebRequest command is not consistently reliable within a docker container. For more information, see docker build exception: Invoke-WebRequest : The remote name could not be resolved in the Docker GitHub repository.

Workaround

Change the script to download the Fluent Bit Windows installation ZIP using DOCKER ADD.

To modify the file:

  1. Back up the Windows Dockerfile located at fluent-bit/dockerfiles/Dockerfile.windows.
  2. Open the Windows Dockerfile for editing.
  3. Locate the following lines:

    RUN Write-Host ('Installing Fluent Bit'); `
    
        $majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2); `
    
        Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip; `
    
        Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `
    
    Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
    
  4. Replace the lines with:

    ...
    ARG FLUENTBIT_VERSION=1.8.0
    ARG majorminor=1.8
    ...
    
    ADD "https://fluentbit.io/releases/$majorminor/td-agent-bit-$FLUENTBIT_VERSION-win64.zip" /local/td-agent-bit.zip
    RUN Write-Host ('Installing Fluent Bit'); `
    Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `
    Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
    
  5. Save the file.
  6. Re-run the Docker build command. For more information, see Build a Windows Fluent Bit Docker Image above.


check-circle-line exclamation-circle-line close-line
Scroll to top icon