This topic describes how to install and configure components and integrations to capture VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows Worker Kubernetes cluster and worker node logs.
Before starting the tasks in this topic:
This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.
To use Fluent Bit to send Windows worker logs to vRLI:
To prepare your working environment:
docker
on the local machine you are working from. If you are working from a TKGI-provisioned Windows worker node, Docker is already installed.To configure Docker to create Windows containers:
For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.
To build a Fluent Bit Windows Docker image:
fluent-bit/dockerfiles/Dockerfile.windows
for editing.FLUENTBIT_VERSION
parameter to 1.8
. Fluent Bit v1.8 supports the native syslog plugin.Change the last line in the Windows Dockerfile from:
ENTRYPOINT ["fluent-bit.exe", "-i", "dummy", "-o", "stdout"]
to instead read:
ENTRYPOINT ["fluent-bit.exe", "--config", "/fluent-bit/etc/fluent-bit.conf"]
To build a Fluent Bit container image, run:
docker.exe build -f Dockerfile.windows -t fluent-bit-1.8 .
While building the Fluent Bit container, Docker downloads the Fluent Bit Windows installation ZIP and Microsoft Visual C++ Redistributable Update and installs vc_redist.x64.exe
in the new container:
* If the process fails while Docker downloads the Microsoft Visual C++ Redistributable Update or while installing vc_redist.x64.exe
in the new container, see ‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.
* If Docker fails to invoke Invoke-WebRequest
while downloading the Fluent Bit Windows installation ZIP from fluentbit.io, see ‘The remote name could not be resolved’ Error When Creating the Fluent Bit Docker Container in Troubleshooting below.
A Fluent Bit configuration defines the ServiceAccount
, ClusterRole
, and ClusterRoleBinding
objects that ensure that the Fluent Bit Kubernetes filter can access and read metadata from the Kubernetes API server kubernetes.default.svc.cluster.local:443
.
You must configure Fluent Bit using a YAML configuration file. For more information on configuring the Syslog output plugin, see Syslog in the Fluent Bit documentation.
Note: Because of Docker container drive mounting limitations, the Fluent Bit configuration below mounts the entire E:
disk to the container, allowing Fluent Bit to read content from the container log path E:\docker-root
.
To create a Fluent Bit deployment configuration YAML file:
fluent-bit.yml
.Populate the file with the following:
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit-win
namespace: pks-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit-win
namespace: pks-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-windows-config
labels:
app: fluent-bit
namespace: pks-system
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level debug
Daemon off
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path C:\var\log\containers\*.log
Parser docker
DB /var/log/flb_kube1.db
Skip_Long_Lines On
Refresh_Interval 60
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Kube_Tag_Prefix kube.c.var.log.containers.
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_Under kubernetes
[OUTPUT]
Name syslog
Match *
Host OUTPUT-ADDRESS
Port OUTPUT-PORT
Mode MODE
syslog_format rfc5424
Syslog_Hostname_key host
Syslog_Appname_key pod_name
Syslog_Procid_key container_name
Syslog_Message_key log
# tls on #only needed when use tls mode
# tls.verify off #only needed when use tls mode
parsers.conf: |
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: fluent-bit
name: fluent-bit-windows
namespace: pks-system
spec:
selector:
matchLabels:
app: fluent-bit
template:
metadata:
labels:
app: fluent-bit
spec:
nodeSelector:
beta.kubernetes.io/os: windows
tolerations:
- key: "windows"
operator: "Equal"
value: "2019"
effect: "NoSchedule"
containers:
- image: fluent-bit-1.8
imagePullPolicy: IfNotPresent
name: fluent-bit
volumeMounts:
- mountPath: /var/log
name: varlog
readOnly: false
- mountPath: /fluent-bit/etc
name: fluent-bit-windows-config
- mountPath: "E:"
name: rootcontainers
volumes:
- name: varlog
hostPath:
path: /var/log
- name: rootcontainers
hostPath:
path: E:/
- configMap:
defaultMode: 420
name: fluent-bit-windows-config
name: fluent-bit-windows-config
serviceAccountName: fluent-bit-win
updateStrategy:
type: RollingUpdate
Where:
OUTPUT-ADDRESS
is the IP address of your vRealize Log Insight installation.OUTPUT-PORT
is the port to use to communicate with your vRealize Log Insight installation. If you enable TLS, set the Port
parameter to 1514
, otherwise set Port
to 514
for most installations.MODE
is the desired transport type. The available options are tcp
, tls
, and udp
. If you set MODE
as tls
, you must also include the required tls
parameters.FLUENT-BIT-IMAGE
is the name of the Fluent Bit image in your registry. For example, fluent-bit-1.8
.Save the file.
To install Fluent Bit:
To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.
To deploy Fluent Bit:
Deploy Fluent Bit using kubectl:
kubectl create -f CONFIG-FILE
Where CONFIG-FILE
is the filename of your Fluent Bit deployment configuration file.
For example:
kubectl create -f fluent-bit.yml
To confirm that logging is working correctly, use a sample app that outputs log entries frequently. The logspewer
sample app defined below outputs a log every 10 seconds when running.
To configure a logspewer
sample app for testing:
sample.yml
.Populate the file with the following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: logspewer
namespace: pks-system
labels:
app: logspewer
spec:
replicas: 1
selector:
matchLabels:
app: logspewer
template:
metadata:
labels:
app: logspewer
spec:
containers:
- name: logspewer
image: pivotalgreenhouse/logspewer:latest
env:
- name: INTERVAL_IN_SECONDS
value: "10"
nodeSelector:
kubernetes.io/os: windows
tolerations:
- key: "windows"
operator: "Equal"
value: "2019"
effect: "NoSchedule"
Save the file.
Deploy the test app using kubectl:
To deploy the test app:
kubectl create -f CONFIG-FILE
Where CONFIG-FILE
is the filename of your sample app deployment configuration file.
For example:
kubectl create -f sample.yml
Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.
To confirm the sample app’s logs are being written to vRLI:
The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll'
is displayed while Docker installs the Microsoft Visual C++ Redistributable Update.
When Docker builds the Fluent Bit container, it installs the Microsoft Visual C++ Redistributable Update to the container. To do this, it downloads the Redistributable Update as vc_redist.x64.exe
, installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/
directory.
If a Copy-Item : Cannot find path
error is returned for either msvcp140.dll
, vccorlib140.dll
, or vcruntime140.dll
the installation of vc_redist.x64.exe
has failed.
To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:
vc_redist.x64.exe
, from Microsoft.vc_redist.x64.exe
on your local Windows 2019 machine.C:\Windows\System32\
to the directory containing dockerfile.windows
: msvcp140.dll
, vccorlib140.dll
, and vcruntime140.dll
.Modify your Dockerfile.windows
file:
Remove the following lines from the file:
RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; `
Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; `
Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;
Replace those lines with the following:
RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait;
COPY msvcp140.dll /fluent-bit/bin/;
COPY vccorlib140.dll /fluent-bit/bin/;
COPY vcruntime140.dll /fluent-bit/bin/;
Save the file.
At the Installing Fluent Bit
step, you see the error Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'
.
For example:
Step 10/16 : RUN Write-Host ('Installing Fluent Bit');
$majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);
Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;
Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;
Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
---> Running in f56e62f47fb5
Installing Fluent Bit
Invoke-WebRequest : The remote name could not be resolved: 'fluentbit.io'
At line:1 char:184
+ ... oString(2); Invoke-WebRequest -Uri https://fluentbit.io/releases/$($m ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
The command
'powershell -Command $ErrorActionPreference = 'Stop';
$ProgressPreference = 'SilentlyContinue';
Write-Host ('Installing Fluent Bit');
$majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2);
Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip;
Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit;
Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;'
returned a non-zero code: 1
To install Fluent Bit while building a Fluent Bit container, Docker uses Invoke-WebRequest
to download the Fluent Bit Windows installation ZIP from fluentbit.io.
The PowerShell Invoke-WebRequest
command is not consistently reliable within a docker container. For more information, see docker build exception: Invoke-WebRequest : The remote name could not be resolved in the Docker GitHub repository.
Change the script to download the Fluent Bit Windows installation ZIP using DOCKER ADD
.
To modify the file:
fluent-bit/dockerfiles/Dockerfile.windows
.Locate the following lines:
RUN Write-Host ('Installing Fluent Bit'); `
$majorminor = ([Version]::Parse("$env:FLUENTBIT_VERSION")).toString(2); `
Invoke-WebRequest -Uri "https://fluentbit.io/releases/$($majorminor)/td-agent-bit-$($env:FLUENTBIT_VERSION)-win64.zip" -OutFile td-agent-bit.zip; `
Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `
Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;
Replace the lines with:
...
ARG FLUENTBIT_VERSION=1.8.0
ARG majorminor=1.8
...
ADD "https://fluentbit.io/releases/$majorminor/td-agent-bit-$FLUENTBIT_VERSION-win64.zip" /local/td-agent-bit.zip
RUN Write-Host ('Installing Fluent Bit'); `
Expand-Archive -Path td-agent-bit.zip -Destination /local/fluent-bit; `
Move-Item -Path /local/fluent-bit/*/* -Destination /fluent-bit/;