This topic describes how to deploy the TKG Extension v1.3.1 for Fluent Bit. Fluent Bit is a fast, lightweight log processor and forwarder that lets you collect application data and logs from different sources, unify them, and send them to multiple destinations. Deploy the TKG Extension for Fluent Bit to collect and forward Tanzu Kubernetes cluster logs to your destination of choice.

Extension Prerequisites

Adhere to the following requirements before deploying the TKG Extension v1.3.1 for Fluent Bit.

Deploy the Fluent Bit Extension

The TKG Extension for Fluent Bit installs a Fluent Bit container on the cluster. For more information on this container, see https://fluentbit.io/.
Container Resource Type Replicas Description
Fluent Bit DaemonSet 6 Log collector, aggregator, forwarder
The extension is configured to pull the containers from the VMware public registry at https://projects.registry.vmware.com/. If you are using a private registry, change the endpoint URL in the data values and extension configurations files to match. See Configure the Fluent Bit Extension for a description of the fields and options.
  1. Verify that you completed each of the extension prerequisites. See Extension Prerequisites.
  2. Change directory to the Fluent Bit extension.
    cd /tkg-extensions-v1.3.1+vmware.1/extensions/logging/fluent-bit
  3. Create the tanzu-system-logging namespace and the Fluent Bit service account and role objects.
    kubectl apply -f namespace-role.yaml
  4. Decide which log destination to use for Fluent Bit. Supported outputs include Elasticsearch, HTTP, Kafka, Splunk, and Syslog. See https://docs.fluentbit.io/manual/pipeline/outputs for more information.
  5. Create a Fluent Bit data values file for your chosen log destination by copying one of the <LOG_BACKEND>/fluent-bit-data-values.example.yaml files.
    There is an example data values file for each supported log destination. The example provides the minimum configuration for that log destination.
    cp elasticsearch/fluent-bit-data-values.yaml.example elasticsearch/fluent-bit-data-values.yaml
    cp http/fluent-bit-data-values.yaml.example http/fluent-bit-data-values.yaml
    cp kafka/fluent-bit-data-values.yaml.example kafka/fluent-bit-data-values.yaml
    cp splunk/fluent-bit-data-values.yaml.example splunk/fluent-bit-data-values.yaml
    cp syslog/fluent-bit-data-values.yaml.example syslog/fluent-bit-data-values.yaml
  6. Configure the Fluent Bit extension by populating the <LOG_BACKEND>/fluent-bit-data-values.yaml. See Configure the Fluent Bit Extension for a description of the fields and options.
    For example, the Fluent Bit syslog configuration requires the following values:
    logging:
      image:
        repository: projects.registry.vmware.com/tkg # Public registry
    tkg:
      instance_name: "<TKG_INSTANCE_NAME>" #mandatory but arbitrary; appears in logs
      cluster_name: "<CLUSTER_NAME>" #name of the target tkgs cluster
    fluent_bit:
      output_plugin: "syslog"
      syslog:
        host: "<SYSLOG_HOST>"
        port: "<SYSLOG_PORT>"
        mode: "<SYSLOG_MODE>"
        format: "<SYSLOG_FORMAT>"
    A populated data values file for Fluent Bit syslog might have the following configuration:
    logging:
      image:
        repository: projects.registry.vmware.com/tkg
    tkg:
      instance_name: "tkgs-cluster-1"
      cluster_name: "tkgs-cluster-1"
    fluent_bit:
      output_plugin: "syslog"
      syslog:
        host: "10.192.175.59"
        port: "514"
        mode: "tcp"
        format: "rfc5424"
  7. Create a Fluent Bit secret with data values for your log destination.
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=elasticsearch/fluent-bit-data-values.yaml -n tanzu-system-logging
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=kafka/fluent-bit-data-values.yaml -n tanzu-system-logging
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=splunk/fluent-bit-data-values.yaml -n tanzu-system-logging
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=http/fluent-bit-data-values.yaml -n tanzu-system-logging
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=syslog/fluent-bit-data-values.yaml -n tanzu-system-logging
    The secret/fluent-bit-data-values is created in the tanzu-system-logging namespace. Verify using the following command:
    kubectl get secrets -n tanzu-system-logging
  8. Deploy the Fluent Bit app.
    kubectl apply -f fluent-bit-extension.yaml

    On success you should see app.kappctrl.k14s.io/fluent-bit created.

  9. Check the status of the Fluent Bit app.
    kubectl get app fluent-bit -n tanzu-system-logging
    On success the status should change from Reconciling to Reconcile succeeded. If the status is Reconcile failed, see Troubleshoot Fluent Bit Deployment.
  10. View detailed status on the app.
    kubectl get app fluent-bit -n tanzu-system-logging -o yaml
  11. Verify the Fluent Bit DeamonSet.
    kubectl get daemonsets -n tanzu-system-logging
    On success you should see the following:
    NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    fluent-bit   6         6         6       6            6           <none>          105s

Troubleshoot Fluent Bit Deployment

If the deployment or reconciliation fails, run kubectl get pods -A to view pod status. The fluent-bit pods should be Running. If a pod status is ImagePullBackOff or ImageCrashLoopBackOff, the container image could not be pulled. Check the registry URL in the data values and the extension YAML files and make sure they are accurate.

Check the container logs, where name-XXXX is the unique pod name that you can see when you run kubectl get pods -A:
kubectl logs pod/fluent-bit-XXXXX -c fluent-bit -n tanzu-system-logging

Update the Fluent Bit Extension

Update the Fluent Bit extension that is deployed to a Tanzu Kubernetes cluster.

  1. Get Fluent Bit data values from the secret.
    kubectl get secret fluent-bit-data-values -n tanzu-system-logging -o 'go-template={{ index .data "values.yaml" }}' | base64 -d > fluent-bit-data-values.yaml
    
  2. Update Fluent Bit data values in fluent-bit-data-values.yaml. See Configure the Fluent Bit Extension.
  3. Update Fluent Bit data values secret.
    kubectl create secret generic fluent-bit-data-values --from-file=values.yaml=fluent-bit-data-values.yaml -n tanzu-system-logging -o yaml --dry-run | kubectl replace -f-
    
    The Fluent Bit extension will be reconciled again with the above data values.
    Note: By default, kapp-controller will sync apps every 5 minutes. The update should take effect in 5 minutes or less. If you want the update to take effect immediately, change syncPeriod in fluent-bit-extension.yaml to a lesser value and apply the Fluent Bit extension using kubectl apply -f fluent-bit-extension.yaml.
  4. Check the status of the extension.
    kubectl get app fluent-bit -n tanzu-system-logging
  5. View detailed status and troubleshoot.
    kubectl get app fluent-bit -n tanzu-system-logging -o yaml
  6. Troubleshoot if necessary. See Troubleshoot Fluent Bit Deployment.

Delete the Fluent Bit Extension

Delete the Fluent Bit extension from a Tanzu Kubernetes cluster.
Note: Complete the steps in order. Do not delete the namespace, service account, and role objects before the Fluent Bit app is fully deleted. Doing so can lead to system errors.
  1. Change directory to the Fluent Bit extension.
    cd extensions/logging/fluent-bit/
  2. Delete the Fluent Bit app.
    kubectl delete app fluent-bit -n tanzu-system-logging

    Expected result: app.kappctrl.k14s.io "fluent-bit" deleted.

  3. Verify that the Fluent Bit app is deleted.
    kubectl get app fluent-bit -n tanzu-system-logging

    Expected result: apps.kappctrl.k14s.io "fluent-bit" not found.

  4. Delete the tanzu-system-logging namespace and the Fluent Bit extension service account and role objects.
    kubectl delete -f namespace-role.yaml

Upgrade the Fluent Bit Extension

If you have an existing Fluent Bit extension deployed, you can upgrade it to the latest version.
  1. Export the Fluent Bit configmap.
    kubectl get configmap fluent-bit -n tanzu-system-logging -o 'go-template={{ index .data "fluent-bit.yaml" }}' > fluent-bit-configmap.yaml
    
  2. Delete the existing Fluent Bit deployment. See Delete the Fluent Bit Extension.
  3. Deploy the latest Fluent Bit extension. See Deploy the Fluent Bit Extension.

Configure the Fluent Bit Extension

The configuration values for are set in extensions/logging/fluent-bit/<LOG_BACKEND>/fluent-bit-data-values.yaml.
Table 1. Fluent Bit Extension Configurations
Parameter Description Type Default
logging.namespace Namespace where Fluent Bit will be deployed string tanzu-system-logging
logging.service_account_name Name of Fluent Bit service account string fluent-bit
logging.cluster_role_name Name of cluster role which grants get, watch and list permissions to fluent bit string fluent-bit-read
logging.image.name Name of Fluent Bit image string fluent-bit
logging.image.tag Fluent Bit image tag. This value may need to be updated if you are upgrading the version. string v1.6.9_vmware.1
logging.image.repository Location of the repository with the Fluent Bit image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment). string projects.registry.vmware.com/tkg
logging.image.pullPolicy Fluent bit image pull policy string IfNotPresent
logging.update_strategy Update strategy to be used when updating DaemonSet string RollingUpdate
tkg.cluster_name Name of the Tanzu Kubernetes cluster string Null (Mandatory parameter)
tkg.instance_name User-defined name of the TKG instance, shared by the Supervisor Cluster and all Tanzu Kubernetes clusters in one deployment. You can use any name related to the installation. string Null (Mandatory parameter)
Note: This field is mandatory but arbitrary. It is a name that appears in the logs.
fluent_bit.log_level Log level to use for Fluent Bit string info
fluent_bit.output_plugin Set the backend to which Fluent Bit should flush the information it gathers string Null (Mandatory parameter)
fluent_bit.elasticsearch.host IP address or hostname of the target Elasticsearch instance string Null (Mandatory parameter when output_plugin is elastic search)
fluent_bit.elasticsearch.port TCP port of the target Elasticsearch instance integer Null (Mandatory parameter when output_plugin is elastic search)
fluent_bit.elasticsearch.buffer_size Specify the buffer size used to read the response from Elasticsearch service. Sets to unlimited if False string False
fluent_bit.elasticsearch.tls Specify the default setting for TLS for Elasticsearch string Off
fluent_bit.kafka.broker_service_name Single of multiple list of Kafka Brokers, e.g., 192.168.1.3:9092 string Null (Mandatory parameter when output_plugin is kafka)
fluent_bit.kafka.topic_name Single entry or list of topics separated by (,) that Fluent Bit will use to send messages to Kafka string Null (Mandatory parameter when output_plugin is kafka)
fluent_bit.splunk.host IP address or hostname of the target Splunk Server string Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.splunk.port TCP port of the target Splunk Server integer Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.splunk.token Specify the Authentication Token for the HTTP Event Collector interface string Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.http.host IP address or hostname of the target HTTP Server string Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.port TCP port of the target HTTP Server integer Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.mode Specify an HTTP URI for the target web server string Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.header_key_value HTTP header key/value pair. Multiple headers can be set string Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.format Specify the data format to be used in the HTTP request body string Null (Mandatory parameter when output_plugin is http)
fluent_bit.syslog.host Domain or IP address of the remote Syslog server string Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.port TCP or UDP port of the remote Syslog server integer Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.mode Specify the transport type from TCP, UDP and TLS string Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.format Specify the data format to be used in the HTTP request body string Null ( Mandatory parameter when output_plugin is syslog)
host_path.volume_1 Directory path from the host node's file system into the pod, for volume 1 string /var/log
host_path.volume_2 Directory path from the host node's file system into the pod, for volume 2 string /var/lib/docker/containers
host_path.volume_3 Directory path from the host node's file system into the pod, for volume 3 string /run/log
systemd.path Path to the Systemd journal directory string /var/log/journal