Install Fluent Bit for Log Forwarding

This topic gives an overview of the Fluent Bit package, which you can install in Tanzu Kubernetes Grid (TKG) workload clusters to provide log forwarding services for the cluster.

Fluent Bit is a fast, lightweight log processor and forwarder that lets you collect application data and logs from different sources, unify them, and send them to multiple destinations.

You can use Fluent Bit as a log forwarder for logs from a TKG cluster. You need to have a logging management server deployed for storing and analyzing logs. You can use one of the supported logging servers.

Syslog
HTTP
Elastic Search
Kafka
Splunk

Installation: Install Fluent Bit on a workload cluster in one of the following ways, based on its deployment option:

TKG on Supervisor:
- Install Fluent Bit Using the Tanzu CLI
- Install Fluent Bit Using Kubectl
Standalone management cluster: Install Fluent Bit in Workload Clusters Deployed by a Standalone Management Cluster

Fluent Bit Components, Configuration, Data Values

The following sections describe Fluent Bit components and show how you can configure the Fluent Bit package.

Fluent Bit Components

The Fluent Bit package installs on the cluster the container listed in the table. For more information, see https://fluentbit.io/. The package pulls the container from the VMware public registry specified in Package Repository.

Container	Resource Type	Replicas	Description
Fluent Bit	DaemonSet	6	Log collector, aggregator, forwarder

Fluent Bit Data Values

The example provided with the installation can be used for Syslog. See Install Fluent Bit Using Kubectl.

The following example can be used for an HTTP logging endpoint.

...
    ---
    namespace: fluentbit-logging
    tkg:
      instance_name: "<TKG_INSTANCE_NAME>"
      cluster_name: "<CLUSTER_NAME>"
    fluentbit:
      pspNames: "vmware-system-restricted"
      output_plugin: "http"
      http:
        host: "<HTTP_HOST>"
        port: "<HTTP_PORT>"
        uri: "<URI>"
        header_key_value: "<HEADER_KEY_VALUE>"
        format: "json"

The following example can be used for Elastic Search.

...
    ---
    namespace: fluentbit-logging
    tkg:
      instance_name: "<TKG_INSTANCE_NAME>"
      cluster_name: "<CLUSTER_NAME>"
    fluentbit:
      pspNames: "vmware-system-restricted"
      output_plugin: "elasticsearch"
      elasticsearch:
        host: "<ELASTIC_SEARCH_HOST>"
        port: "<ELASTIC_SEARCH_PORT>"

The following example can be used for Kafka.

...
    ---
    namespace: fluentbit-logging
    tkg:
      instance_name: "<TKG_INSTANCE_NAME>"
      cluster_name: "<CLUSTER_NAME>"
    fluentbit:
      pspNames: "vmware-system-restricted"
      output_plugin: "kafka"
      kafka:
        broker_service_name: "<BROKER_SERVICE_NAME>"
        topic_name: "<TOPIC_NAME>"

The following example can be used for Splunk.

...
    ---
    namespace: fluentbit-logging
    tkg:
      instance_name: "<TKG_INSTANCE_NAME>"
      cluster_name: "<CLUSTER_NAME>"
    fluentbit:
      pspNames: "vmware-system-restricted"
      output_plugin: "splunk"
      splunk:
        host: "<SPLUNK_HOST>"
        port: "<SPLUNK_PORT>"
        token: "<SPLUNK_TOKEN>"

Fluent Bit Configuration Parameters (TKG on Supervisor)

The configuration values for are set in fluent-bit-data-values.yaml. The table lists and describes the available parameters.

Parameter	Description	Type	Default
logging.namespace	Namespace where Fluent Bit will be deployed	string	tanzu-system-logging
logging.service_account_name	Name of Fluent Bit service account	string	fluent-bit
logging.cluster_role_name	Name of cluster role which grants get, watch and list permissions to fluent bit	string	fluent-bit-read
logging.image.name	Name of Fluent Bit image	string	fluent-bit
logging.image.tag	Fluent Bit image tag. This value may need to be updated if you are upgrading the version.	string	v1.6.9_vmware.1
logging.image.repository	Location of the repository with the Fluent Bit image. The default is the public VMware registry. Change this value if you are using a private repository (e.g., air-gapped environment).	string	projects.registry.vmware.com/tkg
logging.image.pullPolicy	Fluent bit image pull policy	string	IfNotPresent
logging.update_strategy	Update strategy to be used when updating DaemonSet	string	RollingUpdate
tkg.cluster_name	Name of the Tanzu Kubernetes cluster	string	Null (Mandatory parameter)
tkg.instance_name	User-defined name of the TKG instance, shared by the Supervisor Cluster and all Tanzu Kubernetes clusters in one deployment. You can use any name related to the installation.	string	Null (Mandatory parameter)Note: This field is mandatory but arbitrary. It is a name that appears in the logs.
fluent_bit.log_level	Log level to use for Fluent Bit	string	info
fluent_bit.output_plugin	Set the backend to which Fluent Bit should flush the information it gathers	string	Null (Mandatory parameter)
fluent_bit.elasticsearch.host	IP address or hostname of the target Elasticsearch instance	string	Null (Mandatory parameter when output_plugin is elastic search)
fluent_bit.elasticsearch.port	TCP port of the target Elasticsearch instance	integer	Null (Mandatory parameter when output_plugin is elastic search)
fluent_bit.elasticsearch.buffer_size	Specify the buffer size used to read the response from Elasticsearch service. Sets to unlimited if False	string	False
fluent_bit.elasticsearch.tls	Specify the default setting for TLS for Elasticsearch	string	Off
fluent_bit.kafka.broker_service_name	Single of multiple list of Kafka Brokers, e.g., 192.168.1.3:9092	string	Null (Mandatory parameter when output_plugin is kafka)
fluent_bit.kafka.topic_name	Single entry or list of topics separated by (,) that Fluent Bit will use to send messages to Kafka	string	Null (Mandatory parameter when output_plugin is kafka)
fluent_bit.splunk.host	IP address or hostname of the target Splunk Server	string	Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.splunk.port	TCP port of the target Splunk Server	integer	Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.splunk.token	Specify the Authentication Token for the HTTP Event Collector interface	string	Null (Mandatory parameter when output_plugin is splunk)
fluent_bit.http.host	IP address or hostname of the target HTTP Server	string	Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.port	TCP port of the target HTTP Server	integer	Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.mode	Specify an HTTP URI for the target web server	string	Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.header_key_value	HTTP header key/value pair. Multiple headers can be set	string	Null (Mandatory parameter when output_plugin is http)
fluent_bit.http.format	Specify the data format to be used in the HTTP request body	string	Null (Mandatory parameter when output_plugin is http)
fluent_bit.syslog.host	Domain or IP address of the remote Syslog server	string	Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.port	TCP or UDP port of the remote Syslog server	integer	Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.mode	Specify the transport type from TCP, UDP and TLS	string	Null (Mandatory parameter when output_plugin is syslog)
fluent_bit.syslog.format	Specify the data format to be used in the HTTP request body	string	Null ( Mandatory parameter when output_plugin is syslog)
host_path.volume_1	Directory path from the host node’s file system into the pod, for volume 1	string	/var/log
host_path.volume_2	Directory path from the host node’s file system into the pod, for volume 2	string	/var/lib/docker/containers
host_path.volume_3	Directory path from the host node’s file system into the pod, for volume 3	string	/run/log
systemd.path	Path to the Systemd journal directory	string	/var/log/journal

Fluent Bit Configuration Parameters (Standalone MC)

The table below lists configuration parameters of the Fluent Bit package and describes their default values. You can set the following configuration values in your fluent-bit-data-values.yaml file created in Deploy Fluent Bit on a Cluster.

Parameter	Description	Type	Default
`namespace`	Namespace where Fluent Bit will be deployed.	String	`tanzu-system-logging`
`fluent_bit.config.outputs`	For information about the configuration for Fluent Bit outputs, see the Fluent Bit documentation.	Multiline YAML	`Standard output`
`fluent_bit.config.parsers`	For information about the configuration for Fluent Bit parsers, see the Fluent Bit documentation.	Multiline YAML	`JSON parser`
`fluent_bit.config.plugins`	Content for Fluent Bit plugins configuration.	String	`<nil>`
`fluent_bit.config.service`	For information about the configuration for Fluent Bit service, see the Fluent Bit documentation.	Multiline YAML	Default Fluent Bit service config.
`fluent_bit.config.streams`	Content for Fluent Bit streams file.	String	`<nil>`
`fluent_bit.config.filters`	For information about the configuration for Fluent Bit filters, see the Fluent Bit documentation.	Multiline YAML	Default Kubernetes filter.
`fluent_bit.config.inputs`	For information about the configuration for Fluent Bit inputs, see the Fluent Bit documentation.	String	Ingest Kubernetes container logs using the `tail` plugin and ingest `systemd` logs from kubelet.
`fluent_bit.daemonset.resources`	For information about the configuration for Fluent Bit containers resource requirements, see the Fluent Bit documentation.	Multiline YAML	`<nil>`
`fluent_bit.daemonset.podAnnotations`	The Fluent Bit deployments pod annotations.	List	`<nil>`
`fluent_bit.daemonset.podLabels`	The Fluent Bit deployments pod labels.	List	`<nil>`
`fluent_bit.ipv6Primary`	If you are deploying Fluent Bit to an IPv6 cluster, set `ipv6Primary` to `true`.	Boolean	`false`