This topic explains how to deploy Fluent Bit into a standalone management or workload cluster in Tanzu Kubernetes Grid.
Fluent Bit is a lightweight log processor and forwarder. The Fluent Bit implementation provided in this release of Tanzu Kubernetes Grid allows you to collect logs from clusters running on vSphere 8, Amazon Web Services (AWS), and Azure. You can then forward them to a log storage provider such as Elastic Search, Kafka, Splunk, or an HTTP endpoint.
You can deploy Fluent Bit on any standalone management cluster or workload cluster from which you want to collect logs. First, you configure an output plugin on the cluster from which you want to gather logs, depending on the endpoint that you use. Then, you deploy Fluent Bit on the cluster.
kubectl, as described in Install the Tanzu CLI and Other Tools for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Other Tools for Use with Standalone Management Clusters.
You must deploy Fluent Bit on each cluster from which you want to collect logs.
To prepare the cluster:
Get the admin credentials of the cluster into which you want to deploy Fluent Bit. For example:
tanzu cluster kubeconfig get my-cluster --admin
Set the context of kubectl to the cluster. For example:
kubectl config use-context my-cluster-admin@my-cluster
Follow the procedure below to deploy Fluent Bit on a cluster:
If the cluster does not already have the
standard package repository installed, install it:
If you are targeting a plan-based cluster (legacy), skip this step. For plan-based clusters, the
tanzu-standardpackage repository is automatically enabled in every cluster, in the
tanzu package repository add tanzu-standard --url PACKAGE-REPOSITORY-ENDPOINT --namespace tkg-system
PACKAGE-REPOSITORY-ENDPOINT is the URL of the
standard package repository. For this release, the URL is
See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
If you have not already done so, install cert-manager in the cluster. For instructions, see Install cert-manager for Certificate Management.
Confirm that the Fluent Bit package is available in your cluster:
tanzu package available list -A
Retrieve the version of the available package:
tanzu package available list fluent-bit.tanzu.vmware.com -A
Prepare a configuration file for the Fluent Bit package:
Retrieve the default configuration of the package:
tanzu package available get fluent-bit.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
PACKAGE-VERSION is the version of the Fluent Bit package that you want to install and
FILE-PATH is the location to which you want to save the configuration file, for example,
Based on the log management backend you want to configure, add one or more
[Output] configurations to the
fluent_bit.config.outputs key in the
The sample file contains examples for Syslog, Elasticsearch, Kafka, Splunk, and HTTP
[Output] configurations for your reference. To use the provided Syslog output plugin configuration, you must also uncomment the
[Filter] entries under
"Optional - uncomment if using Syslog output plugin".
The configuration blocks for some
[Filter] options are commented out. Uncomment and fill in values for any options you want to configure. You must provide values such as hostnames and ports for your log management backend, as these will be specific to your environment. Locations for providing these values appear in the format
For other Fluent Bit plugins, refer to the Fluent Bit documentation.
Review and if needed, modify other configuration parameters. For a full list of configuration parameters that you can set in
fluent-bit-data-values.yaml, see Fluent Bit Package Configuration Parameters below.
After you prepare your
fluent-bit-data-values.yaml file, remove all comments in the file:
yq -i eval '... comments=""' fluent-bit-data-values.yaml
tanzu package install to deploy the package:
tanzu package install fluent-bit \ --package fluent-bit.tanzu.vmware.com \ --version AVAILABLE-PACKAGE-VERSION \ --values-file fluent-bit-data-values.yaml \ --namespace TARGET-NAMESPACE
TARGET-NAMESPACE is the namespace in which you want to install the Fluent Bit package. For example, the
--namespaceflag is not specified, the Tanzu CLI uses the
defaultnamespace. The Fluent Bit pods and any other resources associated with the Fluent Bit component are created in the namespace that you provided in
fluent-bit-data-values.yaml; do not install the Fluent Bit package into this namespace.
kubectl create namespace my-packages.
AVAILABLE-PACKAGE-VERSION is the version that you retrieved by running
tanzu package available above.
tanzu package install fluent-bit --package fluent-bit.tanzu.vmware.com --namespace my-packages --version 1.8.15+vmware.1-tkg.1
The table below lists configuration parameters of the Fluent Bit package and describes their default values. You can set the following configuration values in your
fluent-bit-data-values.yml file created in Deploy Fluent Bit on a Cluster above.
||Namespace where Fluent Bit will be deployed.||String||
||For information about the configuration for Fluent Bit outputs, see the Fluent Bit documentation.||String||
||For information about the configuration for Fluent Bit parsers, see the Fluent Bit documentation.||String||
||Content for Fluent Bit plugins configuration.||String||
||For information about the configuration for Fluent Bit service, see the Fluent Bit documentation.||String||Default Fluent Bit service config.|
||Content for Fluent Bit streams file.||String||
||For information about the configuration for Fluent Bit filters, see the Fluent Bit documentation.||String||Default Kubernetes filter.|
||For information about the configuration for Fluent Bit inputs, see the Fluent Bit documentation.||String||Ingest Kubernetes container logs using the
||For information about the configuration for Fluent Bit containers resource requirements, see the Fluent Bit documentation.||Object||
||The Fluent Bit deployments pod annotations.||Object||
||The Fluent Bit deployments pod labels.||Object||
After you deploy Fluent Bit, you can verify that the deployment is successful:
Confirm that the fluent-bit package is installed. For example:
tanzu package installed list -A
fluent-bit package and the
fluent-bit app are installed in the namespace that you specify when running the
tanzu package install command.
Confirm that the
fluent-bit app is successfully reconciled in your
kubectl get apps -A
If the status is not
Reconcile succeeded, view the full status details of the
fluent-bit app. Viewing the full status can help you troubleshoot the problem:
kubectl get app fluent-bit --namespace AVAILABLE-PACKAGE-NAMESPACE -o yaml
AVAILABLE-PACKAGE-NAMESPACE is the namespace in which you installed the package.
Confirm that the new services are running by listing all of the pods that are running in the cluster:
kubectl get pods -A
tanzu-system-logging namespace, you should see the
fluent-bit service running in a pod:
NAMESPACE NAME READY STATUS RESTARTS AGE [...] tanzu-system-logging fluent-bit-zd2tn 1/1 Running 0 6m23s [...]
The Fluent Bit pods and any other resources associated with the Fluent Bit component are created in the namespace you provided in the
fluent-bit-data-values.yaml file. If you are using the default namespace, these are created in the
To make changes to the configuration of the Fluent Bit package after deployment, update your deployed Fluent Bit package:
Update the Fluent Bit configuration in the
Update the installed package:
tanzu package installed update fluent-bit \ --version 1.8.15+vmware.1-tkg.1 \ --values-file fluent-bit-data-values.yaml \ --namespace my-packages
| Updating package 'fluent-bit' - Getting package install for 'fluent-bit' | Updating secret 'fluent-bit-my-packages-values' | Updating package install for 'fluent-bit' Updated package install 'fluent-bit' in namespace 'my-packages'
The Fluent Bit package is reconciled using the new value or values that you added. It can take up to five minutes for
kapp-controller to apply the changes.
For information about updating, see Update a Package.
To remove the Fluent Bit package on your cluster, run:
tanzu package installed delete fluent-bit --namespace my-packages
For information about deleting, see Delete a Package.