Install Fluent Bit for Log Forwarding

This topic explains how to deploy Fluent Bit into a standalone management or workload cluster in Tanzu Kubernetes Grid.

Fluent Bit is a lightweight log processor and forwarder. The Fluent Bit implementation provided in this release of Tanzu Kubernetes Grid allows you to collect logs from clusters running on vSphere 8, Amazon Web Services (AWS), and Azure. You can then forward them to a log storage provider such as Elastic Search, Kafka, Splunk, or an HTTP endpoint.

You can deploy Fluent Bit on any standalone management cluster or workload cluster from which you want to collect logs. First, you configure an output plugin on the cluster from which you want to gather logs, depending on the endpoint that you use. Then, you deploy Fluent Bit on the cluster.

Prerequisites

A bootstrap machine with the following installed:
- Tanzu CLI and kubectl, as described in Install the Tanzu CLI and Other Tools for Use with a vSphere with Tanzu Supervisor or Install the Tanzu CLI and Other Tools for Use with Standalone Management Clusters.
- Carvel tools, as described in Install the Carvel Tools.
- yq v4.5 or later.
A standalone management cluster and workload cluster running on vSphere 8, AWS, or Azure. For information about deploying workload clusters, see Creating Workload Clusters.
You have deployed one of the logging management backends for storing and analyzing logs that Fluent Bit supports, such as VMware Aria Operations for Logs. For information, see the Fluent Bit documentation.

Prepare the Cluster for Fluent Bit Deployment

Note
You must deploy Fluent Bit on each cluster from which you want to collect logs.

To prepare the cluster:

Get the admin credentials of the cluster into which you want to deploy Fluent Bit. For example:
```
tanzu cluster kubeconfig get my-cluster --admin
```

Set the context of kubectl to the cluster. For example:

kubectl config use-context my-cluster-admin@my-cluster

Deploy Fluent Bit on a Cluster

Follow the procedure below to deploy Fluent Bit on a cluster:

If the cluster does not have a package repository with the Fluent Bit package installed, such as the tanzu-standard repository, install one:

Note
If you are targeting a plan-based cluster (legacy), skip this step. For plan-based clusters, the tanzu-standard package repository is automatically enabled in every cluster, in the tanzu-package-repo-global namespace.
```
tanzu package repository add PACKAGE-REPO-NAME --url PACKAGE-REPO-ENDPOINT --namespace tkg-system
```
Where:
- PACKAGE-REPO-NAME is the name of the package repository, such as tanzu-standard or the name of a private image registry configured with ADDITIONAL_IMAGE_REGISTRY variables.
- PACKAGE-REPO-ENDPOINT is the URL of the package repository.
  - For this release, the tanzu-standard URL is projects.registry.vmware.com/tkg/packages/standard/repo:v2.2.0. See List Package Repositories to obtain this value from the Tanzu CLI, or in Tanzu Mission Control see the Addons > Repositories list in the Cluster pane.
If you have not already done so, install cert-manager in the cluster. For instructions, see Install cert-manager for Certificate Management.
Confirm that the Fluent Bit package is available in your cluster:
```
tanzu package available list -A
```

Retrieve the version of the available package:

tanzu package available list fluent-bit.tanzu.vmware.com -A

Prepare a configuration file for the Fluent Bit package:
1. Retrieve the default configuration of the package:
```
tanzu package available get fluent-bit.tanzu.vmware.com/PACKAGE-VERSION --default-values-file-output FILE-PATH
```
  Where PACKAGE-VERSION is the version of the Fluent Bit package that you want to install and FILE-PATH is the location to which you want to save the configuration file, for example, fluent-bit-data-values.yaml.
2. Based on the log management backend you want to configure, add one or more [Output] configurations to the fluent_bit.config.outputs key in the fluent-bit-data-values.yaml file.
3. Review and if needed, modify other configuration parameters. For links to full lists of configuration parameters that you can set in fluent-bit-data-values.yaml, see Fluent Bit Package Configuration Parameters below.
After you prepare your fluent-bit-data-values.yaml file, remove all comments in the file:
```
yq -i eval '... comments=""' fluent-bit-data-values.yaml
```
Run tanzu package install to deploy the package:
```
tanzu package install fluent-bit \
--package fluent-bit.tanzu.vmware.com \
--version AVAILABLE-PACKAGE-VERSION \
--values-file fluent-bit-data-values.yaml \
--namespace TARGET-NAMESPACE
```
Where:
- TARGET-NAMESPACE is the namespace in which you want to install the Fluent Bit package. For example, the my-packages namespace.
  - If the --namespace flag is not specified, the Tanzu CLI uses the default namespace. The Fluent Bit pods and any other resources associated with the Fluent Bit component are created in the namespace that you provided in fluent-bit-data-values.yaml; do not install the Fluent Bit package into this namespace.
  - The specified namespace must already exist, for example from running kubectl create namespace my-packages.
- AVAILABLE-PACKAGE-VERSION is the version that you retrieved by running tanzu package available above.
For example:
```
tanzu package install fluent-bit --package fluent-bit.tanzu.vmware.com --namespace my-packages --version 1.8.15+vmware.1-tkg.1
```

Fluent Bit Package Configuration Parameters

The table below lists configuration parameters of the Fluent Bit package and describes their default values. You can set the following configuration values in your fluent-bit-data-values.yml file created in Deploy Fluent Bit on a Cluster above.

Parameter	Description	Type	Default
`namespace`	Namespace where Fluent Bit will be deployed.	String	`tanzu-system-logging`
`fluent_bit.config.outputs`	For information about the configuration for Fluent Bit outputs, see the Fluent Bit documentation.	Multiline YAML	`Standard output`
`fluent_bit.config.parsers`	For information about the configuration for Fluent Bit parsers, see the Fluent Bit documentation.	Multiline YAML	`JSON parser`
`fluent_bit.config.plugins`	Content for Fluent Bit plugins configuration.	String	`<nil>`
`fluent_bit.config.service`	For information about the configuration for Fluent Bit service, see the Fluent Bit documentation.	Multiline YAML	Default Fluent Bit service config.
`fluent_bit.config.streams`	Content for Fluent Bit streams file.	String	`<nil>`
`fluent_bit.config.filters`	For information about the configuration for Fluent Bit filters, see the Fluent Bit documentation.	Multiline YAML	Default Kubernetes filter.
`fluent_bit.config.inputs`	For information about the configuration for Fluent Bit inputs, see the Fluent Bit documentation.	String	Ingest Kubernetes container logs using the `tail` plugin and ingest `systemd` logs from kubelet.
`fluent_bit.daemonset.resources`	For information about the configuration for Fluent Bit containers resource requirements, see the Fluent Bit documentation.	Multiline YAML	`<nil>`
`fluent_bit.daemonset.podAnnotations`	The Fluent Bit deployments pod annotations.	List	`<nil>`
`fluent_bit.daemonset.podLabels`	The Fluent Bit deployments pod labels.	List	`<nil>`

Verify Fluent Bit Deployment

After you deploy Fluent Bit, you can verify that the deployment is successful:

Confirm that the fluent-bit package is installed. For example:
```
tanzu package installed list -A
```
The fluent-bit package and the fluent-bit app are installed in the namespace that you specify when running the tanzu package install command.
Confirm that the fluent-bit app is successfully reconciled in your AVAILABLE-PACKAGE-NAMESPACE:
```
kubectl get apps -A
```
If the status is not Reconcile succeeded, view the full status details of the fluent-bit app. Viewing the full status can help you troubleshoot the problem:
```
kubectl get app fluent-bit --namespace AVAILABLE-PACKAGE-NAMESPACE -o yaml
```
Where AVAILABLE-PACKAGE-NAMESPACE is the namespace in which you installed the package.
Confirm that the new services are running by listing all of the pods that are running in the cluster:
```
kubectl get pods -A
```
In the tanzu-system-logging namespace, you should see the fluent-bit service running in a pod:
```
NAMESPACE               NAME                     READY   STATUS    RESTARTS   AGE
[...]
tanzu-system-logging    fluent-bit-zd2tn         1/1     Running   0          6m23s
[...]
```
The Fluent Bit pods and any other resources associated with the Fluent Bit component are created in the namespace you provided in the fluent-bit-data-values.yaml file. If you are using the default namespace, these are created in the tanzu-system-logging namespace.

Update a Running Fluent Bit Deployment

To make changes to the configuration of the Fluent Bit package after deployment, update your deployed Fluent Bit package:

Update the Fluent Bit configuration in the fluent-bit-data-values.yaml file.

Update the installed package:

tanzu package installed update fluent-bit \
--version 1.8.15+vmware.1-tkg.1 \
--values-file fluent-bit-data-values.yaml \
--namespace my-packages

Expected output:

| Updating package 'fluent-bit'
- Getting package install for 'fluent-bit'
| Updating secret 'fluent-bit-my-packages-values'
| Updating package install for 'fluent-bit'

 Updated package install 'fluent-bit' in namespace 'my-packages'

The Fluent Bit package is reconciled using the new value or values that you added. It can take up to five minutes for kapp-controller to apply the changes.

For information about updating, see Update a Package.

Delete a Fluent Bit Deployment

To remove the Fluent Bit package on your cluster, run:

tanzu package installed delete fluent-bit --namespace my-packages

For information about deleting, see Delete a Package.