Checkout how to configure forwarding of Supervisor control plane logs to external monitoring systems, such as Grafana Loki, or Elastic Search, by using Fluent Bit.

Supervisor control plane logs are automatically forwarded to the syslog server configured to the vCenter Server appliance by using Fluent Bit. Fluent Bit is an open-source lightweight logging and metrics processor and forwarder that provides configurations to support various log data types, filtering and log tag enhancements.

During Supervisor activation or upgrade, bootstrap logs are still forwarded by rsyslog to the syslog servers that are configured to the vCenter Server appliance. Once the Supervisor control plane VMs are up and running, Fluent Bit becomes the default log forwarder for the Supervisor control plane logs.

As a vSphere administrator, you can use Fluent Bit to:

  • Forward Supervisor control plane logs and system journal logs to major external log monitoring platforms such as Loki, Elastic Search, Grafana, and other platforms that are supported by Fluent Bit.

  • Update or reset the log forwarding configuration for the Supervisor control plane by using the k8s API.
Fluent Bit runs as a DeamonSet on Supervisor control plane nodes. It exposes the fluentbit-config-custom ConfigMap under the vmware-system-logging namespace that vSphere administrators can edit to configure log forwarding to external platforms by defining log servers. 
inputs-custom.conf: |
  [INPUT]
      Name              tail
      Alias             audit_apiserver_tail
      Tag               audit.apiserver.*
      Path              /var/log/vmware/audit/kube-apiserver.log
      DB                /var/log/vmware/fluentbit/flb_audit_apiserver.db
      Buffer_Max_Size   12MBb
      Mem_Buf_Limit     32MB
      Skip_Long_Lines   On
      Refresh_Interval  10
 
filters-custom.conf: |
  [FILTER]
      Name           record_modifier
      Alias          audit_apiserver_modifier
      Match          audit.apiserver.*
      Record         hostname ${NODE_NAME}
      Record         appname audit-kube-apiserver
      Record         filename kube-apiserver.log
 
outputs-custom.conf: |
  [OUTPUT]
     Name                 syslog
     Alias                audit_apiserver_output_syslog
     Match                audit.apiserver.*
     Host                 <syslog-server-host>
     Port                 <syslog-server-port>
     Mode                 tcp
     Syslog_Format        rfc5424
     Syslog_Message_key   log
     Syslog_Hostname_key  hostname
     Syslog_Appname_key   appname
     Syslog_Msgid_key     filename

Customize Fluent Bit Log Forwarding

Follow the steps to customize the Fluent Bit log forwarding configuration:

  1. Login to the Supervisor control plane as a vCenter Single Sign-On administrator.
    > kubectl vsphere login --server=<supervisor-cluster-vip> -u [email protected]
> kubectl config use-context <supervisor-cluster-vip>
  2. Update or add a syslog output in the outputs-custom.conf section in the fluentbit-config-custom ConfigMap, which will forward all control plane VM system logs to an external server.
    [OUTPUT]
     Name                 syslog
     Alias                syslog_system
     Match                system*
     Host                 <syslog-server-host>
     Port                 <syslog-server-port>
     Mode                 tcp
     Syslog_Format        rfc5424
     Syslog_Message_key   log
     Syslog_Hostname_key  hostname
     Syslog_Appname_key   appname
     Syslog_Msgid_key     filename
     # add the following if the mode is TLS
     Tls                  on
     Tls.verify           off
     Tls.ca_file          /etc/ssl/certs/vmca.pem
  3. Apply the changes to the fluentbit-config-custom ConfigMap.
    > kubectl -n vmware-system-logging edit cm fluentbit-config-custom
 
# use the below command if the change is stored in outputs-custom.conf file
> kubectl -n vmware-system-logging create configmap fluentbit-config-custom --from-file=filters-custom.conf --from-file=inputs-custom.conf --from-file=outputs-custom.conf -o yaml --from-file=parsers-custom.conf --dry-run | kubectl replace -f -

  4. Monitor the Fluent Bit pod to apply the configuration changes automatically, and query Supervisor logs on the syslog server. If the Fluentbit DaemonSet runs to an error after updated configuration isreloaded, repair or reset the configuration in the fluentbit-config-custom ConfigMap to ensure that the Fluentbit DaemonSet healthy.

    > kubectl -n vmware-system-logging get pod
> kubectl -n vmware-system-logging logs <fluentbit-pod-name>

Forward Kubernetes API Server Audit Logs to a Grafana Loki Server

Follow the steps to configure log forwarding to an external Grafana Loki server:

  1. Login to the Supervisor control plane as a vCenter Single Sign-On administrator.
    > kubectl vsphere login --server=<supervisor-cluster-vip> -u [email protected]
> kubectl config use-context <supervisor-cluster-vip>
  2. Update or add a Loki output in the outputs-custom.conf section in the fluentbit-config-custom ConfigMap, which will forward all control plane VM system logs to the Loki log server.
    [OUTPUT]
        Name loki
        Alias system_output_loki
        Match system*
        Host <loki-server-host>
        Port <loki-server-port>
        Labels $hostname,$appname,$filename,$procid,$labels
  3. Apply the changes to the fluentbit-config-custom ConfigMap.
    > kubectl -n vmware-system-logging edit cm fluentbit-config-custom
 
# use the below command if the change is stored in outputs-custom.conf file
> kubectl -n vmware-system-logging create configmap fluentbit-config-custom --from-file=filters-custom.conf --from-file=inputs-custom.conf --from-file=outputs-custom.conf -o yaml --from-file=parsers-custom.conf --dry-run | kubectl replace -f -

  4. Monitor the Fluent Bit pod to apply the configuration changes automatically, and query Supervisor logs on the syslog server. If the Fluentbit DaemonSet runs to an error after updated configuration isreloaded, repair or reset the configuration in the fluentbit-config-custom ConfigMap to ensure that the Fluentbit DaemonSet healthy.

    > kubectl -n vmware-system-logging get pod
> kubectl -n vmware-system-logging logs <fluentbit-pod-name>

Forward Logs to Elastic Search

Follow the steps to configure log forwarding to an external Elastic Search server:
  1. Login to the Supervisor control plane as a vCenter Single Sign-On administrator.
    > kubectl vsphere login --server=<supervisor-cluster-vip> -u [email protected]
> kubectl config use-context <supervisor-cluster-vip>
  2. Update or add a Elastic Search output in the outputs-custom.conf section in the fluentbit-config-custom ConfigMap, which will forward all control plane VM system logs to the ES log server.
    [OUTPUT]
       Name es
       Alias system_output_es
       Match system*
       Host <es-server-host>
       Port <es-server-port>
       Index supervisor
       Type controlplanevm
  3. Apply the changes to the fluentbit-config-custom ConfigMap.
    > kubectl -n vmware-system-logging edit cm fluentbit-config-custom
 
# use the below command if the change is stored in outputs-custom.conf file
> kubectl -n vmware-system-logging create configmap fluentbit-config-custom --from-file=filters-custom.conf --from-file=inputs-custom.conf --from-file=outputs-custom.conf -o yaml --from-file=parsers-custom.conf --dry-run | kubectl replace -f -

  5. Monitor the Fluent Bit pod to apply the configuration changes automatically, and query Supervisor logs on the syslog server.

    > kubectl -n vmware-system-logging get pod
> kubectl -n vmware-system-logging logs <fluentbit-pod-name>

Forward Kubernetes API Audit Logs to a Syslog Server

Follow the steps to configure forwarding of Kubernetes API audit logs to an external syslog server:

  1. Add kubectl-plugin-vsphere and authproxy input to fluentbit-config ConfigMap:
    [INPUT]
    Name              tail
    Tag               auth.kubectl-plugin.*
    Path              /var/log/containers/audit/kubectl-plugin-vsphere*.log
    DB                /var/log/vmware/fluentbit/flb_auth_kubectl-plugin.db
    Skip_Long_Lines   Off
    Refresh_Interval  10
 
[INPUT]
    Name              tail
    Tag               auth.authproxy.*
    Path              /var/log/containers/audit/wcp-authproxy*.log
    DB                /var/log/vmware/fluentbit/flb_auth_authproxy.db
    Skip_Long_Lines   Off
    Refresh_Interval  10
  2. Add kubectl-plugin-vsphere and authproxy filter to the fluentbit-config ConfigMap:
    [FILTER]
    Name           kubernetes
    Match          auth.*
    Kube_URL       https://localhost:6443
    Tls.verify     Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
 
[FILTER]
    Name           record_modifier
    Match          auth.*
    Operation      lift
    Nested_under   kubernetes
 
[FILTER]
    Name           modify
    Match          auth.*
    Rename         container_name appname
    Rename         host hostname
    Rename         pod_name procid
  3. Add kubectl-plugin-vsphere output to syslog server to the fluentbit-config ConfigMap:
    [OUTPUT]
    Name                 syslog
    Match                auth.*
    Host                 <syslog-server-host>
    Port                 <syslog-server-port>
    Mode                 tcp
    Syslog_Format        rfc5424
    Syslog_Message_key   log
    Syslog_Hostname_key  hostname
    Syslog_Appname_key   appname
    Syslog_Msgid_key     filename
  4. Include the above files in the fluentbit-config ConfigMap under the vmware-system-logging namespace.
    > k -n vmware-system-logging edit cm fluentbit-config
> k -n vmware-system-logging rollout restart ds fluentbit
> k -n vmware-system-logging rollout status ds fluentbit