Collecting Harbor Logs

You can use Fluentbit to collect logs from harbor pods and forward them to SysLog server. Fluentbit can be deployed on Workload cluster as fluentbit addon.

Following is the fluent-bit configuration to collect and forward harbor pod logs to SysLog server:

[Service]  
Flush         1  
Log_Level     info  
Daemon        off  
Parsers_File  parsers.conf  
HTTP_Server   On  
HTTP_Listen   0.0.0.0  
HTTP_Port     2020

[FILTER]  
Name                kubernetes  
Match               kube.*  
Kube_URL            https://kubernetes.default.svc:443  
Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt  
Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token  
Merge_Log           On  
Merge_Log_Key       log_processed  
K8S-Logging.Parser  On  
K8S-Logging.Exclude On

[FILTER]  
 Name nest  
 Match kube.*  
 Operation lift  
 Nested_under kubernetes  
 Add_prefix k8s_

[INPUT]  
Name tail  
Path /var/log/containers/*_<<harbor-cnf-namespace>>_*.log,/var/log/containers/*_<<other-namespaces>>_*  
Parser docker  
Tag kube.*  
Mem_Buf_Limit 5MB  
Skip_Long_Lines On
[OUTPUT]  
Name              stdout  
Match             kube.*
[OUTPUT]  Name syslog  
Match kube.*  
Host <<vrli-ip>>  
Port 514  
Mode tls  
syslog_message_key log  
Syslog_Hostname_key   k8s_host  
Syslog_Appname_key    k8s_pod_name  
Syslog_Procid_key     k8s_container_name  
Syslog_Msgid_key      k8s_namespace_name

NOTE:Replace <<harbor-cnf-namespace>> with k8s namespace (or multiple k8s namespaces separated by comma) to collect the logs from Replace <<vrli-ip>> with vRLI IP and Port number accordingly under OUTPUT Section vRLI can be used as SysLog server(Port 541) as well as vRLI (Port 9543)

Procedure

1. Log in to TCA-M, navigate to Infrastructure > CaaS Infrastructure and select workload cluster.
2. Navigate to Add-Ons tab and select DEPLOY ADD-ON option.
3. Select fluent-bit add-on and click Next.
4. Provide above sample fluent-bit configuration (Change Host, Port, and log Path accordingly). Save the changes and deploy the fluent-bit add-on configuration.
5. Logging to syslog server and validate the logs are getting streamed under the Explore Logs tab as shown in the following image. You can filter the logs based on the field shown in the table. Also, Field mappings is provided as part of fluent bit add-on configuration.

   vRLI Log Field	Harbor pod logs mapping field
   source	K8s Node IP (where harbor pod is present)
   hostname	K8s Node VM name (where harbor pod is present)
   appname	Pod name
   procid	Pod container name
   msgid	K8s namespace

   
   
   

   Note:

   Fluentbit will not work with VMware Aria Operations for Logs directly as fluentbit log format is different than Aria Operations for Logs. FluentD is required to convert the Fluentbit log format to Aria Operations for Logs format.