Issue with writing input syslog to output loki

[akarneliuk]

Bug Report

Describe the bug
I'm using fluent-bit to receive syslog messages from network devices, parse it using built-in parser and write the output to Loki. However, there is no information is written to Loki despite I'm getting 204 status code back. I've enabled debug on Loki side and can see that actually fluent-bit sends empty message.

Here is the output from fluent-bit suggesting that it sends data to Loki:

Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/12/23 23:00:42] [ info] Configuration:
[2023/12/23 23:00:42] [ info]  flush time     | 1.000000 seconds
[2023/12/23 23:00:42] [ info]  grace          | 5 seconds
[2023/12/23 23:00:42] [ info]  daemon         | 0
[2023/12/23 23:00:42] [ info] ___________
[2023/12/23 23:00:42] [ info]  inputs:
[2023/12/23 23:00:42] [ info]      syslog
[2023/12/23 23:00:42] [ info] ___________
[2023/12/23 23:00:42] [ info]  filters:
[2023/12/23 23:00:42] [ info] ___________
[2023/12/23 23:00:42] [ info]  outputs:
[2023/12/23 23:00:42] [ info]      loki.0
[2023/12/23 23:00:42] [ info] ___________
[2023/12/23 23:00:42] [ info]  collectors:
[2023/12/23 23:00:42] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=1
[2023/12/23 23:00:42] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/12/23 23:00:42] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/12/23 23:00:42] [ info] [cmetrics] version=0.6.4
[2023/12/23 23:00:42] [ info] [ctraces ] version=0.3.1
[2023/12/23 23:00:42] [ info] [input:syslog:syslog.0] initializing
[2023/12/23 23:00:42] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2023/12/23 23:00:42] [debug] [syslog:syslog.0] created event channels: read=21 write=22
[2023/12/23 23:00:42] [debug] [downstream] listening on 0.0.0.0:5140
[2023/12/23 23:00:42] [ info] [in_syslog] UDP server binding 0.0.0.0:5140
[2023/12/23 23:00:42] [debug] [loki:loki.0] created event channels: read=24 write=25
[2023/12/23 23:00:42] [ info] [output:loki:loki.0] configured, hostname=loki.monitoring.svc:80
[2023/12/23 23:00:42] [ info] [sp] stream processor started
[2023/12/23 23:00:59] [debug] [input chunk] update output instances with new chunk size diff=179, records=1, input=syslog.0
[2023/12/23 23:00:59] [debug] [task] created task=0x7f15bf636640 id=0 OK
[2023/12/23 23:00:59] [debug] [output:loki:loki.0] could not translate record accessor
[2023/12/23 23:00:59] [debug] [upstream] KA connection #31 to loki.monitoring.svc:80 is connected
[2023/12/23 23:00:59] [debug] [http_client] not using http_proxy for header
[2023/12/23 23:00:59] [debug] [output:loki:loki.0] loki.monitoring.svc:80, HTTP status=204
[2023/12/23 23:00:59] [debug] [upstream] KA connection #31 to loki.monitoring.svc:80 is now available

However, the debug on Loki side suggests that fluent-bit sends empty message:

level=debug ts=2023-12-23T23:00:59.582026131Z caller=push.go:150 org_id=fake msg="push request parsed" path=/loki/api/v1/push contentType=application/json contentEncoding= bodySize="14 B" streams=0 entries=0 streamLabelsSize="0 B" entriesSize="0 B" totalSize="0 B" mostRecentLagMs=1703372459581
level=debug ts=2023-12-23T23:00:59.582247967Z caller=logging.go:101 traceID=27c85a1f74f7d06d orgID=fake msg="POST /loki/api/v1/push (204) 1.079872ms"

If I use the output to stdout, it works nicely:

[2023/12/23 23:02:45] [ info] [in_syslog] UDP server binding 0.0.0.0:5140
[2023/12/23 23:02:45] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2023/12/23 23:02:45] [ info] [sp] stream processor started
[2023/12/23 23:02:45] [ info] [output:stdout:stdout.0] worker #0 started
[2023/12/23 23:02:55] [debug] [input chunk] update output instances with new chunk size diff=184, records=1, input=syslog.0
[2023/12/23 23:02:55] [debug] [task] created task=0x7f1e5b436640 id=0 OK
[2023/12/23 23:02:55] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] syslog.0: [[1703372574.000000000, {}], {"pri"=>"189", "time"=>"Dec 23 23:02:54", "host"=>"lf1", "ident"=>"Ebra", "message"=>"%LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1 ("to: pe1_1/1/c1/5"), changed state to up"}]

It also works great with output to influxdb.

I've also tried the example with dummy data provided in the official guide and it actually worked.

As such it looks to me that there is some issue with combination of input/output.

To Reproduce

• Setup fluent-bit with syslog input, syslog parser and output to Loki
• Send some syslog data to fluent0bit
• Loki has a basic setup

Expected behavior
fluent-bit adds corresponding labels and streams based on the parsed syslog message when it sends data to Loki so that I can use Loki

Screenshots

Your Environment

• Version used: 2.2.0
• Configuration:

fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Parsers_File  /etc/fluent-bit/parsers.conf
        log_level     debug

    [INPUT]
        Name     syslog
        Parser   syslog-rfc3164
        Listen   0.0.0.0
        Port     5140
        Mode     udp

    [OUTPUT]
        Name          loki
        Match         *
        Host          loki.moniotring.svc
        Port          80
        Labels        job=fluentbit pri=$pri, host=$host, ident=$ident, pid=$pid
        Line_format   json

parser.conf: |
    [PARSER]
        Name        syslog-rfc3164
        Format      regex
        Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes:1.28.2, cilium network, rook-ceph for object storage (for Loki)
• Server type and version: Dell Gen 8
• Operating System and version: Kubernetes runs on Ubuntu 22.04
• Filters and plugins: syslog

Additional context
I'm currently using InfluxDB v2 to store all my metrics including the numerical ones and syslogs. I'm experimenting with Loki as a storage for my log messages.

[nokute78]

Could you try v2.2.1 ?
It is fixed by #8223

Note: #8184

[akarneliuk]

That worked indeed @nokute78 , thanks for prompt response. That worked indeed! Merry Christmas!