Promtail: doesn't accept syslog stream from syslog-ng

[secustor]

Describe the bug
Promtail fails to interpret the syslog stream provided by syslog-ng

To Reproduce
Steps to reproduce the behavior:

1. Setup promtail and syslog-ng following this guide: https://github.com/grafana/loki/blob/v1.5.0/docs/clients/promtail/scraping.md#syslog-receiver
2. Send syslog messages to syslog-ng
3. This leads to warnings in the log of promtail

Aug 26 16:10:25 promtail.example.com promtail[16297]: level=warn ts=2020-08-26T14:10:25.019204216Z caller=syslogtarget.go:176 msg="error parsing syslog stream" err="found EOF after \"<13>1 2020-08-26T16:09:26+02:00 192.168.56.254 .... policytype=\"local-in-policy\" service=\", expecting a SYSLOGMSG containing 8246 octets"

Expected behavior
I expected promtail to accept the stream and forward it to loki.

Environment:

• Infrastructure: VM
• Deployment tool: Ansible

Screenshots, Promtail config, or terminal output
promtail:

  - job_name: syslog
    relabel_configs:
    - source_labels:
      - __syslog_message_hostname
      target_label: fqdn
    syslog:
      idle_timeout: 60s
      label_structured_data: true
      labels:
        job: syslog
      listen_address: 0.0.0.0:1514

syslog-ng:

@version: 3.5
@include "scl.conf"

source s_syslog {
	 tcp( ip(0.0.0.0) port(514) );
    };

destination d_loki {
        syslog("localhost" transport("tcp") port(1514));
    };

log {
	source(s_syslog);
	destination(d_loki);
};

If applicable, add any output to help explain your problem.

[adityacs]

@secustor Could you please refer the discussion in #2232 ? It might help you figure out your issue.

[secustor]

I think this are two different issues as:

• I'm not using any pipeline_stages yet
• Promtail can not process any logs from the syslog scrape_config
• the logs look different

[secustor]

We decided instead of using the newer RFC6587 the legacy RFC3195 protocol

So instead of: --RFC6587--> syslog-ng --RFC5424--> promtail --> Loki
We do now: --RFC3195--> syslog-ng --RFC5424--> promtail --> Loki

with the same configs as described in issue.