pkg/promtail: IETF Syslog (RFC5424) Support #1275

bastjan · 2019-11-18T10:38:28Z

What this PR does / why we need it:

Allow promtail to ingest IETF syslog (RFC5424).

Limitations:

No remote time support (tried it but got out-of-order errors due to data races across the network)
No UDP/ No BSD syslog (RFC3164).

Performance:

Tested on my somewhat slow machine (Intel Core i5-3210M). Used the loggen tool from syslog-ng with 200 parallel log streams. Minimal relabeling.

Configuration

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: syslog
  syslog:
    listen_address: localhost:1514
    labels:
      job: "syslog"
  relabel_configs:
  - source_labels: ['__syslog_message_hostname']
    target_label: 'hostname'

> loggen --syslog-proto --active-connections 200 --rate 10000 --stream localhost 1514
average rate = 78671.04 msg/sec, count=2124253, time=27.001, (average) msg size=260, bandwidth=19963.56 kB/sec

Which issue(s) this PR fixes:
Fixes #153, Fixes #935

Special notes for your reviewer:

~~Currently tracks the develop branch of github.com/influxdata/go-syslog since the v2.0.0 releases go mod support is broken.~~

Checklist

Documentation added
Tests updated

CLAassistant · 2019-11-18T10:38:45Z

All committers have signed the CLA.

rfratto · 2019-11-18T14:58:26Z

This is great, and something we definitely want in Promtail! I talked with the team and it seems the UDP limitation is the first thing we might want to get resolved since it's so common in the syslog world.

cc @slim-bean

bastjan · 2019-11-18T21:47:06Z

This is great, and something we definitely want in Promtail! I talked with the team and it seems the UDP limitation is the first thing we might want to get resolved since it's so common in the syslog world.

Great to hear! ~~I'm planning to bring this merge request to a mergeable state in the next two days.~~

I would really appreciate a review of the code/concept before adding more features.

I never implemented an UDP server before in go but it does not look too complicated... I think I can do an initial version over the next two weeks.

bastjan · 2019-11-26T17:15:46Z

I used this feature for a while now in our staging environment with syslog-ng in front for UDP/TLS. Works great with an average of around 4400 lines/s (90% of the traffic is simulated). Memory usage hovers around 75MB.

bastjan · 2019-11-27T08:20:29Z

After using loki and promtail some more I'm wondering if the syslog parser should move to the pipeline stage.

Something like this:

scrape_configs:
- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:1514
    labels:
      job: "syslog"
  entry_parser: syslog-rfc5452
  entry_parser_options:
    label_structured_data: yes

or

scrape_configs:
- job_name: syslog
  syslog:
    listen_address: 0.0.0.0:1514
    labels:
      job: "syslog"
  entry_parser: raw
  pipeline_stages:
  - syslog-rfc5452:
      label_structured_data: yes
  - timestamp:
      source: timetamp
      format: 2006-01-02T15:04:05.999999Z07:00 # (RFC3339Micro)

I haven't used promtail enough to form an opinion on it.

RFC5452 is quite static and the spec is usually correctly implemented. So I don't feel like there is any advantage in flexibility. RFC3164 (BSD Syslog) on the other hand is rarely correctly implemented, the format varies a lot, and I wonder if the parsing it should just be a pipeline stage.

cyriltovena · 2019-12-03T02:39:45Z

I don’t know much about syslog specifications (/cc @slim-bean ) but if there’s is no clear advantage I would go for the first config example.

I think it’s also better to keep the syslog configuration specific to the syslog scrape config.

I’m wondering if the use case of ingesting multiple specifications from one instance of promtail should be covered. Although we could run 2 instances with different config.

bastjan · 2019-12-03T07:43:58Z

I’m wondering if the use case of ingesting multiple specifications from one instance of promtail should be covered. Although we could run 2 instances with different config.

You could always have two or more scrape configs.

A nice feature to have is support for both specifications on the same listener. Syslog-NG does not support this natively, and we had some problems because not every appliance does support setting a non-standard port. I've written a small syslog "router" at work that accepts both protocols and routes them to the correct Syslog-NG listener.

pstibrany · 2019-12-03T13:02:09Z

Thank you for your continuous improvements on this PR. It is not yet clear to us, whether syslog receiving should be part of Promtail or not. Syslog ecosystem is quite big, and there may be better alternatives (eg. push logs from syslog-ng / rsyslog directly through their http or program outputs, or perhaps even adding direct support there). That's not to say that we don't want this contribution, but we're still investigating what the best way forward is.

pstibrany · 2019-12-03T13:26:53Z

I used this feature for a while now in our staging environment with syslog-ng in front for UDP/TLS.

Btw, since you're already running syslog-ng, how would direct exporter for syslog-ng to Loki work for you? I mean, without using promtail. I understand that you would lose some of the power of Promtail, on the other hand, direct export from syslog-ng would not lose logs if Promtail was down. It would also keep Promtail bit simpler, because it wouldn't need to understand different syslog formats or listening to additional tcp/udp ports.

bastjan · 2019-12-03T15:30:36Z

Btw, since you're already running syslog-ng, how would direct exporter for syslog-ng to Loki work for you?

That would work great, but a C syslog-ng plugin won't arrive in the mainstream distros for a while, and a python plugin is hard to package and deploy.

Having at least IETF syslog over TCP makes the integration of rsyslog and syslog-ng a breeze without writing plugins for both (and every other new product).

syslog-ng

destination d_loki {
  syslog("localhost" transport("tcp") port(6514));
};

rsyslog

action(type="omfwd" protocol="tcp" port="6514" Template="RSYSLOG_SyslogProtocol23Format")

I haven't tried to use the http output plugin of syslog-ng for loki yet, but the config will get complicated fast.

direct export from syslog-ng would not lose logs if Promtail was down

For more complicated setups there can still be a syslog-ng/rsyslog in front with the "reliable" parameter set.

pstibrany · 2019-12-03T15:37:09Z

I haven't tried to use the http output plugin of syslog-ng for loki yet, but the config will get complicated fast.

I looked at the documentation earlier today, and while basic http integration would not be that difficult, problem is that it would use one request per message. Syslog-ng http output also supports batching, but it would be difficult to massage output to what Loki currently expects.

bastjan · 2019-12-03T15:54:37Z

I looked at the documentation earlier today, and while basic http integration would not be that difficult, problem is that it would use one request per message. Syslog-ng http output also supports batching, but it would be difficult to massage output to what Loki currently expects.

Yeah that's what I expected. I created a python plugin prototype some time ago which uses the provided gRPC definitions https://gist.github.com/bastjan/cc8f25b6ea7395689e2e110836e312ce. I can provide the full repo but have to strip company stuff out before...

slim-bean · 2019-12-06T01:48:41Z

Sorry I'm late to the party here. I'm in favor of getting this added to promtail, where are we at with this, what do we need to get this merged?

I looked previously at the example config and would like to point out the entry_parser key has been deprecated and we should avoid using it, maybe we just need another key name for this.

pstibrany

I have few minor comments, but overall this looks pretty good, thank you!

After some thinking and internal discussion, I think we should start with TCP-only, and use it mostly as a writing destination for real syslog server, not as a replacement. In that light, perhaps we don't even need to support non-transparent-framing, but since it's so trivial to implement, we can keep that.

To your earlier question, I don't think this should be a pipeline, so we're good here.

docs/clients/promtail/configuration.md

pkg/promtail/targets/syslogtarget.go

pkg/promtail/targets/syslogtarget_test.go

pkg/promtail/targets/syslogparser/syslogparser.go

pstibrany · 2019-12-10T07:40:40Z

@bastjan btw, if you currently don't have time to take another look at this PR, I have some capacity now and can take over and polish it, if you want.

bastjan · 2019-12-10T08:16:32Z

@bastjan btw, if you currently don't have time to take another look at this PR, I have some capacity now and can take over and polish it, if you want.

Thanks for the review!

I've got time to polish the PR, should be able to finish it by today. :)

pstibrany · 2019-12-10T08:20:23Z

I've got time to polish the PR, should be able to finish it by today. :)

That's great, thank you! (No need to hurry :) )

* Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections

bastjan · 2019-12-10T13:12:30Z

@pstibrany I think I resolved most issues. Take a look and tell me what you think. :)

pstibrany

Thank you! Changes look good!

I would like to avoid using 2 goroutines per connection (see suggestion in syslogparser.go).

pkg/promtail/targets/syslogtarget.go

pkg/promtail/targets/syslogtarget_test.go

pkg/promtail/targets/syslogparser/syslogparser.go

pkg/promtail/targets/syslogtarget.go

pstibrany

Thank you for addressing my feedback! I've left one more comment about non-transparent-framing, please take a look.

pkg/promtail/targets/syslogtarget.go

pkg/promtail/targets/syslogparser/syslogparser_test.go

* Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing()

pstibrany

This looks great! I especially like documentation changes. Thank you very much for addressing my feedback!

pkg/promtail/targets/syslogparser/syslogparser.go

docs/clients/promtail/configuration.md

rfratto

Just some small comments around the docs.

docs/clients/promtail/configuration.md

cyriltovena · 2019-12-13T01:27:30Z

pkg/promtail/targets/syslogtarget.go

+// Details returns target-specific details.
+func (t *SyslogTarget) Details() interface{} {
+	return map[string]string{}
+}


I would love to get info useful for checking how does the target, I think we should create an issue, have you seen the /targets and /service-discovery page in Promtail?

I would love to get info useful for checking how does the target, I think we should create an issue, have you seen the /targets and /service-discovery page in Promtail?

Just took a look at those pages and indeed there is some useful information missing from the syslog target.

I think the most obvious candidates would be open connections in /targets and the listen address in /service-discovery.

Have to dig in a bit further to see how those pages look in different setups...

cyriltovena

Well this is one of those awesome contribution !

You really did a great job.

Thanks a lot

bastjan mentioned this pull request Nov 18, 2019

A way to ingest syslog messages over the network and into Loki #935

Closed

bastjan force-pushed the syslog-support branch 2 times, most recently from 7091c6a to 061a0fd Compare November 18, 2019 12:01

bastjan force-pushed the syslog-support branch from fcf1a91 to 685b4fb Compare November 19, 2019 09:06

rfratto assigned pstibrany Nov 19, 2019

bastjan force-pushed the syslog-support branch 2 times, most recently from c5f041c to 2cd1104 Compare November 19, 2019 20:19

bastjan marked this pull request as ready for review November 19, 2019 20:20

roidelapluie mentioned this pull request Nov 20, 2019

promtail Logback Appender #604

Closed

bastjan force-pushed the syslog-support branch from 2cd1104 to 4199dea Compare November 21, 2019 19:56

bastjan force-pushed the syslog-support branch from 00585e0 to 2e08712 Compare December 2, 2019 16:22

pstibrany reviewed Dec 9, 2019

View reviewed changes

bastjan and others added 2 commits December 10, 2019 13:59

IETF syslog (rfc5254) support

80753ea

Upgrade go-syslog to correctly released v2.0.1

ca12b6c

bastjan added 3 commits December 10, 2019 13:59

Incorporate feedback from new linter

487019a

Update documentation with relevant RFC sections.

51b3585

Incorporate feedback from review

2f7a65e

* Use context to shutdown server. * Use util.Backoff from cortex * Do not embed mutex into TestLabeledClient * Use strings.HasPrefix * Better naming of connectionsClosed -> openConnections

bastjan force-pushed the syslog-support branch from b725420 to 2f7a65e Compare December 10, 2019 13:00

pstibrany requested changes Dec 10, 2019

View reviewed changes

bastjan requested a review from pstibrany December 10, 2019 16:40

pstibrany reviewed Dec 11, 2019

View reviewed changes

pkg/promtail/targets/syslogtarget.go Show resolved Hide resolved

pkg/promtail/targets/syslogparser/syslogparser_test.go Outdated Show resolved Hide resolved

Incorporate further feedback from review

094ced2

* Callback instead of channel in ParseStream. Removes one running Goroutine per connection. * Improve parse error log level. * Move mutex above field it protects. * Use backoff.Ongoing()

bastjan force-pushed the syslog-support branch 2 times, most recently from 2a43852 to 03b587f Compare December 11, 2019 16:17

pstibrany approved these changes Dec 11, 2019

View reviewed changes

pkg/promtail/targets/syslogparser/syslogparser.go Outdated Show resolved Hide resolved

docs/clients/promtail/configuration.md Show resolved Hide resolved

Switch to "nontransparent" parser

30beb9a

bastjan force-pushed the syslog-support branch from 03b587f to 30beb9a Compare December 11, 2019 17:17

rfratto reviewed Dec 12, 2019

View reviewed changes

docs/clients/promtail/configuration.md Outdated Show resolved Hide resolved

docs/clients/promtail/configuration.md Outdated Show resolved Hide resolved

Further improvements to the documentation

c499fe1

bastjan requested a review from rfratto December 12, 2019 19:16

cyriltovena reviewed Dec 13, 2019

View reviewed changes

cyriltovena approved these changes Dec 13, 2019

View reviewed changes

cyriltovena merged commit f0f6f24 into grafana:master Dec 13, 2019

bastjan deleted the syslog-support branch December 13, 2019 14:17

johanfleury mentioned this pull request Jun 24, 2021

promtail: add support for (m)TLS in syslog receiver #3898

Closed

catap mentioned this pull request Apr 30, 2024

feat(promtail): Support of RFC3164 aka BSD Syslog #12810

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pkg/promtail: IETF Syslog (RFC5424) Support #1275

pkg/promtail: IETF Syslog (RFC5424) Support #1275

bastjan commented Nov 18, 2019 •

edited

Loading

CLAassistant commented Nov 18, 2019 •

edited

Loading

rfratto commented Nov 18, 2019

bastjan commented Nov 18, 2019 •

edited

Loading

bastjan commented Nov 26, 2019

bastjan commented Nov 27, 2019

cyriltovena commented Dec 3, 2019

bastjan commented Dec 3, 2019

pstibrany commented Dec 3, 2019

pstibrany commented Dec 3, 2019 •

edited

Loading

bastjan commented Dec 3, 2019

pstibrany commented Dec 3, 2019

bastjan commented Dec 3, 2019

slim-bean commented Dec 6, 2019

pstibrany left a comment

pstibrany commented Dec 10, 2019

bastjan commented Dec 10, 2019

pstibrany commented Dec 10, 2019

bastjan commented Dec 10, 2019

pstibrany left a comment

pstibrany left a comment

pstibrany left a comment

rfratto left a comment

cyriltovena Dec 13, 2019

bastjan Dec 13, 2019

cyriltovena left a comment

pkg/promtail: IETF Syslog (RFC5424) Support #1275

pkg/promtail: IETF Syslog (RFC5424) Support #1275

Conversation

bastjan commented Nov 18, 2019 • edited Loading

CLAassistant commented Nov 18, 2019 • edited Loading

rfratto commented Nov 18, 2019

bastjan commented Nov 18, 2019 • edited Loading

bastjan commented Nov 26, 2019

bastjan commented Nov 27, 2019

cyriltovena commented Dec 3, 2019

bastjan commented Dec 3, 2019

pstibrany commented Dec 3, 2019

pstibrany commented Dec 3, 2019 • edited Loading

bastjan commented Dec 3, 2019

pstibrany commented Dec 3, 2019

bastjan commented Dec 3, 2019

slim-bean commented Dec 6, 2019

pstibrany left a comment

Choose a reason for hiding this comment

pstibrany commented Dec 10, 2019

bastjan commented Dec 10, 2019

pstibrany commented Dec 10, 2019

bastjan commented Dec 10, 2019

pstibrany left a comment

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

rfratto left a comment

Choose a reason for hiding this comment

cyriltovena Dec 13, 2019

Choose a reason for hiding this comment

bastjan Dec 13, 2019

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

bastjan commented Nov 18, 2019 •

edited

Loading

CLAassistant commented Nov 18, 2019 •

edited

Loading

bastjan commented Nov 18, 2019 •

edited

Loading

pstibrany commented Dec 3, 2019 •

edited

Loading