[syslogreceiver] Add support for tcp framing

[StefanSa]

Describe the bug
We try to send from our Friewalls syslog data to the syslogreceiver via tcp - udp / rfc5424, but without success.
The following error message is generated.

udp:

2022-07-15T10:03:08.212+0200	error helper/transformer.go:110 Failed to process entry {"kind": "receiver", "name": "syslog", "pipeline": "logs", "operator_id": "syslog_input_internal_parser", "operator_type": "syslog_parser",
"error": "expecting a version value in the range 1-999 [col 4]", "action": "send", "entry":
{"observed_timestamp":"2022-07-15T10:03:08.212157456+02:00",
timestamp":"0001-01-01T00:00:00Z","body":"<30>device_name=\"SFW\" timestamp=\"2022-07-15T10:03:08+0200\" device_model=\"XG111\" device_serial_id=\"CCCCCCCCCCCC\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" fw_rule_id=\"56\" nat_rule_id=\"0\" fw_rule_type=\"USER\" ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"xfrm4\" src_mac=\"34:DB:FD:83:D8:09\" dst_mac=\"C8:4F:86:04:0E:70\" src_ip=\"172.16.34.224\" src_country=\"R1\" dst_ip=\"172.17.34.10\" dst_country=\"R1\" protocol=\"TCP\" src_port=51770 dst_port=10050 src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"VPN\" dst_zone=\"VPN\" con_event=\"Start\" con_id=\"1180777280\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"xfrm4\"","severity":0,"scope_name":""}}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/transformer.go:110
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/parser.go:144
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/parser.go:116
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/parser/syslog/syslog.go:113
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/writer.go:65
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/udp.(*Input).goHandleMessages.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/input/udp/udp.go:199

tcp:

"error": "expecting a priority value within angle brackets [col 0]", "action": "send", "entry":
{"observed_timestamp":"2022-07-15T13:03:40.953158581+02:00",
"timestamp":"0001-01-01T00:00:00Z","body":"958 <30>1 2022-07-15T13:03:40.941716+02:00 fw-1.test.local device_name=\"SFW\" - - -  timestamp=\"2022-07-15T13:03:40+0200\" device_model=\"X111\" device_serial_id=\"CCCCCCCCC\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" duration=10 fw_rule_id=\"56\" nat_rule_id=\"0\" fw_rule_type=\"USER\" ether_type=\"Unknown (0x0000)\" in_interface=\"Port1\" out_interface=\"xfrm4\" src_mac=\"34:DB:FD:83:D8:09\" dst_mac=\"C8:4F:86:04:0E:70\" src_ip=\"172.16.34.224\" src_country=\"R1\" dst_ip=\"172.17.35.112\" dst_country=\"R1\" protocol=\"TCP\" src_port=39018 dst_port=10050 packets_sent=5  packets_received=5 bytes_sent=339 bytes_received=358 src_zone_type=\"LAN\" src_zone=\"LAN\" dst_zone_type=\"VPN\" dst_zone=\"VPN\" con_event=\"Stop\" con_id=\"3422008576\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"Port1\" out_display_interface=\"xfrm4\"","severity":0,"scope_name":""}}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/transformer.go:110
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/parser.go:144
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/parser.go:116
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/parser/syslog/syslog.go:113
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/helper/writer.go:65
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/tcp.(*Input).goHandleMessages.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.55.0/operator/input/tcp/tcp.go:279

Steps to reproduce
As soon as syslog data is sent / received this error occurs.

What did you expect to see?
Receive Syslog UDP data without any problem.

What did you see instead?
Error message as already described above.

What version did you use?
Version: (v0.55.0)

What config did you use?
Config: (e.g. the yaml config file)

extensions:
  health_check:

receivers:
  syslog:
    udp (tcp):
      listen_address: "0.0.0.0:54526"
    protocol: rfc5424

processors:
  batch:

exporters:
  loki:
    endpoint: "https://172.16.34.221:3100/loki/api/v1/push"
    tenant_id: "syslog-fw"
    labels:
      attributes:
        severity: "severity"
  logging:

service:
  pipelines:
    logs:
      receivers: [syslog]
      processors: [batch]
      exporters: [logging, loki]
  extensions: [health_check]

Environment
OS: (e.g., "OpenLeap 15.4")

[github-actions]

Pinging code owners: @djaglowski

[github-actions]

Pinging code owners: @djaglowski

[StefanSa]

hi there,
sorry for the noise here.
So if you look at the specification of rfc5424, the syslog receiver behaves correctly.

• HEADER
  • PRI — or "priority", is a number calculated from Facility (what kind of message) code and Severity (how urgent is the message) code: PRI = Facility * 8 + Severity
    Reference tables for Facility and Severity in RFC 5424
  • VERSION — version is always "1" for RFC 5424
  • TIMESTAMP — valid timestamp examples (must follow ISO 8601 format with uppercase "T" and "Z")
    1985-04-12T23:20:50.52Z
    2003-08-24T05:14:15.000003-07:00
    - ("nil" value) if time not available
  • HOSTNAME — using FQDN (fully qualified domain name) is recommended, e.g. mymachine.example.com
  • APP-NAME — usually the name of the device or application that provided the message
  • PROCID — often used to provide the process name or process ID (is - "nil" in the example)
  • MSGID — should identify the type of message, more detail in RFC 5424 Section 6.2.7. MSGID
• STRUCTURED-DATA — named lists of key-value pairs for easy parsing and searching, more detail in RFC 5424 Section 6.3. STRUCTURED-DATA
• MSG — details about the event
  if the MSG is encoded in UTF-8, the string must start with the Unicode byte order mask (BOM), more detail in RFC 5424 Section 6.4. MSG

Looking at the error message from udp, expecting a version value in the range 1-999 [col 4]
the "1" (version) is missing,

And for tcp, there is a value for PRI that does not belong there. expecting a priority value within angle brackets [col 0] -> "body":"958 <30>1

Tcp is special because the syslog sender from the firewalls can only do udp. For this reason i have created a syslog relay from udp to tcp with rsyslog.
The value 958 is the current message length because framing was enabled during forwarding TCP_Framing="octet-counted" and therefore the syslogreceiver could not decode the message.
It would be good if the syslogreceiver could also recognize / decode tcp_framing.
What you can see is that the firewalls (sophos xg) do not adhere to any rfc5424 standard, which is bad.

[djaglowski]

The syslog receiver relies on influxdata/go-syslog for parsing rules. It looks like the module provides some handling of framing, though we are not currently utilizing it. Do you think these capabilities can be used to address your suggestion? If so, what changes to the configuration would be necessary?

[StefanSa]

Hi Daniel @djaglowski
I think this would work.
TCP_Framing knows two options traditional or octet-counted [default traditional] TCP_Framing.
You could add a flag / option tcp_framing in the exporter with these two options.

  syslog:
    tcp:
      listen_address: "0.0.0.0:54526"
    protocol: rfc5424
    tcp_framing: traditional / octet-counted

For example, if the octet-counted option is enabled, the prefix must be evaluated with the current message lenght.
-> example file

[djaglowski]

I'm rather unfamiliar with some of the nuances here, so would you mind helping me pin down some details?

1. Am I correct in thinking that tcp_framing is not currently being used in any form? Or is it always in play (ie. traditional enabled by default)?
   1a. If it is not enabled by default, then would it make sense that the tcp_framing option should have three settings, none | traditional | octet-counted, and none should be the default?
2. It's not immediately clear how framing relates to syslog protocols rfc5424 and rfc3164. Would a framing parser be applied instead of these protocols, or as a step before one or both of these protocols?
3. Do you have any interest in contributing the implementation of this feature? I am happy to advise on the particulars of this codebase as necessary. Otherwise, would you be willing to specify a few of specific inputs that you would expect to parse successfully?

[StefanSa]

Daniel @djaglowski
I'm not really the expert on syslog either, but as a slightly older admin, have to deal with it often 😇
I find as a good template you can take from telegraf the 'syslog input plugin', in the readme the options are described quite in detail.
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/syslog
The telegraf syslog plugin, currently working without problems.
I'm happy to help you as much as i can.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/syslog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

TCP framing was added in #15358