Skip to content

Commit

Permalink
[syslogreceiver] Add for RFC 6587 octet counting and non-transparent-…
Browse files Browse the repository at this point in the history
…framing (open-telemetry#15358)

* Added support for syslog rfc 6587 octet counting and non-transparent-framing

Signed-off-by: Corbin Phelps <corbin.phelps@bluemedora.com>
  • Loading branch information
Corbin Phelps authored and shalper2 committed Dec 6, 2022
1 parent 699c304 commit 1b6c1bd
Show file tree
Hide file tree
Showing 16 changed files with 331 additions and 33 deletions.
16 changes: 16 additions & 0 deletions .chloggen/syslogreceiver-rfc6587-support.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: "enhancement"

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: syslogreceiver

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Added RFC 6587 Octet Counting and Non-Transparent-Framing support.

# One or more tracking issues related to the change
issues: [8390]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:
1 change: 1 addition & 0 deletions cmd/configschema/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ require (
github.com/knadh/koanf v1.4.3 // indirect
github.com/kolo/xmlrpc v0.0.0-20201022064351-38db28db192b // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/leodido/ragel-machinery v0.0.0-20181214104525-299bdde78165 // indirect
github.com/leoluk/perflib_exporter v0.2.0 // indirect
github.com/lib/pq v1.10.7 // indirect
github.com/lightstep/go-expohisto v1.0.0 // indirect
Expand Down
1 change: 1 addition & 0 deletions cmd/configschema/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,7 @@ require (
github.com/knadh/koanf v1.4.3 // indirect
github.com/kolo/xmlrpc v0.0.0-20201022064351-38db28db192b // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/leodido/ragel-machinery v0.0.0-20181214104525-299bdde78165 // indirect
github.com/leoluk/perflib_exporter v0.2.0 // indirect
github.com/lib/pq v1.10.7 // indirect
github.com/lightstep/go-expohisto v1.0.0 // indirect
Expand Down
1 change: 1 addition & 0 deletions go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 14 additions & 12 deletions pkg/stanza/docs/operators/syslog_parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,20 @@ The `syslog_parser` operator parses the string-type field selected by `parse_fro

### Configuration Fields

| Field | Default | Description |
| --- | --- | --- |
| `id` | `syslog_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `parse_from` | `body` | The [field](../types/field.md) from which the value will be parsed. |
| `parse_to` | `attributes` | The [field](../types/field.md) to which the value will be parsed. |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](../types/on_error.md). |
| `protocol` | required | The protocol to parse the syslog messages as. Options are `rfc3164` and `rfc5424`. |
| `location` | `UTC` | The geographic location (timezone) to use when parsing the timestamp (Syslog RFC 3164 only). The available locations depend on the local IANA Time Zone database. [This page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) contains many examples, such as `America/New_York`. |
| `timestamp` | `nil` | An optional [timestamp](../types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator |
| `severity` | `nil` | An optional [severity](../types/severity.md) block which will parse a severity field before passing the entry to the output operator |
| `if` | | An [expression](../types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |
| Field | Default | Description |
| --- | --- | --- |
| `id` | `syslog_parser` | A unique identifier for the operator. |
| `output` | Next in pipeline | The connected operator(s) that will receive all outbound entries. |
| `parse_from` | `body` | The [field](../types/field.md) from which the value will be parsed. |
| `parse_to` | `attributes` | The [field](../types/field.md) to which the value will be parsed. |
| `on_error` | `send` | The behavior of the operator if it encounters an error. See [on_error](../types/on_error.md). |
| `protocol` | required | The protocol to parse the syslog messages as. Options are `rfc3164` and `rfc5424`. |
| `location` | `UTC` | The geographic location (timezone) to use when parsing the timestamp (Syslog RFC 3164 only). The available locations depend on the local IANA Time Zone database. [This page](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) contains many examples, such as `America/New_York`. |
| `enable_octet_counting` | `false` | Wether or not to enable [RFC 6587](https://www.rfc-editor.org/rfc/rfc6587#section-3.4.1) Octet Counting on syslog parsing (Syslog RFC 5424 only). |
| `non_transparent_framing_trailer` | `nil` | The framing trailer, either `LF` or `NUL`, when using [RFC 6587](https://www.rfc-editor.org/rfc/rfc6587#section-3.4.2) Non-Transparent-Framing (Syslog RFC 5424 only). |
| `timestamp` | `nil` | An optional [timestamp](../types/timestamp.md) block which will parse a timestamp field before passing the entry to the output operator |
| `severity` | `nil` | An optional [severity](../types/severity.md) block which will parse a severity field before passing the entry to the output operator |
| `if` | | An [expression](../types/expression.md) that, when set, will be evaluated to determine whether this operator should be used for the given entry. This allows you to do easy conditional parsing without branching logic with routers. |

### Embedded Operations

Expand Down
1 change: 1 addition & 0 deletions pkg/stanza/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ require (
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/knadh/koanf v1.4.3 // indirect
github.com/leodido/ragel-machinery v0.0.0-20181214104525-299bdde78165 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
Expand Down
1 change: 1 addition & 0 deletions pkg/stanza/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions pkg/stanza/operator/input/syslog/syslog.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
package syslog // import "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/syslog"

import (
"errors"
"fmt"

"go.uber.org/zap"
Expand Down Expand Up @@ -88,9 +89,15 @@ func (c Config) Build(logger *zap.SugaredLogger) (operator.Operator, error) {
}

if c.UDP != nil {

udpInputCfg := udp.NewConfigWithID(inputBase.ID() + "_internal_udp")
udpInputCfg.BaseConfig = *c.UDP

// Octet counting and Non-Transparent-Framing are invalid for UDP connections
if syslogParserCfg.EnableOctetCounting || syslogParserCfg.NonTransparentFramingTrailer != nil {
return nil, errors.New("octet_counting and non_transparent_framing is not compatible with UDP")
}

udpInput, err := udpInputCfg.Build(logger)
if err != nil {
return nil, fmt.Errorf("failed to resolve upd config: %w", err)
Expand Down
17 changes: 11 additions & 6 deletions pkg/stanza/operator/input/syslog/syslog_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,17 @@ func TestInput(t *testing.T) {
require.NoError(t, err)

for _, tc := range cases {
t.Run(fmt.Sprintf("TCP-%s", tc.Name), func(t *testing.T) {
InputTest(t, NewConfigWithTCP(&tc.Config.BaseConfig), tc)
})
t.Run(fmt.Sprintf("UDP-%s", tc.Name), func(t *testing.T) {
InputTest(t, NewConfigWithUDP(&tc.Config.BaseConfig), tc)
})
if tc.ValidForTCP {
t.Run(fmt.Sprintf("TCP-%s", tc.Name), func(t *testing.T) {
InputTest(t, NewConfigWithTCP(&tc.Config.BaseConfig), tc)
})
}

if tc.ValidForUDP {
t.Run(fmt.Sprintf("UDP-%s", tc.Name), func(t *testing.T) {
InputTest(t, NewConfigWithUDP(&tc.Config.BaseConfig), tc)
})
}
}
}

Expand Down
92 changes: 92 additions & 0 deletions pkg/stanza/operator/parser/syslog/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,98 @@ func TestParserMissingProtocol(t *testing.T) {
require.Contains(t, err.Error(), "missing field 'protocol'")
}

func TestRFC6587ConfigOptions(t *testing.T) {
validFramingTrailer := NULTrailer
invalidFramingTrailer := "bad"
testCases := []struct {
desc string
cfg *Config
errContents string
}{
{
desc: "Octet Counting with RFC3164",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC3164,
EnableOctetCounting: true,
},
},
errContents: "octet_counting and non_transparent_framing are only compatible with protocol rfc5424",
},
{
desc: "Non-Transparent-Framing with RFC3164",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC3164,
NonTransparentFramingTrailer: &validFramingTrailer,
},
},
errContents: "octet_counting and non_transparent_framing are only compatible with protocol rfc5424",
},
{
desc: "Non-Transparent-Framing and Octet counting both enabled with RFC5424",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC5424,
NonTransparentFramingTrailer: &validFramingTrailer,
EnableOctetCounting: true,
},
},
errContents: "only one of octet_counting or non_transparent_framing can be enabled",
},
{
desc: "Valid Octet Counting",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC5424,
NonTransparentFramingTrailer: nil,
EnableOctetCounting: true,
},
},
errContents: "",
},
{
desc: "Valid Non-Transparent-Framing Trailer",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC5424,
NonTransparentFramingTrailer: &validFramingTrailer,
EnableOctetCounting: false,
},
},
errContents: "",
},
{
desc: "Invalid Non-Transparent-Framing Trailer",
cfg: &Config{
ParserConfig: helper.NewParserConfig(operatorType, operatorType),
BaseConfig: BaseConfig{
Protocol: RFC5424,
NonTransparentFramingTrailer: &invalidFramingTrailer,
EnableOctetCounting: false,
},
},
errContents: "invalid non_transparent_framing_trailer",
},
}

for _, tc := range testCases {
t.Run(tc.desc, func(t *testing.T) {
_, err := tc.cfg.Build(testutil.Logger(t))
if tc.errContents != "" {
require.ErrorContains(t, err, tc.errContents)
} else {
require.NoError(t, err)
}
})
}
}

func TestParserInvalidLocation(t *testing.T) {
config := NewConfig()
config.Location = "not_a_location"
Expand Down
97 changes: 97 additions & 0 deletions pkg/stanza/operator/parser/syslog/data.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
package syslog // import "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog"

import (
"strings"
"time"

"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/entry"
Expand All @@ -25,6 +26,10 @@ type Case struct {
Config *Config
Input *entry.Entry
Expect *entry.Entry

// These signal if a test is valid for UDP and/or TCP protocol
ValidForTCP bool
ValidForUDP bool
}

func testLocations() (map[string]*time.Location, error) {
Expand All @@ -50,6 +55,14 @@ func CreateCases(basicConfig func() *Config) ([]Case, error) {
return nil, err
}

// We need to build out the Non-Transparent-Framing body to ensure we control the Trailer byte
nonTransparentBodyBuilder := strings.Builder{}
nonTransparentBodyBuilder.WriteString(`<86>1 2015-08-05T21:58:59.693Z 192.168.2.132 SecureAuth0 23108 ID52020 [SecureAuth@27389 UserHostAddress="192.168.2.132" Realm="SecureAuth0" UserID="Tester2" PEN="27389"] Found the user for retrieving user's profile`)
nonTransparentBodyBuilder.WriteByte(0x00)
nonTransparentBody := nonTransparentBodyBuilder.String()

nulFramingTrailer := NULTrailer

var cases = []Case{
{
"RFC3164",
Expand All @@ -75,6 +88,8 @@ func CreateCases(basicConfig func() *Config) ([]Case, error) {
},
Body: "<34>Jan 12 06:30:00 1.2.3.4 apache_server: test message",
},
true,
true,
},
{
"RFC3164Detroit",
Expand All @@ -100,6 +115,8 @@ func CreateCases(basicConfig func() *Config) ([]Case, error) {
},
Body: "<34>Jan 12 06:30:00 1.2.3.4 apache_server: test message",
},
true,
true,
},
{
"RFC3164Athens",
Expand All @@ -125,6 +142,8 @@ func CreateCases(basicConfig func() *Config) ([]Case, error) {
},
Body: "<34>Jan 12 06:30:00 1.2.3.4 apache_server: test message",
},
true,
true,
},
{
"RFC5424",
Expand Down Expand Up @@ -160,6 +179,84 @@ func CreateCases(basicConfig func() *Config) ([]Case, error) {
},
Body: `<86>1 2015-08-05T21:58:59.693Z 192.168.2.132 SecureAuth0 23108 ID52020 [SecureAuth@27389 UserHostAddress="192.168.2.132" Realm="SecureAuth0" UserID="Tester2" PEN="27389"] Found the user for retrieving user's profile`,
},
true,
true,
},
{
"RFC6587 Octet Counting",
func() *Config {
cfg := basicConfig()
cfg.Protocol = RFC5424
cfg.EnableOctetCounting = true
return cfg
}(),
&entry.Entry{
Body: `215 <86>1 2015-08-05T21:58:59.693Z 192.168.2.132 SecureAuth0 23108 ID52020 [SecureAuth@27389 UserHostAddress="192.168.2.132" Realm="SecureAuth0" UserID="Tester2" PEN="27389"] Found the user for retrieving user's profile`,
},
&entry.Entry{
Timestamp: time.Date(2015, 8, 5, 21, 58, 59, 693000000, time.UTC),
Severity: entry.Info,
SeverityText: "info",
Attributes: map[string]interface{}{
"appname": "SecureAuth0",
"facility": 10,
"hostname": "192.168.2.132",
"message": "Found the user for retrieving user's profile",
"msg_id": "ID52020",
"priority": 86,
"proc_id": "23108",
"structured_data": map[string]map[string]string{
"SecureAuth@27389": {
"PEN": "27389",
"Realm": "SecureAuth0",
"UserHostAddress": "192.168.2.132",
"UserID": "Tester2",
},
},
"version": 1,
},
Body: `215 <86>1 2015-08-05T21:58:59.693Z 192.168.2.132 SecureAuth0 23108 ID52020 [SecureAuth@27389 UserHostAddress="192.168.2.132" Realm="SecureAuth0" UserID="Tester2" PEN="27389"] Found the user for retrieving user's profile`,
},
true,
false,
},
{
"RFC6587 Non-Transparent-framing",
func() *Config {
cfg := basicConfig()
cfg.Protocol = RFC5424
cfg.NonTransparentFramingTrailer = &nulFramingTrailer
return cfg
}(),
&entry.Entry{
Body: nonTransparentBody,
},
&entry.Entry{
Timestamp: time.Date(2015, 8, 5, 21, 58, 59, 693000000, time.UTC),
Severity: entry.Info,
SeverityText: "info",
Attributes: map[string]interface{}{
"appname": "SecureAuth0",
"facility": 10,
"hostname": "192.168.2.132",
"message": "Found the user for retrieving user's profile",
"msg_id": "ID52020",
"priority": 86,
"proc_id": "23108",
"structured_data": map[string]map[string]string{
"SecureAuth@27389": {
"PEN": "27389",
"Realm": "SecureAuth0",
"UserHostAddress": "192.168.2.132",
"UserID": "Tester2",
},
},
"version": 1,
},
Body: nonTransparentBody,
},
true,
false,
},
}

Expand Down
Loading

0 comments on commit 1b6c1bd

Please sign in to comment.