[receiver/filelog] attributes.log.file.name attribute field not found when operated on within filelog stanza

[GuyAtTheFront]

Component(s)

receiver/filelog

What happened?

Description

attributes.file.log.name field not found when move/copy operators are applied to said field

Expected Result

Since this datapoint is injected by receiver/filelog, it should be configurable at this point and not require enhancement by downstream receivers / processors.

Actual Result

field not found, error in log output below

Collector version

v0.88.0

Environment information

Environment

OS: Ubuntu 22.04

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [/var/log/app/sample.log]
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}) \[?(?P<sev>[A-Z]*)\]? (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S.%L'
        severity:
          parse_from: attributes.sev

      - type: copy
        from: attributes.log.file.name
        to: resource.filename

    resource:
      deployment.env: "dev"
      service.name: "app"
      service.version: "1.0.0"
      host.name: ${env:HOSTNAME}

processors:
  batch:

exporters:
  otlphttp:
    endpoint: http://otel-collector:4318

  debug:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [debug, otlphttp]

Log output

2023-12-07T10:30:48.829Z        error   helper/transformer.go:98        Failed to process entry {"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "copy", "operator_type": "copy", "error": "copy: from field does not exist in this entry: attributes.log.file.name", "action": "send", "entry": {"observed_timestamp":"2023-12-07T10:30:48.827639436Z","timestamp":"2023-12-07T10:00:40.123Z","body":"2023-12-07 10:00:40.123 [ERROR] This is a test error message 11111111111111","attributes":{"log.file.name":"sample.log","msg":"This is a test error message 11111111111111","sev":"ERROR","time":"2023-12-07 10:00:40.123"},"resource":{"deployment.env":"dev","host.name":"HOSTNAME","service.name":"app","service.version":"1.0.0"},"severity_text":"ERROR","severity":17,"scope_name":""}}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/transformer.go:90
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/copy.(*Transformer).Process
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/transformer/copy/copy.go:73
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:122
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/parser/regex/regex.go:99
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/helper/writer.go:53
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emit
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/operator/input/file/file.go:52
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/internal/reader/reader.go:106
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:156

Additional context

1. attributes.log.file.name does exist in the error log, just cannot be operated on in the filelog stanza

{
  "kind": "receiver",
  "name": "filelog",
  "data_type": "logs",
  "operator_id": "copy",
  "operator_type": "copy",
  "error": "copy: from field does not exist in this entry: attributes.log.file.name",
  "action": "send",
  "entry": {
    "observed_timestamp": "2023-12-07T10:30:48.827639436Z",
    "timestamp": "2023-12-07T10:00:40.123Z",
    "body": "2023-12-07 10:00:40.123 [ERROR] This is a test error message 11111111111111",
    "attributes": {
      "log.file.name": "sample.log",
      "msg": "This is a test error message 11111111111111",
      "sev": "ERROR",
      "time": "2023-12-07 10:00:40.123"
    },
    "resource": {
      "deployment.env": "dev",
      "host.name": "HOSTNAME",
      "service.name": "app",
      "service.version": "1.0.0"
    },
    "severity_text": "ERROR",
    "severity": 17,
    "scope_name": ""
  }
}

2. Copying another attribute to resource like below works:

      - type: copy
        from: attributes.sev
        to: resource.sev

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @GuyAtTheFront, I found this in documentation here:

If a field contains a dot in it, a field can alternatively use bracket syntax for traversing through a map. For example, to select the key k8s.cluster.name on the entry's body, you can use the field body["k8s.cluster.name"].

Can you try again using attributes["log.file.name"]?

      - type: copy
        from: attributes["log.file.name"]
        to: resource.filename

[GuyAtTheFront]

Hi @crobert-1, can confirm your suggestion works. Thank you!

I'm not a golang developer (yet), so I can't make sense of why behavior would be different under the hood. Leaving this to owner / community to decide it the issue should be closed.

Edit: (Clicked the wrong button, re-opened for owner / community to decide)

[crobert-1]

I think this was done to remove ambiguity in the structure of JSON data with periods, and this is specific to the functionality of the pkg/stanza package.

Stealing and modifying the example dataset from here:

{
  "timestamp": "",
  "attributes": {},
  "body": {
    "key.with.periods": "value1",
    "key": {
      "with": {
        "periods": "value2"
      }
    }
  }
}

This is valid data that could potentially be operated on, so there needs to be a way to differentiate sub-fields vs. full key names. This syntax was chosen to simply be able to differentiate body.key.with.periods (returns value2) from body["key.with.periods"] (returns value1).