[processor/filter] Add telemetry for dropped metrics, logs, and spans

[MacroPower]

Description:

Adds telemetry for metrics, logs, and spans that were intentionally dropped via a filterprocessor. Specifically, the following metrics are added:

otelcol_processor_filter_datapoints_filtered
otelcol_processor_filter_logs_filtered
otelcol_processor_filter_spans_filtered

Please let me know any feedback/thoughts on the naming or anything else!

Link to tracking Issue: #13169

Testing: I've used batchprocessor as an example for a couple of tests, Filter*ProcessorTelemetryWithOC. I kept the wrapping code so that OTEL versions can be easily added when that is ready in contrib. Overall the tests are not super comprehensive and I could improve them if needed, but as-is they were helpful for debugging.

Additionally, here's some stuff you can use for manually testing.

There might be a better way to do this, but I just used hostmetrics, filelog, and this article from honeycomb with otlp/http.

Note, this should be run from the root of the contrib repo.

Add/overwrite local/config.yaml, local/span.json, and run:

mkdir -p local

cat >local/config.yaml <<EOL
receivers:
  hostmetrics:
    collection_interval: 30s
    initial_delay: 1s
    scrapers:
      load:
  filelog:
    include:
      ## echo '{"timestamp":"2023-12-18 12:00:00","msg":"foo"}' >> /tmp/otel-test.log
      ## echo '{"timestamp":"2023-12-18 12:00:00","msg":"bar"}' >> /tmp/otel-test.log
      ## echo '{"timestamp":"2023-12-18 12:00:00","msg":"baz"}' >> /tmp/otel-test.log
      - /tmp/otel-test.log
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout: "%Y-%m-%d %H:%M:%S"
  otlp:
    protocols:
      ## curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @local/span.json
      http:

processors:
  filter/test:
    metrics:
      metric:
        # Should drop 2 of the 3 metrics, 5m average remains
        - 'name=="system.cpu.load_average.1m"'
        - 'name=="system.cpu.load_average.15m"'
    logs:
      log_record:
        # Should filter out "bar" and "baz"
        - 'IsMatch(body, ".*ba.*")'
    traces:
      span:
        # Should drop 1 of the 2 spans
        - 'name == "foobar"'

exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200

service:
  extensions: []
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [filter/test]
      exporters: [debug]
    logs:
      receivers: [filelog]
      processors: [filter/test]
      exporters: [debug]
    traces:
      receivers: [otlp]
      processors: [filter/test]
      exporters: [debug]

  telemetry:
    logs:
      level: debug
    metrics:
      level: detailed
      address: 0.0.0.0:8888
EOL

cat >local/span.json <<EOL
{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "test-with-curl"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "scope": {
            "name": "manual-test"
          },
          "spans": [
            {
              "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
              "spanId": "3c191d03fa8be065",
              "name": "spanitron",
              "kind": 2,
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": {
                "code": 1
              }
            },
            {
              "traceId": "71699b6fe85982c7c8995ea3d9c95df2",
              "spanId": "2f357b34d32f77b4",
              "name": "foobar",
              "kind": 2,
              "droppedAttributesCount": 0,
              "events": [],
              "droppedEventsCount": 0,
              "status": {
                "code": 1
              }
            }
          ]
        }
      ]
    }
  ]
}
EOL

make run

Send some data to the receivers:

# Write some logs
echo '{"timestamp":"2023-12-18 12:00:00","msg":"foo"}' >> /tmp/otel-test.log
echo '{"timestamp":"2023-12-18 12:00:00","msg":"bar"}' >> /tmp/otel-test.log
echo '{"timestamp":"2023-12-18 12:00:00","msg":"baz"}' >> /tmp/otel-test.log

# Write some spans
curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json" -d @local/span.json

Check the results:

$ curl http://localhost:8888/metrics | grep filtered
# HELP otelcol_processor_filter_datapoints_filtered Number of metric data points dropped by the filter processor
# TYPE otelcol_processor_filter_datapoints_filtered counter
otelcol_processor_filter_datapoints_filtered{filter="filter/test",service_instance_id="a99d9078-548b-425f-8466-3e9e2e9bf3b1",service_name="otelcontribcol",service_version="0.91.0-dev"} 2
# HELP otelcol_processor_filter_logs_filtered Number of logs dropped by the filter processor
# TYPE otelcol_processor_filter_logs_filtered counter
otelcol_processor_filter_logs_filtered{filter="filter/test",service_instance_id="a99d9078-548b-425f-8466-3e9e2e9bf3b1",service_name="otelcontribcol",service_version="0.91.0-dev"} 2
# HELP otelcol_processor_filter_spans_filtered Number of spans dropped by the filter processor
# TYPE otelcol_processor_filter_spans_filtered counter
otelcol_processor_filter_spans_filtered{filter="filter/test",service_instance_id="a99d9078-548b-425f-8466-3e9e2e9bf3b1",service_name="otelcontribcol",service_version="0.91.0-dev"} 1

Documentation: I do not believe we document telemetry exposed by components, but I could add this if needed.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: MacroPower / name: Jacob Colvin (2a7e764, 1c91896, d26d745, 0b6cbe1, e6497ac, 7137e77, b8d135e, f369d41, ceb5772, 1723cc4, 5fc1dbe, c7fafe4, 9cd8d28, fda6172, 17baacc, e9c8424, d3369ad, 91d1ab8, 4a300fd)
• ✅ login: TylerHelmuth / name: Tyler Helmuth (b7d2a42, 6606f71, 2f863ef)

[MacroPower]

It took a while but I think this is ready for review for real. Thanks for the feedback on my draft. Also wanted to say that Contribfest was really awesome! I had a ton of fun working with everyone, and I'm also glad to finally work on the collector a bit considering how frequently I use it day-to-day. 🙂

[TylerHelmuth]

@MacroPower can you share some details on the new metrics in action from your manual tests?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[TylerHelmuth]

Thanks for sticking with this!

[MacroPower]

@MacroPower can you share some details on the new metrics in action from your manual tests?

@TylerHelmuth I edited my PR description to include some stuff that can be used for manual testing. It's not the best and there's probably a better way I imagine, but hopefully it's useful regardless. 🙂

[TylerHelmuth]

@MacroPower thanks for the testing details, I was able to confirm locally as well. Take care of the conflicts and we'll be good to merge.

[vainiusd]

Hi, Noticed this only on the v0.92.0 release notes so sorry for the late feedback.
And also thanks, @MacroPower, for adding this!

The new metric names stand out a bit compared to others:

otelcol_receiver_refused_signal
otelcol_processor_refused_signal
otelcol_processor_filter_signal_filtered
otelcol_processor_dropped_signal
otelcol_exporter_enqueue_failed_signal
otelcol_exporter_send_failed_signal

Maybe it would be worth renaming to otelcol_processor_filter_filtered_signal?