"SERVICE" column in DataDog Logs not showing "service.name" attribute: shows entire Json section in string form

[julealgon]

Component(s)

exporter/datadog

What happened?

Description

"Service" log column in DataDog showing entire "service" json section from log instead of just "service.name" as before.

Instead of just showing "SequenceNumberTracker API", the UI just shows the entire "service" section in serialized Json form.

(third column is the "SERVICE" column):

Information inside the log event:

This started happening after I upgraded the collector to the latest version (v0.91.0) from a fairly old version, v0.62.0

Steps to Reproduce

Upgrade to v0.91.0? It's hard to tell precisely which version actually introduced this issue, since we migrated from a fairly old release and did a big version jump.

Expected Result

I expect the "SERVICE" column to still contain just the service.name value, and not the entire Json section in string form.

Actual Result

The "SERVICE" column now contains the entire "service" section of the log event in Json string form.

Collector version

0.91.0

Environment information

Environment

OS: Windows Server 2019 Datacenter 10.0.17763 (Azure VM)
Compiler(if manually compiled): N/A (downloaded Windows amd64 binary)

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:

  hostmetrics:
    collection_interval: 10s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
      network:
      processes:

processors:
  batch:
    timeout: 10s

  resource:
    attributes:
    - key: deployment.environment
      value: "prod"
      action: upsert

  transform:
    trace_statements:
      - context: scope
        statements:
          - set(name, "aspnet.request") where name == "Microsoft.AspNetCore" or name == "OpenTelemetry.Instrumentation.AspNetCore" or name == "OpenTelemetry.Instrumentation.AspNet.Telemetry"
          - set(name, "sql_server.query") where name == "OpenTelemetry.Instrumentation.SqlClient"
          - set(name, "http.request") where name == "OpenTelemetry.Instrumentation.Http.HttpWebRequest" or name == "OpenTelemetry.Instrumentation.Http.HttpClient"

exporters:
  datadog:
    api:
      key: ${DATADOG_APIKEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, transform, batch]
      exporters: [datadog]
    metrics:
      receivers: [hostmetrics, otlp]
      processors: [resource, batch]
      exporters: [datadog]
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [datadog]

Log output

N/A

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @gbbr @dineshg13 @liustanley @songy23 @mackjmr

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[songy23]

👋 This is a known issue, please refer to https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datadogexporter#remap-otels-servicename-attribute-to-service-for-logs for a temporary fix. We're working on the long term fix by migrating the Datadog logs exporter to use logs agent.

[julealgon]

@songy23 when you say "it's a known issue", is there an actual issue registered somewhere that I can keep track of? I'll follow the instructions for the workaround in the readme, but I'd like to know when this is "fixed" so the workaround can be removed.

Can you link the other issue in here if it exists? If it doesn't, would it make sense to keep this one open instead for future reference when its actually implemented?

[songy23]

@julealgon Good point, I just filed #30099 to track in this repo. The migration is part of a big refactor effort and currently we don't have a public ETA to share.

[jdt]

@songy23 Is this issue considered fixed? I've been trying to follow the trail of issues related to the big refactor effort that involves using the log agent for the Datadog logs exporter and that seems finished and released with v0.100, but the workaround is still needed. Is there another issue that still needs to be fixed so the workaround can be removed?

[songy23]

@jdt the workaround is not needed as long as you enable the "exporter.datadogexporter.UseLogsAgentExporter" feature gate. It is disabled by default.

[jdt]

Thanks, that wasn't immediately obvious to me! Is there any timeline for when this would be moved to stable?

[songy23]

Ultimately we want to move it to stable but there is no exact timeline as of now. We will wait for at least 2 releases before moving the feature gate to beta, per the lifecycle spec