Not able to send logs to collector using auto instrument

[droger88]

Describe the bug

We have deployed a Java application with auto instrument sending data to collector, the metrics and trace works perfect fine. But we don't see any log data.

Steps to reproduce

1. Instrument Java application using auto instrument via CRD, configuration below

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
  namespace: apps
spec:
  exporter:
    endpoint: http://adot-collector.apps.svc.cluster.local:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  java:
    image: otel/autoinstrumentation-java:1.30.0

2. enable OTEL_LOGS_EXPORTER=logging-otlp via environment variable, then i can see logs

{
    "timestamp": "2023-10-16T14:57:55.111-04:00",
    "sequence": 150,
    "loggerClassName": "org.jboss.logging.Logger",
    "loggerName": "com.ServiceFilter",
    "level": "INFO",
    "message": "received request [endpoint] of method [POST] with payload [payload]",
    "threadName": "executor-thread-2",
    "threadId": 29,
    "mdc": {
        "trace_flags": "01",
        "trace_id": "9cfe9e1f9cef829958e244f651518580",
        "referenceId": "dd1e5ca7-f5c3-4f89-9dd7-2e54a57b04d1",
        "span_id": "305bc45933a44fb8",
        "orderId": ""
    },
    "ndc": "",
    "hostName": "fake-api",
    "processName": "server.jar",
    "processId": 1
}
[otel.javaagent 2023-10-16 14: 57: 55: 317 -0400] [BatchLogRecordProcessor_WorkerThread-1] INFO io.opentelemetry.exporter.logging.otlp.OtlpJsonLoggingLogRecordExporter - 
{
    "resource": {
        "attributes": [
            {
                "key": "container.id",
                "value": {
                    "stringValue": "dd11e04b8e548b2032a7f58dc1c00f8494b4bdcdcb43220fb49d2b88e74ded94"
                }
            },
            {
                "key": "host.arch",
                "value": {
                    "stringValue": "amd64"
                }
            },
            {
                "key": "host.name",
                "value": {
                    "stringValue": "fake-api"
                }
            },
            {
                "key": "k8s.container.name",
                "value": {
                    "stringValue": "app"
                }
            },
            {
                "key": "k8s.deployment.name",
                "value": {
                    "stringValue": "fake-deployment"
                }
            },
            {
                "key": "k8s.namespace.name",
                "value": {
                    "stringValue": "apps"
                }
            },
            {
                "key": "k8s.node.name",
                "value": {
                    "stringValue": "node"
                }
            },
            {
                "key": "k8s.pod.name",
                "value": {
                    "stringValue": "fake-api"
                }
            },
            {
                "key": "k8s.replicaset.name",
                "value": {
                    "stringValue": "fake-deployment-5bc75875d"
                }
            },
            {
                "key": "os.description",
                "value": {
                    "stringValue": "Linux 5.10.186-179.751.amzn2.x86_64"
                }
            },
            {
                "key": "os.type",
                "value": {
                    "stringValue": "linux"
                }
            },
            {
                "key": "process.command_args",
                "value": {
                    "arrayValue": {
                        "values": [
                            {
                                "stringValue": "/opt/java/openjdk/bin/java"
                            },
                            {
                                "stringValue": "-jar"
                            },
                            {
                                "stringValue": "server.jar"
                            }
                        ]
                    }
                }
            },
            {
                "key": "process.executable.path",
                "value": {
                    "stringValue": "/opt/java/openjdk/bin/java"
                }
            },
            {
                "key": "process.pid",
                "value": {
                    "intValue": "1"
                }
            },
            {
                "key": "process.runtime.description",
                "value": {
                    "stringValue": "Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.8.1+1"
                }
            },
            {
                "key": "process.runtime.name",
                "value": {
                    "stringValue": "OpenJDK Runtime Environment"
                }
            },
            {
                "key": "process.runtime.version",
                "value": {
                    "stringValue": "17.0.8.1+1"
                }
            },
            {
                "key": "service.name",
                "value": {
                    "stringValue": "fake-server"
                }
            },
            {
                "key": "service.version",
                "value": {
                    "stringValue": "a1aef8dc"
                }
            },
            {
                "key": "telemetry.auto.version",
                "value": {
                    "stringValue": "1.30.0"
                }
            },
            {
                "key": "telemetry.sdk.language",
                "value": {
                    "stringValue": "java"
                }
            },
            {
                "key": "telemetry.sdk.name",
                "value": {
                    "stringValue": "opentelemetry"
                }
            },
            {
                "key": "telemetry.sdk.version",
                "value": {
                    "stringValue": "1.30.1"
                }
            }
        ]
    },
    "scopeLogs": [
        {
            "scope": {
                "name": "com.ServiceFilter",
                "attributes": []
            },
            "logRecords": [
                {
                    "timeUnixNano": "1697482675111000000",
                    "observedTimeUnixNano": "1697482675111346304",
                    "severityNumber": 9,
                    "severityText": "INFO",
                    "body": {
                        "stringValue": "received request [endpoint] of method [POST] with payload [payload]"
                    },
                    "attributes": [],
                    "flags": 1,
                    "traceId": "9cfe9e1f9cef829958e244f651518580",
                    "spanId": "305bc45933a44fb8"
                }
            ]
        }
    ],
    "schemaUrl": "https://opentelemetry.io/schemas/1.21.0"
}

3. collector configuration, I have enable the logging pipeline for logs, but from the collector logs, it does not show anything related for the logs sent from the app.

extensions:
  health_check:

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
  resource:
    attributes:
      - key: host.id
        from_attribute: host.name
        action: upsert
      - key: deployment.environment
        action: insert
        value: dev
      - key: service.instance.id
        from_attribute: k8s.pod.name
        action: upsert
      - key: k8s.cluster.name
        value: ${CLUSTER_NAME}
        action: upsert
  resourcedetection:
    detectors: [ eks, ecs ]
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
      node_from_env_var: KUBE_NODE_NAME
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.pod.start_time
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.name
          - from: resource_attribute
            name: k8s.namespace.name
exporters:
  otlp:
    endpoint: $OTEL_EXPORTER_OTLP_ENDPOINT
    headers:
      api-key: $LICENSE_KEY
  logging:
    verbosity: detailed
service:
  extensions: [health_check]
  pipelines:
    metrics/otlp:
      receivers: [ otlp ]
      processors: [resourcedetection, k8sattributes, resource, batch]
      exporters: [ otlp ]
    traces:
      receivers: [ otlp ]
      processors: [ resourcedetection, k8sattributes, resource, batch ]
      exporters: [ otlp ]
    logs:
      receivers: [ otlp ]
      processors: [ resourcedetection, k8sattributes, resource, batch ]
      exporters: [ logging ]

Expected behavior

logs data with resource attributes sent to collector and backend (newrelic)

Actual behavior

we do not see the log data sent to collector

Javaagent or library instrumentation version

otel/autoinstrumentation-java:1.30.0

Environment

JDK: Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.8.1+1
OS: Linux 5.10.186-179.751.amzn2.x86_64

Additional context

No response

[laurit]

By default agent does not enable log exporter, try with OTEL_LOGS_EXPORTER=otlp

[droger88]

@laurit Thanks, it works now

[trask]

Just a note for future readers, this behavior will change in the 2.0 release, at which time logs will be captured by default (#9008)

[droger88]

another question, we noticed that some of the json attribute is not "parsed" such as loggerName, threadName, or in general if we were adding some arbitrary key/value pairs to the structured log message, will the agent automatically parse it? I can't seem to find any documentation for that

[laurit]

Agent does not parse the log message. If your ask is that original message includes attributes like loggerName and threadName that are not present in the logging-otlp output then the reason is that by default agent only captures the attributes described in otel specification. As this part is not specified capturing such attributes needs to be explicitly enabled. https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/logback/logback-appender-1.0/javaagent and https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/log4j/log4j-appender-2.17/javaagent show the supported flags for log4j/logback, currently we don't have similar flags for jboss logging.

[breedx-splk]

I suspect this is resolved, so I am closing. Please feel free to reopen if necessary.