Logging support in OpenTelemetry

[ntyrewalla]

Provide logging support in ADOT (collector, EKS-add on, Lambda layers, SDKs, etc.). Tracking the maturity of components via upstream status.

---

Update by @mhausenblas on 2023-01-19:

In terms of supporting the logs signal type in ADOT we're particularly interested in feedback on:

• What's your environment (EKS, ECS, Lambda, EC2, on-prem)?
• What kind of logs (application-level vs. system-level or both) would you like to handle?
• Anything that you can share in terms of logs formats mappable to OTLP do you need supported?
• Any requirements around throughput?

[pstephens]

Would be nice to have a logDriver option to integrate with ECS (Fargate or EC2) directly and eliminate the need for the fluentbit side car. Ideally the otel-collector should be a one stop shop for telemetry, metrics, and logging.

[mhausenblas]

Thanks for your feedback @pstephens and this is certainly on our long-term roadmap (2023).

[jyidiego]

Hi @mhausenblas I was curious to know the status of logging for ADOT. I was checking the OTEL status page for Logging and it seems like overall it is stable (with event api being experimental) Thanks in advance.

[kevinslin]

@jyidiego

from what I can tell - ADOT supports logging on the exporter side. on the receiver side, it currently lacks integration with native aws services (eg. EKS-add on, Lambda layers, etc)

[mhausenblas]

ADOT at this point does not support logs in GA. The fact that there are certain components in the collector that support the logs signal type doesn't change the fact. We plan logs support by end of year but until then please be aware that logs are not GA and hence not supported.

[dluongiop]

Is there a possibility that ADOT can include this exporter or similar feature from collector-contrib? We currently use the contrib branch because ADOT doesn't have a way to export logs to cloudwatch otherwise.
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awscloudwatchlogsexporter

[mhausenblas]

@dluongiop good news: we're about to add it. Give it a few more days and you can use it, first behind a feature gate and the plan is by end of year to make it generally available.

[dluongiop]

Excellent! Thank you.

[mhausenblas]

Update: with today's v0.34.0 release you can now use the Filelog receiver and CloudWatch Logs exporter behind a feature gate. Please test and let us know!

[dluongiop]

@mhausenblas Thank you for the update. I am about to try it out. Is there anything I must configure to be able to use the Cloudwatch Log exporter that's behind the feature gate?

[dluongiop]

@mhausenblas I've figured out to use the feature gate on the command line.

Note for other people: pass "--feature-gates=adot.exporter.awscloudwatchlogs" argument to the collector.

[dluongiop]

The awscloudwatchlogs is working well so far for my needs. The current version seems to be correlating the awsxray traces with logs nicely without any additional tweaks. When I was using the collector-contrib with the awsxray exporter and awscloudwatchlogs exporter, it was not correlating the traces with logs unless I use a transform to write a new field in the log such as my example below:

{
    "body": "Blah blah",
    "severity_number": 5,
    "severity_text": "DEBUG",
    "flags": 1,
    "trace_id": "65206d6cb21cf3db383cc34fad3687fb",
    "span_id": "c81b4daa0c424437",
    "attributes": {
        "trace_id_xray": "1-65206d6c-b21cf3db383cc34fad3687fb"
    },
    "resource": {
        "service.name": "example.service",
        "telemetry.sdk.language": "cpp",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.9.1"
    }
}

[mhausenblas]

@dluongiop thanks a lot for your feedback and indeed the instructions on how to use it are in the accompanying launch blog post, my bad.

[mhausenblas]

After only 3 short years, here we go: https://aws.amazon.com/about-aws/whats-new/2023/11/logs-support-aws-distro-opentelemetry/

[ConProgramming]

Is Filelog the only receiver? Does ADOT have some API where I can send individual logs to over a web request?