OpenTelemetry plugin not sending logs to collector

[DMaxter]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.9.0

Current Behavior

When setting the OpenTelemetry plugin and setting the logs endpoint, it sends a request to the collector (sometimes 2), but after that nothing else is sent (even after waiting some minutes).

I noticed that if the log level is debug the requests are then sent.

I compiled kong from source and played around a bit, and the behavior is super strange:

If I add

ngx.log(ngx.ERR, "TESTING OTEL")

in either of these functions

kong/kong/plugins/opentelemetry/logs.lua

Line 22 in 0118b3c

local function http_export_logs(conf, logs_batch)

kong/kong/plugins/opentelemetry/utils.lua

Line 16 in 0118b3c

local function http_export_request(conf, pb_data, headers)

in the path of success, I get both my log message and the request on OpenTelemetry collector.

To check that this was indeed a problem on Kong side, I even checked with tcpdump and there seems to be no traffic at all. Connection is established and around 1min later we have the connection being closed and nothing more.

This seems to be either some buffer that needs to be flushed or the worker thread that is supposed to make the request not running. If I used ngx.DEBUG instead of ngx.ERR, I would have the same behavior as if the statement wasn't there, because default log level is higher (info)

Expected Behavior

Logs sent to OpenTelemetry collector after a timeout (which should be some seconds) or right after making them if timeout is too low.

Steps To Reproduce

1. Deploy an OpenTelemetry Collector with the following configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  debug/basic:
    verbosity: basic

service:
  pipelines:
    logs:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - debug/basic

  telemetry:
    logs:
      level: info

2. Deploy kong with a minimal configuration

_format_version: "3.0"
plugins:
- config:
    connect_timeout: 1000
    header_type: ot
    logs_endpoint: http://localhost:4318/v1/logs
    propagation:
      clear: null
      default_format: ot
      extract: null
      inject:
      - preserve
    queue:
      concurrency_limit: -1
      initial_retry_delay: 0.01
      max_batch_size: 1
      max_bytes: null
      max_coalescing_delay: 1
      max_entries: 10000
      max_retry_delay: 30
      max_retry_time: 30
    read_timeout: 60000
    sampling_rate: 0
    send_timeout: 60000
  enabled: true
  name: opentelemetry
  protocols:
  - grpc
  - grpcs
  - http
  - https
routes:
- https_redirect_status_code: 426
  name: otel
  path_handling: v0
  paths:
  - /
  plugins:
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
      - apikey
      realm: null
      run_on_preflight: true
    enabled: true
    name: key-auth
    protocols:
    - grpc
    - grpcs
    - http
    - https
  preserve_host: false
  protocols:
  - http
  - https
  regex_priority: 0
  request_buffering: true
  response_buffering: true
  strip_path: true

3. Perform some requests to the gateway
4. Should only see 2 lines similar to

2024-12-31T15:15:50.346Z        info    Logs    {"kind": "exporter", "data_type": "logs", "name": "debug/basic", "resource logs": 1, "log records": 1}
2024-12-31T15:15:59.377Z        info    Logs    {"kind": "exporter", "data_type": "logs", "name": "debug/basic", "resource logs": 1, "log records": 1}

Anything else?

No response

[samugi]

Hi @DMaxter, thank you for opening this issue.

Is any log produced in your Kong error_log output, which you are expecting to be reported? I tested the configuration that you provided, and with that, no log is produced for incoming requests, in which case the behavior of no logs being reported would be expected, as described in our docs:

Logs are reported based on the log level that is configured for Kong Gateway. If a log is emitted with a level that is lower than the configured log level, it is not exported.

Note: Not all logs are guaranteed to be exported. Logs that are not exported include those produced by the Nginx master process and low-level errors produced by Nginx. Operators are expected to capture the Nginx error.log file in addition to using this feature for observability purposes.

If understood correctly, the moment you added a log with a level that is compatible to your logging configuration, i.e. higher than INFO: ngx.log(ngx.ERR, "TESTING OTEL"), you started receiving logs on your collector as expected.

The two log entries you mention in your description are likely the following:

worker #0 is ready to accept events from unix:/usr/local/kong/sockets/we
worker #1 is ready to accept events from unix:/usr/local/kong/sockets/we

Which are expected to only be outputed once for each worker.

To reiterate, you should only expect logs that exist in the error_log to be reported in OTel format via the configured logs_endpoint.
Let me know in case you think I missed anything or if you have any doubts.

[DMaxter]

Hello

The logs I'm expecting to be exported are in the access.log file. According to the documentation those should be exported, or am I missing something. From the quote you posted, logs in error.log aren't guaranteed to be exported that's why we need operators

[samugi]

Hi @DMaxter , at this time only the error log output is reported in OpenTelemetry format via the OpenTelemetry plugin when this feature is enabled. In addition to that, any entry that is generated in custom plugins, using the kong.telemetry.log pdk function is also reported: you may want to use that if you intend to report additional/custom entries as OTel-formatted logs.

According to the documentation those should be exported

Could you point me to that information? It's possible that this is incorrectly documented, in which case I'll make sure it is addressed.

[DMaxter]

Okay, I was reading it again and it doesn't say they should be exported, my bad. But I find this information very unclear.

Two different kinds of logs are exported: Request and Non-Request scoped.

• Request logs are directly associated with requests. They are produced during the request lifecycle. For example, this could be logs generated during a plugin’s Access phase.

Here it says that they are produced during the request lifecycle and that could be logs generated by plugins, not that it is only generated by them, which from the information you gave me, seems to be the case? Or are there still other logs associated with requests that are sent?

Note: Not all logs are guaranteed to be exported. Logs that are not exported include those produced by the Nginx master process and low-level errors produced by Nginx. Operators are expected to capture the Nginx error.log file in addition to using this feature for observability purposes.

Here the confusing part is that operators should capture the error.log, which led me to believe this file was not taken into account for logging, but rather the access.log file

[samugi]

hey @DMaxter, thank you for clarifying.

Where the documentation mentions non-request scoped logs it practically refers to logs that are produced asynchronously, for example during timers, that don't necessarily belong to one specific request in particular.

request-scoped logs, on the other hand, are all logs that are produced during a certain, specific, request's lifecycle, which include all logs generated by plugins (as the example says, during a certain phase of a plugin's execution), but also all the logs produced by core Kong code (internal errors / infos, etc), that are produced during said request.

Regarding the last paragraph, it is stating that not all (error) logs are guaranteed to always be captured, in fact, some low-level errors produced by Nginx and logs produced by the Nginx master process are not captured, for that reason, the error.log should still be captured and kept in case access to those specific logs is required for deeper troubleshooting.

I hope that helps clarifying the functionality of OTel formatted logs. Let me know in case of any doubt, thank you!