[exporters/otlphttpexporter] Support for json_log_encoding extension

[mishranant]

Component(s)

extension/encoding/jsonlogencoding

Is your feature request related to a problem? Please describe.

My use-case is to get data from the Kubernetes API, the data I am receiving on http endpoint is like so:

  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "logRecords": [
            {
              "attributes": [
                {
                  "key": "k8s.resource.name",
                  "value": {
                    "stringValue": "nodes"
                  }
                }
              ],
              "body": {
                "kvlistValue": {
                  "values": [
                    {
                      "key": "spec",
                      "value": {
                        "kvlistValue": {
                          "values": [
                            {
                              "key": "podCIDR",
                              "value": {
                                "stringValue": "10.244.0.0/24"
                              }
                            },
                            {
                              "key": "podCIDRs",
                              "value": {
                                "arrayValue": {
                                  "values": [
                                    {
                                      "stringValue": "10.244.0.0/24"
                                    }
                                  ]
                                }
                              }
                            },
                            {
                              "key": "providerID",
                              "value": {
                                "stringValue": "kind://docker/agent/agent-control-plane"
                              }
                            }
                          ]
                        }
                      }
                    },
                      "key": "kind",
                      "value": {
                        "stringValue": "Node"
                      }
                    },
                    {
                      "key": "apiVersion",
                      "value": {
                        "stringValue": "v1"
                      }
                    },
                  ]
                }
              },
              "observedTimeUnixNano": "1730192641520626677",
              "spanId": "",
              "traceId": ""
            }
          ],
          "scope": {}
        }
      ]
    }
  ]
}

I want to get the raw event object JSON data instead of OTLP JSON data in the log body. Referring to this issue, I added json_log_extension in my config_map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otelcontribcol
  labels:
    app: otelcontribcol
data:
  config.yaml: |
    receivers:
      k8sobjects:
        objects:
        - name: events
          mode: watch
        - name: pods
          mode: pull
          interval: 15m
        - name: configmaps
          mode: pull
          interval: 15m
        - name: cronjobs
          mode: pull
          interval: 15m
        - name: daemonsets
          mode: pull
          interval: 15m
        - name: deployments
          mode: pull
          interval: 15m
        - name: ingresses
          mode: pull
          interval: 15m
        - name: jobs
          mode: pull
          interval: 15m
        - name: namespaces
          mode: pull
          interval: 15m
        - name: nodes
          mode: pull
          interval: 15m
        - name: persistentvolumes
          mode: pull
          interval: 15m
        - name: persistentvolumeclaims
          mode: pull
          interval: 15m
        - name: replicasets
          mode: pull
          interval: 15m
        - name: serviceaccounts
          mode: pull
          interval: 15m
        - name: services
          mode: pull
          interval: 15m
        - name: statefulsets
          mode: pull
          interval: 15m

    exporters:
      otlphttp:
        endpoint: "http://echo-server:8000/client/kube_agent"
        encoding: json
        compression: none
        headers:
          Content-Type: "application/json"

    extensions:
      json_log_encoding:

    service:
      extensions: [json_log_encoding]
      pipelines:
        logs:
          receivers: [k8sobjects]
          exporters: [otlphttp]

But that did not help at all. What could be the issue?

Describe the solution you'd like

If support for json_log_encoding is missing, would request it to be added.

Describe alternatives you've considered

No response

Additional context

No response

[mishranant]

/label extension/encoding/jsonlogencoding

[mishranant]

/label extension/encoding

[github-actions]

Pinging code owners for extension/encoding: @atoulme @dao-jun @dmitryax @MovieStoreGuy @VihasMakwana. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mishranant]

@Frapschen do you have any ideas?

[VihasMakwana]

Did you face any errors?

[mishranant]

@VihasMakwana The extension starts fine, don't see any errors even later:

2024-11-05T11:01:54.710Z	info	extensions/extensions.go:39	Starting extensions...
2024-11-05T11:01:54.710Z	info	extensions/extensions.go:42	Extension is starting...	{"kind": "extension", "name": "json_log_encoding"}
2024-11-05T11:01:54.710Z	info	extensions/extensions.go:59	Extension started.	{"kind": "extension", "name": "json_log_encoding"}
2024-11-05T11:01:54.710Z	info	k8sobjectsreceiver@v0.112.0/receiver.go:74	Object Receiver started	{"kind": "receiver", "name": "k8sobjects", "data_type": "logs"}

but the result is like it is unaffected whether the extension is enabled or not.

[Frapschen]

@mishranant I opted for the file exporter instead and used Fluent Bit to scrape the data and forward it to the backend.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• extension/encoding: @atoulme @dao-jun @dmitryax @MovieStoreGuy @VihasMakwana

See Adding Labels via Comments if you do not have permissions to add labels yourself.