Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[connector/otlpjson] Add connector's implementations #34249

Merged

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Jul 25, 2024

Description:

This is the 2nd PR for the new otlpjson connector (#34208). Adds the ConsumeLogs methods for the logs, metrics and traces respective connector types.

At the moment, the connector retrieves the otlp json from the incoming Log's Body. This is not configurable yet. It can be decided if that needs to be.

In addition, there is no attribute/metadata extraction/transfer from the incoming Log. This can be added in future iterations.

Link to tracking Issue: #34208 #34239

Testing: Added.

Documentation: ~

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark force-pushed the implement_oltpjson_connector branch from 6afe625 to 1b525bc Compare July 25, 2024 11:02
@ChrsMark ChrsMark marked this pull request as ready for review July 25, 2024 11:55
@ChrsMark ChrsMark requested a review from a team July 25, 2024 11:55
@ChrsMark
Copy link
Member Author

How to test this manually

  1. Use the folllowing builder config:
dist:
  name: otelcol-dev-bin
  description: Basic OpenTelemetry collector distribution for Developers
  output_path: ./otelcol-dev
  otelcol_version: 0.105.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.105.0

receivers:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.105.0

connectors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/otlpjsonconnector v0.105.0

replaces:
  - github.com/open-telemetry/opentelemetry-collector-contrib/connector/otlpjsonconnector => ~/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/connector/otlpjsonconnector
  1. Use the following Collector's config:
receivers:
  filelog:
    include:
      - /var/log/pods/prod_my-target-pod_49cc7c1fd3702c40b2686ea7486091d3/my-target-pod/1.log

exporters:
  debug:
    verbosity: detailed

connectors:
  otlpjson:

service:
  pipelines:
    logs/raw:
      receivers: [filelog]
      exporters: [otlpjson]
    metrics/otlp:
      receivers: [ otlpjson ]
      exporters: [ debug ]
    logs/otlp:
      receivers: [ otlpjson ]
      exporters: [ debug ]
    traces/otlp:
      receivers: [ otlpjson ]
      exporters: [ debug ]
  1. Run the Collector: ./otelcol-dev/otelcol-dev-bin --config=config.yaml
  2. Write some lines to the log file:
> echo '{"resourceSpans":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeSpans":[{"scope":{},"spans":[{"traceId":"","spanId":"","parentSpanId":"","name":"operationA","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","droppedAttributesCount":1,"events":[{"timeUnixNano":"1581452773000000123","name":"event-with-attr","attributes":[{"key":"span-event-attr","value":{"stringValue":"span-event-attr-val"}}],"droppedAttributesCount":2},{"timeUnixNano":"1581452773000000123","name":"event","droppedAttributesCount":2}],"droppedEventsCount":1,"status":{"message":"status-cancelled","code":2}},{"traceId":"","spanId":"","parentSpanId":"","name":"operationB","startTimeUnixNano":"1581452772000000321","endTimeUnixNano":"1581452773000000789","links":[{"traceId":"","spanId":"","attributes":[{"key":"span-link-attr","value":{"stringValue":"span-link-attr-val"}}],"droppedAttributesCount":4},{"traceId":"","spanId":"","droppedAttributesCount":1}],"droppedLinksCount":3,"status":{}}]}]}]}' >> /var/log/pods/prod_my-target-pod_49cc7c1fd3702c40b2686ea7486091d3/my-target-pod/1.log
> echo '{"resourceLogs":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1581452773000000789","severityNumber":9,"severityText":"Info","body":{"stringValue":"This is a log message"},"attributes":[{"key":"app","value":{"stringValue":"server"}},{"key":"instance_num","value":{"intValue":"1"}}],"droppedAttributesCount":1,"traceId":"08040201000000000000000000000000","spanId":"0102040800000000"},{"timeUnixNano":"1581452773000000789","severityNumber":9,"severityText":"Info","body":{"stringValue":"something happened"},"attributes":[{"key":"customer","value":{"stringValue":"acme"}},{"key":"env","value":{"stringValue":"dev"}}],"droppedAttributesCount":1,"traceId":"","spanId":""}]}]}]}' >> /var/log/pods/prod_my-target-pod_49cc7c1fd3702c40b2686ea7486091d3/my-target-pod/1.log
> echo '{"resourceMetrics":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeMetrics":[{"scope":{},"metrics":[{"name":"counter-int","unit":"1","sum":{"dataPoints":[{"attributes":[{"key":"label-1","value":{"stringValue":"label-value-1"}}],"startTimeUnixNano":"1581452773000000789","timeUnixNano":"1581452773000000789","asInt":"123"},{"attributes":[{"key":"label-2","value":{"stringValue":"label-value-2"}}],"startTimeUnixNano":"1581452772000000321","timeUnixNano":"1581452773000000789","asInt":"456"}],"aggregationTemporality":2,"isMonotonic":true}},{"name":"counter-int","unit":"1","sum":{"dataPoints":[{"attributes":[{"key":"label-1","value":{"stringValue":"label-value-1"}}],"startTimeUnixNano":"1581452772000000321","timeUnixNano":"1581452773000000789","asInt":"123"},{"attributes":[{"key":"label-2","value":{"stringValue":"label-value-2"}}],"startTimeUnixNano":"1581452772000000321","timeUnixNano":"1581452773000000789","asInt":"456"}],"aggregationTemporality":2,"isMonotonic":true}}]}]}]}' >> /var/log/pods/prod_my-target-pod_49cc7c1fd3702c40b2686ea7486091d3/my-target-pod/1.log
  1. Verify in the console that otlp logs, metrics, traces are properly extracted.

@djaglowski djaglowski merged commit 3cb736b into open-telemetry:main Jul 25, 2024
156 checks passed
@github-actions github-actions bot added this to the next release milestone Jul 25, 2024
@ChrsMark ChrsMark deleted the implement_oltpjson_connector branch July 25, 2024 14:05
djaglowski pushed a commit that referenced this pull request Jul 26, 2024
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This is the 3rd PR for the new otlpjson connector
(#34208).
Moves its stability to `alpha`.

**Link to tracking Issue:** <Issue number if applicable>
#34208
#34239
#34249

**Testing:** <Describe what testing was performed and which tests were
added.> ~

**Documentation:** <Describe the documentation added.> Updated

---------

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants