Fix OpenTelemetry span processing when using logstasher #372
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fa9820b to
e4aaeca
Compare
e4aaeca to
f43f543
Compare
theseanything
commented
Jun 4, 2024
ChrisBAshton
reviewed
Jun 4, 2024
lib/govuk_app_config/govuk_json_logging/rails_ext/action_controller/metal/instrumentation.rb
Show resolved
Hide resolved
lib/govuk_app_config/govuk_json_logging/rails_ext/action_controller/metal/instrumentation.rb
Show resolved
Hide resolved
The logstasher gem monkey patches the `process_action.action_controller` instrumentation. However the gem hasn't been updated for Rails 7, which assumes that the payload has the 'request' object available. This request object is used by the OpenTelemetry SDK to process spans. This generates lots of error logs from the SDK, and prevents us from recording some spans. This re-monkey patches the instrumentation to use a raw_payload that is expected in Rails 7. Ideally, this would be fixed in the logstasher gem however there has been little active development over the last year. Longer term fix might be to move to a maintained logging library.
f43f543 to
ae4603e
Compare
ChrisBAshton
approved these changes
Jun 4, 2024
richardTowers
approved these changes
Jun 4, 2024
theseanything
added a commit
to alphagov/govuk-helm-charts
that referenced
this pull request
Jun 5, 2024
This was previously disable due to a bug causing the OpenTel SDK to produce significant numbers of error logs. This bug has been fixed in alphagov/govuk_app_config#372.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The logstasher gem monkey patches the
process_action.action_controllerinstrumentation. However the gem hasn't been updated for Rails 7, which assumes that the payload has the 'request' object available. This request object is used by the OpenTelemetry SDK to process spans. This generates lots of error logs from the SDK, and prevents us from recording some spans.This re-monkey patches the instrumentation to use a raw_payload that is expected in Rails 7. Ideally, this would be fixed in the logstasher gem however there has been little active development over the last year. Longer term fix might be to move to a maintained logging library.