Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract context clearing logic for ActiveSupport events #897

Closed
wants to merge 2 commits into from

Conversation

marcotc
Copy link
Member

@marcotc marcotc commented Dec 20, 2019

When gems that utilize thread pools are instrumented it is very important to ensure that the thread-bound Context does not leak information between executions.

We have existing logic to clear the Context for a few integrations that instrument using ActiveSupport events.

This PR extracts that logic to a central place, adds a health metric to count how many times the context is leaking and cleans up testing for integrations affected.

@marcotc marcotc added the integrations Involves tracing integrations label Dec 20, 2019
@marcotc marcotc requested a review from a team December 20, 2019 22:10
@marcotc marcotc self-assigned this Dec 20, 2019
@marcotc marcotc changed the base branch from feat/action-cable-integration to master January 6, 2020 20:35
@marcotc
Copy link
Member Author

marcotc commented Jan 6, 2020

Now that #824 has been merged, I'm rebasing this on master instead.

@marcotc marcotc force-pushed the feat/metric-leaky-thread-context branch from d7d5da1 to cf0adf1 Compare January 6, 2020 20:38
@marcotc marcotc added the dev/refactor Involves refactoring existing components label Jan 6, 2020
@marcotc
Copy link
Member Author

marcotc commented Jan 6, 2020

Rebased, ready for review again.

Copy link
Contributor

@delner delner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Standardizing this is 👍 from me. I have a couple of suggestions and some additional information I'd like though.

unfinished_span = configuration[:tracer].call_context.current_span
return unless unfinished_span

Diagnostics::Health.metrics.error_unfinished_context(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know how often this occurs roughly? It's not clear to me whether resetting the context is normal operating procedure, or an error as its been identified here.

I think it'd be best to avoid emitting this as an error if this is normal; we should only submit things as errors if they are things that should be preventable or fixable.

private

# Clears context if there are unfinished spans in it
def ensure_clean_context!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of extracting this function out and standardizing this here.

Does it make a sense to go a step further and extract this function to a module available to all integrations? Other integrations do the same, such as Rack and SuckerPunch, where they reset the context as well. If we're standardizing here, I don't think its a stretch to standardize everywhere, if plausible.

#
# This module *cannot* be used for events can be nested, as
# it drops all spans currently active in the {Context}.
module RootEvent
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add unit tests for this module specifically? I'd like to see some assurance that if you use this module, it will prevent contexts from leaking.

I saw there's a test in Racecar, but I'd prefer something more explicit and directly related to this module, as its the component that owns this behavior.

@marcotc marcotc added the do-not-merge/WIP Not ready for merge label Mar 31, 2020
@marcotc marcotc closed this Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev/refactor Involves refactoring existing components do-not-merge/WIP Not ready for merge integrations Involves tracing integrations
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants