Support for cross thread span tracing (e.g. via concurrent-ruby)

[bessey]

We have a few places in our application with multiple independent slow ActiveRecord queries that we use concurrent-ruby's Concurrent::Future class to parallelize, then block on the combined results. Here's some pseudo-code

scope = MyModel.slow_sql_query
# Start both queries in parallel
future_a_1 = Concurrent::Future.execute { scope.where(a: 1).to_a }
future_a_2 = Concurrent::Future.execute { scope.where(a: 2).to_a } 
# Blocks, ensuring we don't attempt to render before the queries are complete
@a_1 = future_a_1.value
@a_2 = future_a_2.value
render :show

To my surprise, the traces of the queries executed in the Future don't appear in DD. I'm assuming this has something to do with the trace ID being stored in Thread.current, or the Future threads in some other way being unaware of the trace ID.

What can I do to ensure these query spans are included in my trace?

[delner]

I think you're probably right: trace context is derived from thread local variable, meaning executions on other threads won't inherit trace context by default. There might be a way to relate these spans to the parent thread though, by setting the parent ID on the child thread traces to the parent thread's span ID.

Those child threads will need to know what the parent thread is, so it can derive the span ID from that thread's trace context. There might be a few ways to do this, but the first way I can think of is by adding a reference to the child thread's local variables when it's started by the parent thread. I'm not yet clear on the details of this, but something like that should work.

[bessey]

Great, yes, I figured the solution might look something like

span_id = Datadog.current_span_id
Concurrent::Future.execute { Datadog.with_span_id(span_id) { scope.where(a: 1).to_a } }

I just don't know what the real methods are

[delner]

If Concurrent::Future.execute works like you showed above, then maybe something like this will work:

# Gets current span from context in thread local variable
parent_span = Datadog.tracer.active_span

Concurrent::Future.execute do
  Datadog.tracer.trace('concurrent.future.execute', child_of: parent_span) do
    scope.where(a: 1).to_a
  end
end

[delner]

@bessey Did this work at all for you?

[jamiehodge]

I use something like this ATM:

def traced_thread
  context = Datadog.tracer.provider.context
  Thread.new {
    Datadog.tracer.provider.context = context
      yield
   }
end

[delner]

@bessey I hope the above snippet worked for you. I'm going to close this issue for now; please feel free to comment here to follow up if need be!

[SampsonCrowley]

@jamiehodge would safer not be:

Thread.new(Datadog.tracer.provider.context) {|context|
  Datadog.tracer.provider.context = context
  yield
}

[TSMMark]

@SampsonCrowley confirmed your solution works, thanks!

[pablobm]

This has changed in v1.0. See: https://github.com/DataDog/dd-trace-rb/blob/master/docs/UpgradeGuide.md#between-threads, where we can find the new idiom for this:

### New 1.0 ###
# Get trace digest
trace = Datadog::Tracing.active_trace

# NOTE: We must produce the digest BEFORE starting the thread.
#       Otherwise if it's lazily evaluated within the thread,
#       the thread's trace may follow the wrong parent span.
trace_digest = trace.to_digest

Thread.new do
  # Inherits trace properties from the trace digest
  Datadog::Tracing.trace('my.job', continue_from: trace_digest) do |span, trace|
    trace.id == trace_digest.trace_id
  end
end