Adds delay to Sidekiq metrics

[gottfrois]

Hey guys,

I've been trying to add a delay metric to Sidekiq which would represent the time spent for a job waiting in queue.
I've forked the gem and added the following code:

But I actually don't see those metrics in the metric explorer on Datadog nor the existing one like sidekiq.job.id or sidekiq.job.retry

So i'm not sure how/where those metrics are available. Can you help?
Thanks

[delner]

@gottfrois Really like the idea here! Tags work a little differently than metrics however; what you added above should appear as tags on each of the Sidekiq traces your application is producing, if you highlight the sidekiq.job span. I don't believe these values will be aggregated or displayed in the Metrics Explorer.

[gottfrois]

@delner oh ok i got this wrong!
So should delay be a tag or an actual metric?

[gottfrois]

@delner you were right, i see the new delay field in the span trace directly. Any idea how I can actually create a real metric out of it? Also, how would it integrate within the APM dashboards?

[delner]

@gottfrois I think most of our metrics are handled through our metrics gem https://github.com/DataDog/dogstatsd-ruby or https://github.com/DataDog/dogapi-rb.

As far as integrating that directly into the tracing portion of the code, off the top of my head, I might suggest using the Processing Pipeline to listen into these spans, then taking advantage of that to send off metrics using the metrics gem for each tag you'd like to measure.

Might look something like this in your datadog.rb initializer file or equivalent:

Datadog::Pipeline.before_flush(
  Datadog::Pipeline::SpanProcessor.new do |span|
    if span.name == 'sidekiq.job'
      # Send your metric here:
      # value = span.get_tag('sidekiq.job.delay').to_f
    end
  end
)

Give that a shot and let me know if that works for you!

[gottfrois]

@delner thanks a log for the code snippet. I do believe it works but i cannot confirm because i think i reached the limit of the number of custom metrics i can send on datadog :(

What's your input on custom metrics usage here? For example, the sidekiq trace uses a lot of custom metrics quotas:

Due to the number of tags and ressources i guess?

[gottfrois]

Nevertheless i think i could submit a PR here to add those new metrics:

• delay
• started_at
• finished_at
• enqueued_at

What do you think?

[delner]

Regarding custom metrics, admittedly, I'm not sure I can give you a good answer myself. I think the best I can offer right now is checking out our documentation on custom metrics, which hopefully has the details you need.

As for adding those datapoints to the Sidekiq tracing code, a PR that adds those basic values as tags (as opposed to metrics) sounds good to me.

[delner]

This was released as a part of 0.13.0. Thanks for your contribution!