Metrics tags from tracing could be more helpful

[SpyMachine]

First of all, I'm not totally sure if this is the right place to ask this question...I've spent a few hours going down the rabbit hole trying to figure out how metrics from tracing are created but have come up empty.

I've added the net::http integration for ddtrace and this has resulted in very helpful tracing with appropriate tagging. That part is great!

What I'm confused by however are the metrics created from the tracing. When I go and check metrics, I get random "resource" tags and "resource_name" tags that are created from the method of the request.

This makes creating dashboards and monitors from these metrics very difficult because I don't really know what path is being referred to here. Can the resource be updated to be a path so that it is easier to filter the metrics? Or can we add more tags? Is this even done at the client level or is this all handled on the backend?

[delner]

Hey @SpyMachine, I think part of your question is relevant here.

Each span in a trace has a "span name", the name of the operation being done, and the "resource" which is the domain name of what's being operated upon.

The Net/HTTP integration out-of-the-box has a span name of "http.request", and defines its resource as the HTTP method: this is why you're seeing the verb show up as resource name most likely. "Resource name" is just the verb by default because URLs/paths can contain sensitive information; something we don't want the integration to automatically collect.

We do have a workaround in place you can use, to explicitly handle a callback and override the resource name to a path: see #277 (comment) for details. This spec also shows how the workaround is used in practice:

dd-trace-rb/spec/ddtrace/contrib/http/request_spec.rb

Line 93 in 2515b5d

context 'when configured with #after_request hook' do

You can also use this to decorate the span with additional tags with any content you wish.

(Keep in mind, this is deliberately not documented because it is just considered a workaround, not a part of the "official" API. It might be subject to change/removal without deprecation in the future when we have a better solution.)

The part of your question that is not relevant here are the metrics themselves: ddtrace for Ruby does not generate metrics, it sends traces to the trace agent which then auto-generates metrics on its behalf. The tag generation is largely determined by what spans are present, their tags, and other attributes. Beyond that, I unfortunately don't have any other answers regarding metrics.

Hopefully the first part is useful enough to tweak your implementation to a good setup. Let me know if you have any other questions about that!

[SpyMachine]

Hi @delner thanks. So yeah, I'm specifically interested in changing the tags on the metrics, not the span itself. I played around a bit with the workaround and while I did see the span's tags being updated, it did not update the metrics tag. So barring any way to manipulate the metrics tags that are created, I think I'm stuck.

This would be really helpful from my point of view because it would allow the client integrations here be for both tracing and obvious metrics like HTTP request timing.

In any event, if that's all that can be done here, feel free to close. Thanks for your time.

[delner]

Yeah unfortunately we can't do anything about metric tags for metrics generated in the agent here. I'll forward this scenario to the team for their consideration. Going to close for now, but feel free to reach out again if you have any more questions about the Ruby tracer.