Akka http server span names are always akka.request

[mattypb]

Describe the bug
I've added the Open Telemetry java instrumentation to a Scala Akka service. Traces are created when the service's api's are hit, but no matter which api is hit they all have a span name of akka.request.

Looking at the Akka http server instrumentation, this is appears to be the current expected behaviour - AkkaHttpServerInstrumentationTest. However some other java instrumentations use the api path as the span name, for example Tomcat 7.0 - TomcatHandlerTest.

Why does the Akka server instrumentation have akka.request hard coded for the span name instead of using the api path? Will this change in the future?

What did you expect to see?
Hitting an api on my service, e.g. GET /v1/{user}, would generate a trace with span name /v1/{user}

What did you see instead?
Span name of akka.request

What version are you using?
1.3.0

Environment
Scala: 2.13.5
Akka-Http: 10.2.4
Compiler: AdoptOpenJDK 11.0.10
OS: Mac 10.15.7
Runtime (if different from JDK above): Oracle 11.0.11+9-LTS-194
OS (if different from OS compiled on): RHEL 7.1

[nibhuran]

@mattypb , @trask Can I take a crack at this issue.

[trask]

Can I take a crack at this issue.

sure!

[mcmho]

@trask, I am looking into this a bit. Does it make sense to add a SpanNameExtractor for this case and pass it to the Instrumenter?

thoughts? @mateuszrzeszutek

[trask]

@trask, I am looking into this a bit. Does it make sense to add a SpanNameExtractor for this case and pass it to the Instrumenter?

yes, that makes sense 👍

[mcmho]

@trask, @mateuszrzeszutek, I see that there are many span name extractors having null implementation for route().

  @Override
  @Nullable
  protected String route(Request request) {
    return null;
  }

I realized that it is hard to get the right span name with low cardinality with just the Request object.
Do you have any advice on that?

In our company's product, we provide configuration where customers can specify their own custom rule to name a span.
or we provide a default... say for example, the servlet or http case, we use the first two segments of the URI.
so an uri like this one: http://localhost:11001/foo/bar/baz?id=66
the span name would be /foo/bar

I am wondering if that would be sufficient. On the other hand, I understand that
the second segment can potentially be an id. in that case, we will end up with a span name that has high cardinality.

Does it make sense to parse the second segment skip it if it is a purely digit?

[trask]

hey @mcmho!

I realized that it is hard to get the right span name with low cardinality with just the Request object.
Do you have any advice on that?

if the mvc framework that we are instrumenting has a routing concept, then we use that (e.g. spring's @RequestMapping).

if the mvc framework has no routing concept then we don't capture any route, and the span name is the mostly unhelpful (but guaranteed low-cardinality) HTTP {method}.

It looks like akka http does have a route concept (https://doc.akka.io/docs/akka-http/current/routing-dsl/routes.html). if we can capture the route from akka that would be ideal

In our company's product, we provide configuration where customers can specify their own custom rule to name a span.
or we provide a default... say for example, the servlet or http case, we use the first two segments of the URI.
so an uri like this one: http://localhost:11001/foo/bar/baz?id=66
the span name would be /foo/bar

ya, this is a nice option. I suspect we may introduce something like this in the future (or possibly this could be supported as post-processing in the collector).

[mcmho]

Thanks @trask! I did look into the Route concept. As far as I understand, Route is the static configuration logic rather the runtime request route. It seems to be used for constructing a mapping to route request to the handling logic. i.e: https://developer.lightbend.com/guides/akka-http-quickstart-java/http-server.html
I can't find a way to get Route from HttpRequest.

My approach is to inspect the path in the request and come up with a span name with low cardinality.

[trask]

My approach is to inspect the path in the request and come up with a span name with low cardinality.

limiting the path to two segments doesn't guarantee low-cardinality, which is a requirement for span name

[mcmho]

@trask , I understand that the second segment can potentially be an id. in that case, we will end up with a span name that has high cardinality.

Does it make sense to parse the second segment skip it if it is a purely digit?

[trask]

I think that if we can't capture the real akka route that we should probably stick to capturing just HTTP {method} in the akka instrumentation itself

[mcmho]

Okay, I will do HTTP GET, HTTP POST, ... etc for now.

Wouldn't it be great if we can allow some sort of customization rules in the collector to post process span names,
such that users can name their span say /foo/10, /foo/11, as /foo
or /bar/111, /bar/123 as /bar.

[mcmho]

@trask, what about doing something like this:
/foo/10, /foo/11, become /foo/{id}
/bar/111, /bar/123 become /bar/{id}

Or do you just want to stick with HTTP {method}?

[trask]

@trask, what about doing something like this: /foo/10, /foo/11, become /foo/{id} /bar/111, /bar/123 become /bar/{id}

this is hard to do reliably, as not all parameterized path elements have to be numeric (e.g. guid, username, ...)

i think this would be a useful feature to build out separately as a span processor that can apply to all spans, and users can opt into it, and can customize it to their needs. we probably don't have a rich enough configuration model to support this in the agent yet, but in the meantime this could be added in contrib and configured programmatically similar to RuleBasedRoutingSampler

[mcmho]

Okay, sure @trask. We'll stick with HTTP {method} for this ticket.

And let me take a look at RuleBaseRoutingSampler and create a separate ticket for it.