Probabilistic sampler: emit metrics for number of spans sampled

[shinkle-procore]

Component(s)

processor/probabilisticsampler

Is your feature request related to a problem? Please describe.

We are trying to build SLOs (Service Level Objectives) around our telemetry pipeline. One of the indicators we use is the ratio of "spans received by receiver" to "spans sent by the exporter" expecting that ratio to stay close to 1 (otherwise it means we're dropping spans during processing).

Now that we have probabilistic sampling set up (at 10%), we're finding that ratio is around 0.1 instead of 1, which makes sense. However, just adding a factor of 10 to our SLO measurement to compensate is hacky. We would like to factor something like "otelcol_processor_probabilistic_sampler_count_spans_sampled" into our calculation, but right now we only have access to "otelcol_processor_probabilistic_sampler_count_traces_sampled"

The telemetry in the core collector library seems to revolve around "spans" while the sampler metrics revolve around "traces", so it's difficult to compare the two in calculations.

Describe the solution you'd like

Emit a metric of "otelcol_processor_probabilistic_sampler_count_spans_sampled" from the probabilistic sampler processor.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/probabilisticsampler: @jpkrohling @jmacd

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

@shinkle-procore, would you like to add it yourself? It should be relatively easy to add this.

[shinkle-procore]

@jpkrohling If I had more bandwidth and experience with the stack, yes :) unfortunately it's not something I can work on now.

[mahadzaryab1]

@jpkrohling I'm interested in working on this! Would you be able to guide me on what needs to be done?

[mahadzaryab1]

would the correct place to add this metric be at https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/tracesprocessor.go#L174?

[jpkrohling]

Close: I think this would be the right place: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/probabilisticsamplerprocessor/sampler_mode.go#L522

Instead of adding "1" to the new metric, it would add the count of items in the item T.

[amritanshu-pandey]

Any progress on this @mahadzaryab1? Otherwise I am also keen to contribute to this. Thanks!

[avereshchak]

Just a note, and I understand it's out of scope here.

I know both Application Insights SDK for .NET and Azure.Monitor.OpenTelemetry.Exporter are producing metrics such as total number of spans. They're being used later by "Application Map" and "Failures"/"Performance" blades to show correct statistics even if most of the spans were sampled out and only few ones being retained.

The exporter/azuremonitor would also benefit from converting these metrics into AzureMon-specific one to provide similar experience.