Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sampling by trace tags #3587

Merged
merged 1 commit into from
Apr 15, 2024
Merged

Add sampling by trace tags #3587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/GettingStarted.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2132,7 +2132,7 @@ For example, if `tracing.sampling.default_rate` is configured by [Remote Configu
| `tracing.sampler` | | `nil` | Advanced usage only. Sets a custom `Datadog::Tracing::Sampling::Sampler` instance. If provided, the tracer will use this sampler to determine sampling behavior. See [Application-side sampling](#application-side-sampling) for details. |
| `tracing.sampling.default_rate` | `DD_TRACE_SAMPLE_RATE` | `nil` | Sets the trace sampling rate between `0.0` (0%) and `1.0` (100%). See [Application-side sampling](#application-side-sampling) for details. |
| `tracing.sampling.rate_limit` | `DD_TRACE_RATE_LIMIT` | `100` (per second) | Sets a maximum number of traces per second to sample. Set a rate limit to avoid the ingestion volume overages in the case of traffic spikes. |
| `tracing.sampling.rules` | `DD_TRACE_SAMPLING_RULES` | `nil` | Sets trace-level sampling rules, matching against the local root span. The format is a `String` with JSON, containing an Array of Objects. Each Object must have a float attribute `sample_rate` (between 0.0 and 1.0, inclusive), and optionally `name`, `service`, and `resource` string attributes. `name`, `service`, and `resource` control to which traces this sampling rule applies; if they are all absent, then this rule applies to all traces. Rules are evaluted in order of declartion in the array; only the first to match is applied. If none apply, then `tracing.sampling.default_rate` is applied. |
| `tracing.sampling.rules` | `DD_TRACE_SAMPLING_RULES` | `nil` | Sets trace-level sampling rules, matching against the local root span. The format is a `String` with JSON, containing an Array of Objects. Each Object must have a float attribute `sample_rate` (between 0.0 and 1.0, inclusive), and optionally `name`, `service`, `resource`, and `tags` string attributes. `name`, `service`, `resource`, and `tags` control to which traces this sampling rule applies; if they are all absent, then this rule applies to all traces. Rules are evaluted in order of declartion in the array; only the first to match is applied. If none apply, then `tracing.sampling.default_rate` is applied. |
| `tracing.sampling.span_rules` | `DD_SPAN_SAMPLING_RULES`,`ENV_SPAN_SAMPLING_RULES_FILE` | `nil` | Sets [Single Span Sampling](#single-span-sampling) rules. These rules allow you to keep spans even when their respective traces are dropped. |
| `tracing.trace_id_128_bit_generation_enabled` | `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED` | `true` | `true` to generate 128 bits trace ID and `false` to generate 64 bits trace ID |
| `tracing.report_hostname` | `DD_TRACE_REPORT_HOSTNAME` | `false` | Adds hostname tag to traces. |
Expand Down
23 changes: 20 additions & 3 deletions lib/datadog/tracing/sampling/matcher.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,23 +29,40 @@ def ===(other)
end
end.new

attr_reader :name, :service, :resource
attr_reader :name, :service, :resource, :tags

# @param name [String,Regexp,Proc] Matcher for case equality (===) with the trace name,
# defaults to always match
# @param service [String,Regexp,Proc] Matcher for case equality (===) with the service name,
# defaults to always match
# @param resource [String,Regexp,Proc] Matcher for case equality (===) with the resource name,
# defaults to always match
def initialize(name: MATCH_ALL, service: MATCH_ALL, resource: MATCH_ALL)
def initialize(name: MATCH_ALL, service: MATCH_ALL, resource: MATCH_ALL, tags: {})
super()
@name = name
@service = service
@resource = resource
@tags = tags
end

def match?(trace)
name === trace.name && service === trace.service && resource === trace.resource
name === trace.name && service === trace.service && resource === trace.resource && tags_match?(trace)
end

private

# Match against the trace tags and metrics.
def tags_match?(trace)
@tags.all? do |name, matcher|
tag = trace.get_tag(name)

# Format metrics as strings, to allow for partial number matching (/4.*/ matching '400', '404', etc.).
# Because metrics are floats, we use the '%g' format specifier to avoid trailing zeros, which
# can affect exact string matching (e.g. '400' matching '400.0').
tag = format('%g', tag) if tag.is_a?(Numeric)

matcher === tag
end
end
end

Expand Down
4 changes: 2 additions & 2 deletions lib/datadog/tracing/sampling/rule.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ class SimpleRule < Rule
# @param sample_rate [Float] Sampling rate between +[0,1]+
def initialize(
name: SimpleMatcher::MATCH_ALL, service: SimpleMatcher::MATCH_ALL,
resource: SimpleMatcher::MATCH_ALL, sample_rate: 1.0
resource: SimpleMatcher::MATCH_ALL, tags: {}, sample_rate: 1.0
)
# We want to allow 0.0 to drop all traces, but {Datadog::Tracing::Sampling::RateSampler}
# considers 0.0 an invalid rate and falls back to 100% sampling.
Expand All @@ -69,7 +69,7 @@ def initialize(
sampler = RateSampler.new
sampler.sample_rate = sample_rate

super(SimpleMatcher.new(name: name, service: service, resource: resource), sampler)
super(SimpleMatcher.new(name: name, service: service, resource: resource, tags: tags), sampler)
end
end
end
Expand Down
1 change: 1 addition & 0 deletions lib/datadog/tracing/sampling/rule_sampler.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ def self.parse(rules, rate_limit, default_sample_rate)
name: rule['name'],
service: rule['service'],
resource: rule['resource'],
tags: rule['tags'],
sample_rate: sample_rate,
}

Expand Down
85 changes: 83 additions & 2 deletions spec/datadog/tracing/sampling/matcher_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,17 @@

RSpec.describe Datadog::Tracing::Sampling::SimpleMatcher do
let(:trace_op) do
Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
Datadog::Tracing::TraceOperation.new(
name: trace_name,
service: trace_service,
resource: trace_resource,
tags: trace_tags
)
end
let(:trace_name) { 'operation.name' }
let(:trace_service) { 'test-service' }
let(:trace_resource) { 'test-resource' }
let(:trace_tags) { {} }

describe '#match?' do
subject(:match?) { rule.match?(trace_op) }
Expand Down Expand Up @@ -108,6 +114,81 @@
end
end

context 'with a tags matcher' do
let(:rule) { described_class.new(tags: tags) }

context 'when span tags are present' do
let(:trace_tags) { { 'tag1' => 'value1', 'tag2' => 'value2' } }

context 'with a regexp' do
context 'matching' do
let(:tags) { { 'tag1' => /value.*/, 'tag2' => /.*/ } }

it { is_expected.to eq(true) }
end

context 'not matching' do
let(:tags) { { 'tag1' => /value.*/, 'tag2' => /not_value/ } }

it { is_expected.to eq(false) }
end
end

context 'with a string' do
context 'matching' do
let(:tags) { trace_tags }

it { is_expected.to eq(true) }
end

context 'not matching' do
let(:tags) { { 'tag1' => 'value1', 'tag2' => 'not_value' } }

it { is_expected.to eq(false) }
end
end
end

context 'when span metrics are present' do
# Metrics are stored as tags, but have numeric values
let(:trace_tags) { { 'metric1' => 1.0, 'metric2' => 2 } }

context 'with a regexp' do
context 'matching' do
let(:tags) { { 'metric1' => /1/, 'metric2' => /.*/ } }

it { is_expected.to eq(true) }
end

context 'not matching' do
let(:tags) { { 'metric1' => /1/, 'metric2' => 3 } }

it { is_expected.to eq(false) }
end
end

context 'with a string' do
context 'matching' do
let(:tags) { { 'metric1' => '1', 'metric2' => '2' } }

it { is_expected.to eq(true) }
end

context 'not matching' do
let(:tags) { { 'metric1' => '1', 'metric2' => 'not_value' } }

it { is_expected.to eq(false) }
end
end
end

context 'when span tags are not present' do
let(:tags) { { 'tag1' => 'value1', 'tag2' => 'value2' } }

it { is_expected.to eq(false) }
end
end

context 'when trace service is not present' do
let(:trace_service) { nil }
let(:service) { /.*/ }
Expand Down Expand Up @@ -196,7 +277,7 @@
end

RSpec.describe Datadog::Tracing::Sampling::ProcMatcher do
let(:trace_op) { Datadog::Tracing::SpanOperation.new(trace_name, service: trace_service) }
let(:trace_op) { Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service) }
let(:trace_name) { 'operation.name' }
let(:trace_service) { nil }

Expand Down
9 changes: 9 additions & 0 deletions spec/datadog/tracing/sampling/rule_sampler_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,15 @@
end
end

context 'and tags' do
let(:rule) { { sample_rate: 0.1, tags: { tag: 'test-tag' } } }

it 'parses matching tag' do
expect(actual_rule.matcher.tags).to eq({ 'tag' => 'test-tag' })
expect(actual_rule.sampler.sample_rate).to eq(0.1)
end
end

context 'with multiple rules' do
let(:rules) { [{ sample_rate: 0.1 }, { sample_rate: 0.2 }] }

Expand Down
22 changes: 19 additions & 3 deletions spec/datadog/tracing/sampling/rule_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,17 @@

RSpec.describe Datadog::Tracing::Sampling::Rule do
let(:trace_op) do
Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
Datadog::Tracing::TraceOperation.new(
name: trace_name,
service: trace_service,
resource: trace_resource,
tags: trace_tags
)
end
let(:trace_name) { 'operation.name' }
let(:trace_service) { 'test-service' }
let(:trace_resource) { 'test-resource' }
let(:trace_tags) { {} }

let(:rule) { described_class.new(matcher, sampler) }
let(:matcher) { instance_double(Datadog::Tracing::Sampling::Matcher) }
Expand Down Expand Up @@ -79,24 +85,34 @@

RSpec.describe Datadog::Tracing::Sampling::SimpleRule do
let(:trace_op) do
Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
Datadog::Tracing::TraceOperation.new(
name: trace_name,
service: trace_service,
resource: trace_resource,
tags: trace_tags
)
end
let(:trace_name) { 'operation.name' }
let(:trace_service) { 'test-service' }
let(:trace_resource) { 'test-resource' }
let(:trace_tags) { {} }

describe '#initialize' do
subject(:rule) { described_class.new(name: name, service: service, resource: resource, sample_rate: sample_rate) }
subject(:rule) do
described_class.new(name: name, service: service, resource: resource, sample_rate: sample_rate, tags: tags)
end

let(:name) { double('name') }
let(:service) { double('service') }
let(:resource) { double('resource') }
let(:tags) { { 'tag' => 'value' } }
let(:sample_rate) { 0.123 }

it 'initializes with the correct values' do
expect(rule.matcher.name).to eq(name)
expect(rule.matcher.service).to eq(service)
expect(rule.matcher.resource).to eq(resource)
expect(rule.matcher.tags).to eq(tags)
expect(rule.sampler.sample_rate).to eq(sample_rate)
end
end
Expand Down
Loading