Merge pull request #3587 from DataDog/tag-sampling

docs/GettingStarted.md

@@ -2132,7 +2132,7 @@ For example, if `tracing.sampling.default_rate` is configured by [Remote Configu
 | `tracing.sampler`                                       |                                                         | `nil`                        | Advanced usage only. Sets a custom `Datadog::Tracing::Sampling::Sampler` instance. If provided, the tracer will use this sampler to determine sampling behavior. See [Application-side sampling](#application-side-sampling) for details.                                                                                                                                                                                                                                                                                                                                                |
 | `tracing.sampling.default_rate`                         | `DD_TRACE_SAMPLE_RATE`                                  | `nil`                        | Sets the trace sampling rate between `0.0` (0%) and `1.0` (100%). See [Application-side sampling](#application-side-sampling) for details.                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | `tracing.sampling.rate_limit`                           | `DD_TRACE_RATE_LIMIT`                                   | `100` (per second)           | Sets a maximum number of traces per second to sample. Set a rate limit to avoid the ingestion volume overages in the case of traffic spikes.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
-| `tracing.sampling.rules`                                | `DD_TRACE_SAMPLING_RULES`                               | `nil`                        | Sets trace-level sampling rules, matching against the local root span. The format is a `String` with JSON, containing an Array of Objects. Each Object must have a float attribute `sample_rate` (between 0.0 and 1.0, inclusive), and optionally `name`, `service`, and `resource` string attributes. `name`, `service`, and `resource` control to which traces this sampling rule applies; if they are all absent, then this rule applies to all traces. Rules are evaluted in order of declartion in the array; only the first to match is applied. If none apply, then `tracing.sampling.default_rate` is applied. |
+| `tracing.sampling.rules`                                | `DD_TRACE_SAMPLING_RULES`                               | `nil`                        | Sets trace-level sampling rules, matching against the local root span. The format is a `String` with JSON, containing an Array of Objects. Each Object must have a float attribute `sample_rate` (between 0.0 and 1.0, inclusive), and optionally `name`, `service`, `resource`, and `tags` string attributes. `name`, `service`, `resource`, and `tags` control to which traces this sampling rule applies; if they are all absent, then this rule applies to all traces. Rules are evaluted in order of declartion in the array; only the first to match is applied. If none apply, then `tracing.sampling.default_rate` is applied. |
 | `tracing.sampling.span_rules`                           | `DD_SPAN_SAMPLING_RULES`,`ENV_SPAN_SAMPLING_RULES_FILE` | `nil`                        | Sets [Single Span Sampling](#single-span-sampling) rules. These rules allow you to keep spans even when their respective traces are dropped.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | `tracing.trace_id_128_bit_generation_enabled`           | `DD_TRACE_128_BIT_TRACEID_GENERATION_ENABLED`           | `true`                       | `true` to generate 128 bits trace ID and `false` to generate 64 bits trace ID                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
 | `tracing.report_hostname`                               | `DD_TRACE_REPORT_HOSTNAME`                              | `false`                      | Adds hostname tag to traces.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |

lib/datadog/tracing/sampling/matcher.rb

@@ -29,23 +29,40 @@ def ===(other)
           end
         end.new
 
-        attr_reader :name, :service, :resource
+        attr_reader :name, :service, :resource, :tags
 
         # @param name [String,Regexp,Proc] Matcher for case equality (===) with the trace name,
         #             defaults to always match
         # @param service [String,Regexp,Proc] Matcher for case equality (===) with the service name,
         #                defaults to always match
         # @param resource [String,Regexp,Proc] Matcher for case equality (===) with the resource name,
         #                defaults to always match
-        def initialize(name: MATCH_ALL, service: MATCH_ALL, resource: MATCH_ALL)
+        def initialize(name: MATCH_ALL, service: MATCH_ALL, resource: MATCH_ALL, tags: {})
           super()
           @name = name
           @service = service
           @resource = resource
+          @tags = tags
         end
 
         def match?(trace)
-          name === trace.name && service === trace.service && resource === trace.resource
+          name === trace.name && service === trace.service && resource === trace.resource && tags_match?(trace)
+        end
+
+        private
+
+        # Match against the trace tags and metrics.
+        def tags_match?(trace)
+          @tags.all? do |name, matcher|
+            tag = trace.get_tag(name)
+
+            # Format metrics as strings, to allow for partial number matching (/4.*/ matching '400', '404', etc.).
+            # Because metrics are floats, we use the '%g' format specifier to avoid trailing zeros, which
+            # can affect exact string matching (e.g. '400' matching '400.0').
+            tag = format('%g', tag) if tag.is_a?(Numeric)
+
+            matcher === tag
+          end
         end
       end
 

lib/datadog/tracing/sampling/rule.rb

@@ -56,7 +56,7 @@ class SimpleRule < Rule
         # @param sample_rate [Float] Sampling rate between +[0,1]+
         def initialize(
           name: SimpleMatcher::MATCH_ALL, service: SimpleMatcher::MATCH_ALL,
-          resource: SimpleMatcher::MATCH_ALL, sample_rate: 1.0
+          resource: SimpleMatcher::MATCH_ALL, tags: {}, sample_rate: 1.0
         )
           # We want to allow 0.0 to drop all traces, but {Datadog::Tracing::Sampling::RateSampler}
           # considers 0.0 an invalid rate and falls back to 100% sampling.
@@ -69,7 +69,7 @@ def initialize(
           sampler = RateSampler.new
           sampler.sample_rate = sample_rate
 
-          super(SimpleMatcher.new(name: name, service: service, resource: resource), sampler)
+          super(SimpleMatcher.new(name: name, service: service, resource: resource, tags: tags), sampler)
         end
       end
     end

lib/datadog/tracing/sampling/rule_sampler.rb

@@ -62,6 +62,7 @@ def self.parse(rules, rate_limit, default_sample_rate)
               name: rule['name'],
               service: rule['service'],
               resource: rule['resource'],
+              tags: rule['tags'],
               sample_rate: sample_rate,
             }
 

spec/datadog/tracing/sampling/matcher_spec.rb

@@ -4,11 +4,17 @@
 
 RSpec.describe Datadog::Tracing::Sampling::SimpleMatcher do
   let(:trace_op) do
-    Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
+    Datadog::Tracing::TraceOperation.new(
+      name: trace_name,
+      service: trace_service,
+      resource: trace_resource,
+      tags: trace_tags
+    )
   end
   let(:trace_name) { 'operation.name' }
   let(:trace_service) { 'test-service' }
   let(:trace_resource) { 'test-resource' }
+  let(:trace_tags) { {} }
 
   describe '#match?' do
     subject(:match?) { rule.match?(trace_op) }
@@ -108,6 +114,81 @@
         end
       end
 
+      context 'with a tags matcher' do
+        let(:rule) { described_class.new(tags: tags) }
+
+        context 'when span tags are present' do
+          let(:trace_tags) { { 'tag1' => 'value1', 'tag2' => 'value2' } }
+
+          context 'with a regexp' do
+            context 'matching' do
+              let(:tags) { { 'tag1' => /value.*/, 'tag2' => /.*/ } }
+
+              it { is_expected.to eq(true) }
+            end
+
+            context 'not matching' do
+              let(:tags) { { 'tag1' => /value.*/, 'tag2' => /not_value/ } }
+
+              it { is_expected.to eq(false) }
+            end
+          end
+
+          context 'with a string' do
+            context 'matching' do
+              let(:tags) { trace_tags }
+
+              it { is_expected.to eq(true) }
+            end
+
+            context 'not matching' do
+              let(:tags) { { 'tag1' => 'value1', 'tag2' => 'not_value' } }
+
+              it { is_expected.to eq(false) }
+            end
+          end
+        end
+
+        context 'when span metrics are present' do
+          # Metrics are stored as tags, but have numeric values
+          let(:trace_tags) { { 'metric1' => 1.0, 'metric2' => 2 } }
+
+          context 'with a regexp' do
+            context 'matching' do
+              let(:tags) { { 'metric1' => /1/, 'metric2' => /.*/ } }
+
+              it { is_expected.to eq(true) }
+            end
+
+            context 'not matching' do
+              let(:tags) { { 'metric1' => /1/, 'metric2' => 3 } }
+
+              it { is_expected.to eq(false) }
+            end
+          end
+
+          context 'with a string' do
+            context 'matching' do
+              let(:tags) { { 'metric1' => '1', 'metric2' => '2' } }
+
+              it { is_expected.to eq(true) }
+            end
+
+            context 'not matching' do
+              let(:tags) { { 'metric1' => '1', 'metric2' => 'not_value' } }
+
+              it { is_expected.to eq(false) }
+            end
+          end
+        end
+
+        context 'when span tags are not present' do
+          let(:tags) { { 'tag1' => 'value1', 'tag2' => 'value2' } }
+
+          it { is_expected.to eq(false) }
+        end
+      end
+
       context 'when trace service is not present' do
         let(:trace_service) { nil }
         let(:service) { /.*/ }
@@ -196,7 +277,7 @@
 end
 
 RSpec.describe Datadog::Tracing::Sampling::ProcMatcher do
-  let(:trace_op) { Datadog::Tracing::SpanOperation.new(trace_name, service: trace_service) }
+  let(:trace_op) { Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service) }
   let(:trace_name) { 'operation.name' }
   let(:trace_service) { nil }
 

spec/datadog/tracing/sampling/rule_sampler_spec.rb

@@ -131,6 +131,15 @@
         end
       end
 
+      context 'and tags' do
+        let(:rule) { { sample_rate: 0.1, tags: { tag: 'test-tag' } } }
+
+        it 'parses matching tag' do
+          expect(actual_rule.matcher.tags).to eq({ 'tag' => 'test-tag' })
+          expect(actual_rule.sampler.sample_rate).to eq(0.1)
+        end
+      end
+
       context 'with multiple rules' do
         let(:rules) { [{ sample_rate: 0.1 }, { sample_rate: 0.2 }] }
 

spec/datadog/tracing/sampling/rule_spec.rb

@@ -7,11 +7,17 @@
 
 RSpec.describe Datadog::Tracing::Sampling::Rule do
   let(:trace_op) do
-    Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
+    Datadog::Tracing::TraceOperation.new(
+      name: trace_name,
+      service: trace_service,
+      resource: trace_resource,
+      tags: trace_tags
+    )
   end
   let(:trace_name) { 'operation.name' }
   let(:trace_service) { 'test-service' }
   let(:trace_resource) { 'test-resource' }
+  let(:trace_tags) { {} }
 
   let(:rule) { described_class.new(matcher, sampler) }
   let(:matcher) { instance_double(Datadog::Tracing::Sampling::Matcher) }
@@ -79,24 +85,34 @@
 
 RSpec.describe Datadog::Tracing::Sampling::SimpleRule do
   let(:trace_op) do
-    Datadog::Tracing::TraceOperation.new(name: trace_name, service: trace_service, resource: trace_resource)
+    Datadog::Tracing::TraceOperation.new(
+      name: trace_name,
+      service: trace_service,
+      resource: trace_resource,
+      tags: trace_tags
+    )
   end
   let(:trace_name) { 'operation.name' }
   let(:trace_service) { 'test-service' }
   let(:trace_resource) { 'test-resource' }
+  let(:trace_tags) { {} }
 
   describe '#initialize' do
-    subject(:rule) { described_class.new(name: name, service: service, resource: resource, sample_rate: sample_rate) }
+    subject(:rule) do
+      described_class.new(name: name, service: service, resource: resource, sample_rate: sample_rate, tags: tags)
+    end
 
     let(:name) { double('name') }
     let(:service) { double('service') }
     let(:resource) { double('resource') }
+    let(:tags) { { 'tag' => 'value' } }
     let(:sample_rate) { 0.123 }
 
     it 'initializes with the correct values' do
       expect(rule.matcher.name).to eq(name)
       expect(rule.matcher.service).to eq(service)
       expect(rule.matcher.resource).to eq(resource)
+      expect(rule.matcher.tags).to eq(tags)
       expect(rule.sampler.sample_rate).to eq(sample_rate)
     end
   end