1.13.1

Fixed

• Tracing: net/http instrumentation excludes query string for http.url tag (#3045)
• Tracing: Remove log_tags warning when given hash for log injection (#3022)
• Tracing: Fix OpenSearch integration loading (#3019)
• Core: Fix default hostname/port when mixing http and uds configuration (#3037)
• Core: Disable Telemetry and Remote Configuration in development environments (#3039)
• Profiling: Improve Datadog::Profiling::HttpTransport error logging (#3038)
• Docs: Document known issues with hanging Resque workers (#3033)

Read the full changeset and the release milestone.

1.13.0

Automated User Login Events for AppSec (Click to expand for more information)

Starting from ddtrace version 1.13.0, Automated User Login Events are enabled by default and currently only available for Devise.

What to expect from Automated User Login Events?

When enabled, any login or signup events will be automatically monitored. We will fill in the span with information extracted from the event.

Modes

The Automated User Login Events are configured to run in "safe" mode by default.

Safe mode focuses on preventing personally identifiable information (PII) from being reported. During safe mode, we only extract the ID from the user object. We report only IDs in the GUID format to avoid disclosing private information. In cases where IDs are not GUID, events will be reported without metadata.

Here is an example of a login event in safe mode:

{
  "appsec.events.users.login.success.track" => true,
  "usr.id" => "5ff35d04-445a-4d00-b6f1-31960153eaf6"
}

Automated User Login Events also provide an "extended" mode.

The extended mode aims to extract the maximum amount of information from the user. In extended mode, we report the ID regardless of its format and try to extract the user's email and username. Depending on the user application, we may be able to extract the user's email and username.

Here is an example of a login event in extended mode:

{
  "appsec.events.users.login.success.track" => true,
  "appsec.events.users.login.success.email" => "john@doe.com",
  "appsec.events.users.login.success.username" => "John Doe",
  "usr.id" => "5ff35d04-445a-4d00-b6f1-31960153eaf6"
}

In cases where safe or extended mode is unable to extract information, we encourage users to manually monitor user login events using our public API.

How to configure Automated User Login Events:

• Use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING to disable or change the mode.
  • To disable, use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING=disable.
  • To change the mode, use DD_APPSEC_AUTOMATED_USER_EVENTS_TRACKING=safe|extended.
• Add to your Datadog.configure block to configure via code:

Datadog.configure do |c|
  c.appsec.track_user_events.enabled = true | false
  c.appsec.track_user_events.mode = "safe" | "extended"
end

Configuration options are now type checked (Click to expand for details)

Starting with ddtrace 1.13, configuration values are type checked when ddtrace is starting up.

In older versions, when setting configuration via Datadog.configure, no validation was performed on values. We've found this could lead to misleading or incorrect configurations, where something seems to be accepted but then did not work as expected. To address this, type checking is now applied to make sure settings are correct.

We've tested this extensively and don't expect this to impact correct configuration setups.

But, if you do have an incorrect configuration, this will now be flagged using an ArgumentError that includes a descriptive message for the setting name and the expected type.

Alternatively, we've provided a way to skip the type checking, by setting the Environment variable DD_EXPERIMENTAL_SKIP_CONFIGURATION_VALIDATION=true. We don't recommend using this option, since it hides configuration problems and thus may lead to ddtrace misbehaving.

As usual, your feedback is welcome -- let us know if we missed anything.

(Feature added in #2983, #2987, #2998 and #2999)

Added

• Core: Add support for Option precedence (#2915)
• Core: Add support for unsetting options (#2972)
• Core: Gauge primitive RubyVM::YJIT.runtime_stats, if YJIT is enabled (#2711, #2959)
• Core: Add Telemetry app-client-configuration-change event (#2977)
• Core: Improve our SafeDup module (#2960)
• Tracing: Add OpenSearch Integration (#2940)
• Tracing: Implement peer.service tag to integrations (#2982)
• Tracing: Add mandatory rpc and grpc tags for grpc integration (#2620, #2962)
• Tracing: Include _dd.profiling.enabled tag (#2913)
• Tracing: Support host injection (#2941, #3007)
• Tracing: Implement Dynamic Configuration for tracing (#2848, #2973)
• Tracing: Add for dynamic log injection configuration (#2992)
• Tracing: Add sampling configuration with DD_TRACE_SAMPLING_RULES (#2968)
• Tracing: Add HTTP header tagging with DD_TRACE_HEADER_TAGS for clients and servers (#2946, #2935)
• Profiling: Add fallback name/invoke location for unnamed threads started in native code (#2993)
• Profiling: Use invoke location as a fallback for nameless threads in the profiler (#2950)
• Profiling: Add fallback name for main thread in profiling data (#2939)
• Ci-app: Add Minitest CI integration (#2932)
• Appsec: Devise integration and automatic user events (#2877)
• Appsec: Handle disabled tracing and appsec events (#2572)
• Appsec: Automate user events check for UUID in safe mode (#2952)
• Docs: Add Ruby 3.2 support to compatibility matrix (#2971)

Changed

• Core: Set maximum supported Ruby version (#2497)
• Core: Prevent telemetry requests from being traced (#2961)
• Core: Add env and type to Core configuration option (#2983, #2988, #2994)
• Core: Remove lazy from Core configuration option (#2931, #2999)
• Profiling: Bump libdatadog dependency to version 3 (#2948)
• Profiling: Improve error message when ddtrace_profiling_loader fails to load (#2957)
• Tracing: Improve log injection runtime conditionals (#2926, #2882)

Fixed

• Core: Fix polynomial-time regular expressions (#2814)
• Core: Fix environment variable for dynamic configuration polling interval (#2967)
• Core: Reduce remote configuration error logging noise (#3011)
• Tracing: Fix manual log injection for 128 bit trace_id (#2974)
• Tracing: Ensure the GRPC client interceptor return the response (#2928)
• Tracing: Remove dynamic input used in regular expression (#2867)
• Tracing: Fix distributed tracing header formats (#3005 )
• Profiling: Fix profiler libmysqlclient version detection with mysql2-aurora gem (#2956)
• Profiling: Automatically enable profiler "no signals" workaround for passenger web server (#2978)

Read the full changeset and the release milestone.

1.12.1

Highlights

Remote configuration

For remote configuration to operate properly its reported service name must match the main application integration service name.

We highly recommend setting the main service name (set via DD_SERVICE or Datadog.configure { |c| c.service = '...' }) and the Rack integration service name to match. Note that when other integrations such as Rails are present and Rack integration configuration is not set, the Rack integration service name will default to the same integration name.

In general, we recommend using DD_SERVICE or c.service instead of c.remote.service if at all possible, but for situations where this is not possible, this version provides an override via a new configuration setting:

Datadog.configure { |c| c.remote.service = '...' }`

It must be made to match the Rack integration service name, which can be found appearing on rack.request tracing spans.

Examples

Case 1

Datadog.configure do |c|
  c.service = 'foo' # or DD_SERVICE env var
  c.tracing.instrument :rails, service_name: 'foo_web' # Rack will inherit this name
end

We recommend:

• either changing c.service to 'foo_web'
• or :rails, service_name: to 'foo'

If none of these changes are possible, as a last resort add c.remote.service = 'foo_web'.

Case 2

Datadog.configure do |c|
  # no c.service = '...' nor DD_SERVICE env var
  c.tracing.instrument :rails, service_name: 'foo_web'
end

We recommend:

• either setting c.service (or DD_SERVICE) to 'foo_web'
• or setting c.service (or DD_SERVICE) to 'foo' and :rails, service_name: to 'foo'

If none of these changes are possible, as a last resort add c.remote.service = 'foo_web'.

Case 3

Datadog.configure do |c|
  # no c.service = '...' nor DD_SERVICE env var
  c.tracing.instrument :rack, service_name: 'foo_rack'
  c.tracing.instrument :rails, service_name: 'foo_rails'
end

We recommend:

• either setting c.service (or DD_SERVICE) to 'foo_rack'
• or setting c.service (or DD_SERVICE) to 'foo' and :rack, service_name: to 'foo'

If none of these changes are possible, as a last resort add c.remote.service = 'foo_rack'.

Added

• Appsec: Add appsec.blocked tag to span (#2895)
• Profiling: Add workaround for legacy profiler incompatibility with ruby-cloud-profiler gem (#2891)
• Core: Allow setting remote configuration service name (#2853)

Changed

• Appsec: Change the value format for the WAF address server.request.query (#2903)
• Profiling: Log pkg-config command when building profiling native extension

Fixed

• Appsec: Update blocked response content_type resolution (#2900)
• Appsec: Ensure to use service entry span. (#2898)
• Tracing: Fix AWS integration constant loading (#2896)

Read the full changeset and the release milestone.

1.12.0

Highlights

Remote configuration

• We fixed an issue with Passlist.

• We have added support for Custom In-App Waf Rules

• In previous versions, there was an issue with identifying the service name correctly. Everything, by default now should work as expected.

Mounted applications

• Appsec now supports mounted apps. That means you can instrument multiple services with appsec.
  Using the example of a Sinatra app mounted as part of a Rails application, you can now instrument both with AppSec. The configuration below should work now.

require 'datadog/appsec'

Datadog.configure do |c|
  c.appsec.enabled = true
  c.appsec.instrument :rails
  c.appsec.instrument :sinatra
end

Added

• Profiling: Add support for profiling Ruby 3.3.0-preview1 (#2860)
• Appsec: Appsec support nested apps (#2836)
• Appsec: Appsec add support for custom rules (#2856)
• Appsec: Update appsec static rules to 1.7.0 version (#2869)
• Appsec: Tag AppSec on Rack top-level span (#2858)
• Profiling: Implement "no signals" workaround and enable CPU Profiling 2.0 for all customers (#2873)
• Ci: Update CI Visibility spec (#2874)
• Appsec: Added missing waf addresses to request operation (#2883)

Changed

• Tracing: Consistent APM Span tags for AWS SDK Requests (#2730)
• Tracing: Change default service_name values Part 2 (#2765)
• Profiling: Bump debase-ruby_core_source dependency to 3.2.1 (#2875)

Fixed

• Telemetry: Disable for non-HTTP agent connection (#2815)
• Tracing: Fix circular requires (#2826)
• Tracing: Update comment about Datadog::Tracing::Distributed::Ext to correct modules (#2840)
• Appsec: Check if :appsec setting is present before accessing it in remote component (#2854)
• Telemetry: Do not send Dependency hash when version is present (#2855)
• Core: Fix symbol configuration for env and service (#2864)
• Tracing: Fix sql comment propagation full mode when tracing is disabled (#2866)
• Appsec: Use relative URI for server.request.uri.raw (#2890)

Read the full changeset and the release milestone.

1.11.1

Fixed

• Appsec: Remove misreported ASM_CUSTOM_RULES capability (#2829)
• Appsec: Fix block response content negotiation (#2824)
• Appsec: Fix incorrect remote configuration payload key handling (#2822)

Read the full changeset and the release milestone.

1.11.0

Highlights

Remote Configuration

The Remote Configuration feature is now in General Availability.

What to expect from Remote Configuration?

ASM can now receive live updates via Remote Configuration, such as:

• protection rules, including request blocking
• IP and path passlists
• IP and user denylists

Remote Configuration will be progressively expanded to support other Datadog products such as APM.

How to configure Remote Configuration?

• Make sure ASM is enabled (see our documentation)
• Upgrade the Datadog agent to 7.42.0 or above (7.43.0 or above recommended)
• Configure the agent to allow Remote Configuration via e.g the environment variableDD_REMOTE_CONFIGURATION_ENABLED=true

To opt out of Remote Configuration, it can be done via either the environment variable DD_REMOTE_CONFIGURATION_ENABLED=false or Datadog.configure { |c| c.remote.enabled = false }.

Read more about Remote Configuration in our documentation

Ruby CPU Profiling 2.0

The new CPU Profiling 2.0 feature is now in General Availability.

What to expect from Ruby CPU Profiling 2.0?

• Finer-grained profiling data due to our sampling engine rewritten in C+Rust. The profiler will be able to run more often and get more information while keeping the same 2% overhead target you're used to, and with a lower impact on latency. Especially when looking at the "Code Hotspots" panel for a distributed trace, expect more and finer grained profiles.
• Thread id information now includes the operating system thread id for Ruby 3.1+, so you'll be able to correlate your thread information when looking at other system monitoring tools
• Thread names are now collected and you're able to filter your profiles by these names
• The profiler now exposes a Datadog::Profiling.allocation_count API that can be used to measure how many objects were allocated in parts of your application
• Experimental support for capturing CPU and Wall-time spent doing Garbage Collection. This is disabled by default as we're still improving the performance of this feature and fixing a few incompatibilities with Ruby Ractors. You can enable it by adding DD_PROFILING_FORCE_ENABLE_GC=true or c.profiling.advanced.force_enable_gc_profiling = true to the instructions seen above.

...with more and faster improvements to come soon!

If you're using the profiler, the new feature will be automatically enabled, except in the following cases:

• When running on Ruby 2.5 and below
• When the mysql2 or rugged gems are installed

Known issues:

• Rare incompatibilities with native extensions/libraries.

  Ruby CPU Profiling 2.0 gathers profiling data by sending SIGPROF unix signals to Ruby applications. This is a common approach used by many other profilers, and it may cause system calls performed by native extensions/libraries to be interrupted with an EINTR error code (reference).

  Most native extensions/libraries are unaffected by this issue, but we know that the mysql2 and rugged gems can trigger these issues (details). When either of these gems is detected, the new feature is not enabled.

  We expect these occurrences to be rare, and will be working to both improve the ecosystem as well as to deploy countermeasures in the profiler itself to avoid triggering these issues.

• Ruby 2.5 and below are missing an API that allows the profiler to detect the currently-active Ruby thread. We plan to ship a workaround for this issue soon.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection is incompatible with Ractors due to Ruby upstream bugs (https://bugs.ruby-lang.org/issues/18464 and https://bugs.ruby-lang.org/issues/19112). We plan to work with the Ruby developers to incorporate fixes for these issues.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection can cause a lot of overhead in Ruby applications with high object allocation rates.

Telemetry

The Telemetry feature is now in General Availability and enabled by default. It will allow Datadog to provide more efficient support.

Added

• Add remote configuration, enabled by default (#2674, #2678, #2686, #2687, #2688, #2689, #2696, #2705, #2731, #2732, #2733, #2739, #2756, #2769, #2771, #2773, #2789, #2805, #2794)
• AppSec: Add response headers passing to WAF (#2701)
• Tracing: Distributed tracing for Sidekiq (#2513)
• Tracing: Add Roda integration (#2368)
• Profiling: Support disabling endpoint names collection in new profiler (#2698)
• Tracing: Support Sidekiq 7 (#2810)
• Core: Add support for Unix Domain Socket (UDS) configuration via DD_TRACE_AGENT_URL (#2806)
• Core: Enable Telemetry by default (#2762)

Changed

• Core: Allow 1 as true value in environment variables (#2710)
• Profiling: Enable CPU Profiling 2.0 by default (#2702)
• Tracing: Improve controller instrumentation and deprecate option exception_controller (#2726)
• Tracing: Implement Span Attribute Schema Environment Variable (#2727)
• Tracing: Change default service_name values (gated by feature flag) (#2760)

Fixed

• Bug: Tracing: Fix w3c propagation special character handling (#2720)
• Performance: Tracing: Use +@ instead of dup for duplicating strings (#2704)
• Profiling: Avoid triggering allocation sampling during sampling (#2690)
• Integrations: Tracing: Fix Rails < 3 conditional check in Utils#railtie_supported? (#2695)
• Profiling: Do not auto-enable new profiler when rugged gem is detected (#2741)
• Tracing: Fix using SemanticLogger#log(severity, message, progname) (#2748) (@rqz13)
• Profiling: Improve detection of mysql2 gem incompatibilities with profiler (#2770)
• AppSec: Remove check for ::Rack::Request.instance_methods.include?(:each_header) at load time (#2778)
• Tracing: Fix quadratic backtracking on invalid URI (#2788)
• Community: Correctly set mutex (#2757) (@ixti)

Read the full changeset and the release milestone.

1.11.0.beta1

As of ddtrace 1.11.0.beta1, the Remote Configuration feature is now in public beta.

What to expect from Ruby Remote Configuration?

ASM can now receive live updates via Remote Configuration, such as:

• protection rules
• IP and path passlists
• IP and user denylists

How to enable?

• Make sure ASM is enabled (see our documentation)
• Configure the agent to allow Remote Configuration via e.g the environment variableDD_REMOTE_CONFIGURATION_ENABLED=true
• Configure your app to start using Remote Configuration via either the environment variableDD_REMOTE_CONFIGURATION_ENABLED=true or Datadog.configure { |c| c.remote.enabled = true }

As of ddtrace 1.11.0.beta1, the new CPU Profiling 2.0 feature is now in General Availability.((Click to expand for details))

What to expect from Ruby CPU Profiling 2.0?

• Finer-grained profiling data due to our sampling engine rewritten in C+Rust. The profiler will be able to run more often and get more information while keeping the same 2% overhead target you're used to, and with a lower impact on latency. Especially when looking at the "Code Hotspots" panel for a distributed trace, expect more and finer grained profiles.
• Thread id information now includes the operating system thread id for Ruby 3.1+, so you'll be able to correlate your thread information when looking at other system monitoring tools
• Thread names are now collected and you're able to filter your profiles by these names
• The profiler now exposes a Datadog::Profiling.allocation_count API that can be used to measure how many objects were allocated in parts of your application
• Experimental support for capturing CPU and Wall-time spent doing Garbage Collection. This is disabled by default as we're still improving the performance of this feature and fixing a few incompatibilities with Ruby Ractors. You can enable it by adding DD_PROFILING_FORCE_ENABLE_GC=true or c.profiling.advanced.force_enable_gc_profiling = true to the instructions seen above.

...with more and faster improvements to come soon!

If you're using the profiler, the new feature will be automatically enabled, except in the following cases:

• When running on Ruby 2.5 and below
• When the mysql2 or rugged gems are installed

Known issues:

• Rare incompatibilities with native extensions/libraries.

  Ruby CPU Profiling 2.0 gathers profiling data by sending SIGPROF unix signals to Ruby applications. This is a common approach used by many other profilers, and it may cause system calls performed by native extensions/libraries to be interrupted with an EINTR error code (reference).

  Most native extensions/libraries are unaffected by this issue, but we know that the mysql2 and rugged gems can trigger these issues (details). When either of these gems is detected, the new feature is not enabled.

  We expect these occurrences to be rare, and will be working to both improve the ecosystem as well as to deploy countermeasures in the profiler itself to avoid triggering these issues.

• Ruby 2.5 and below are missing an API that allows the profiler to detect the currently-active Ruby thread. We plan to ship a workaround for this issue soon.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection is incompatible with Ractors due to Ruby upstream bugs (https://bugs.ruby-lang.org/issues/18464 and https://bugs.ruby-lang.org/issues/19112). We plan to work with the Ruby developers to incorporate fixes for these issues.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection can cause a lot of overhead in Ruby applications with high object allocation rates.

Added

• Add remote configuration beta, disabled by default (#2674, #2678, #2686, #2687, #2688, #2689, #2696, #2705, #2731, #2732, #2733, #2739, #2756, #2769, #2771, #2773, #2789)
• AppSec: Add response headers passing to WAF (#2701)
• Tracing: Distributed tracing for Sidekiq (#2513)
• Tracing: Add Roda integration (#2368)
• Profiling: Support disabling endpoint names collection in new profiler (#2698)

Changed

• Core: Allow 1 as true value in environment variables (#2710)
• Profiling: Enable CPU Profiling 2.0 by default (#2702)
• Tracing: Improve controller instrumentation and deprecate option exception_controller (#2726)
• Tracing: Implement Span Attribute Schema Environment Variable (#2727)

Fixed

• Bug: Tracing: Fix w3c propagation special character handling (#2720)
• Performance: Tracing: Use +@ instead of dup for duplicating strings (#2704)
• Profiling: Avoid triggering allocation sampling during sampling (#2690)
• Integrations: Tracing: Fix Rails < 3 conditional check in Utils#railtie_supported? (#2695)
• Profiling: Do not auto-enable new profiler when rugged gem is detected (#2741)
• Tracing: Fix using SemanticLogger#log(severity, message, progname) (#2748) (@rqz13)
• Profiling: Improve detection of mysql2 gem incompatibilities with profiler (#2770)
• AppSec: Remove check for ::Rack::Request.instance_methods.include?(:each_header) at load time (#2778)
• Tracing: Fix quadratic backtracking on invalid URI (#2788)

Read the full changeset and the release milestone.

1.10.1

Fixed

• CI: Update TeamCity environment variable support (#2668)
• Core: Fix spurious dependency on AppSec when loading CI with require 'datadog/ci' (#2679)
• Core: Allow multiple headers and multiple IPs per header for client IP (#2665)
• AppSec: prevent side-effect on AppSec login event tracking method arguments (#2663)

Read the full changeset and the release milestone.

1.10.0

The release of ddtrace 1.10.0 now supports Ruby 3.2 and 128-bit trace id.

Furthermore, CPU Profiling 2.0 is now in opt-in (that is, disabled by default) beta 3. (Click to expand for details)

You can enable it:

• Using an environment variable by setting DD_PROFILING_FORCE_ENABLE_NEW=true
• Or via code by adding to your Datadog.configure block:

Datadog.configure do |c|
  # ... existing configuration ...
  c.profiling.advanced.force_enable_new_profiler = true
end

What to expect from Ruby CPU Profiling 2.0 beta 3?

• Finer-grained profiling data due to our sampling engine rewritten in C+Rust. The profiler will be able to run more often and get more information while keeping the same 2% overhead target you're used to, and with a lower impact on latency. Especially when looking at the "Code Hotspots" panel for a distributed trace, expect more and finer grained profiles.
• Thread id information now includes the operating system thread id for Ruby 3.1+, so you'll be able to correlate your thread information when looking at other system monitoring tools
• Thread names are now collected and you're able to filter your profiles by these names
• The profiler now exposes a Datadog::Profiling.allocation_count API that can be used to measure how many objects were allocated in parts of your application
• Experimental support for capturing CPU and Wall-time spent doing Garbage Collection. This is disabled by default as we're still improving the performance of this feature and fixing a few incompatibilities with Ruby Ractors. You can enable it by adding DD_PROFILING_FORCE_ENABLE_GC=true or c.profiling.advanced.force_enable_gc_profiling = true to the instructions seen above.

...with more and faster improvements to come soon!

Give it a try, and we'd love to hear your feedback. Below, you'll find a list of known issues that we're still looking into.

Known issues:

• Rare incompatibilities with native extensions/libraries.

  Ruby CPU Profiling 2.0 gathers profiling data by sending SIGPROF unix signals to Ruby applications. This is a common approach used by many other profilers, and it may cause system calls performed by native extensions/libraries to be interrupted with an EINTR error code (reference).

  Most native extensions/libraries are unaffected by this issue, but we know of at least one case: when using the mysql2 gem together with versions of libmysqlclient older than 8.0.0 this can lead to failed database requests (reference). The affected libmysqlclient version is known to be present on Ubuntu 18.04, but not 20.04 and later releases.

  We expect these occurrences to be rare, and will be working to both improve the ecosystem as well as to deploy countermeasures in the profiler itself to avoid triggering these issues.

• Ruby 2.5 and below are missing an API that allows the profiler to detect the currently-active Ruby thread. We have deployed a workaround, but suspect that it may lead to crashes in extremely rare situations. We are still researching a solution for this issue and do not plan on rolling out CPU Profiling 2.0 automatically to Ruby 2.5 and below applications until it is fixed.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection is incompatible with Ractors due to Ruby upstream bugs (https://bugs.ruby-lang.org/issues/18464 and https://bugs.ruby-lang.org/issues/19112). We plan to work with the Ruby developers to incorporate fixes for these issues.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection can cause a lot of overhead in Ruby applications with high object allocation rates. We will be fixing this soon!

Added

• Support Ruby 3.2 (#2601)
• Publish init container image (beta) for dd-trace-rb injection through K8s admission controller (#2606)
• Tracing: Support 128 bits trace id (#2543)
• Tracing: Add tags to integrations (que / racecar / resque/ shoryken / sneakers / qless / delayed_job / kafka / sidekiq / dalli / presto / elasticsearch) (#2619, #2613 , #2608, #2590)
• Appsec: Introduce AppSec::Instrumentation::Gateway::Argument (#2648)
• Appsec: Block request when user ID matches rules (#2642)
• Appsec: Block request base on response addresses matches (#2605)
• Appsec: Allow to set user id denylist (#2612)
• Profiling: Show profiler overhead in flamegraph for CPU Profiling 2.0 (#2607)
• Profiling: Add support for allocation samples to ThreadContext (#2657)
• Profiling: Exclude disabled profiling sample value types from output (#2634)
• Profiling: Extend stack collector to be able to record the alloc-samples metric (#2618)
• Profiling: Add Profiling.allocation_count API for new profiler (#2635)

Changed

• Tracing: rack instrumentation counts time spent in queue as part of the http_server.queue span (#2591) (@agrobbin)
• Appsec: Update waf ruleset to 1.5.2 (#2662, #2659, #2598)
• Appsec: Update libddwaf version to 1.6.2.0.0 (#2614)
• Profiling: Upgrade profiler to use libdatadog v2.0.0 (#2599)
• Profiling: Remove support for profiling Ruby 2.2 (#2592)

Fixed

• Fix broken Ruby VM statistics for Ruby 3.2 (#2600)
• Tracing: Fix 'uninitialized constant GRPC::Interceptor' with gapic-common gem (#2649)
• Appsec: Make sure to assign a valid processor to appsec component (#2637)
• Profiling: Fix profiler not adding the "In native code" placeholder (#2594)
• Fix profiler detection for google-protobuf installation (#2595)

Read the full changeset and the release milestone

1.9.0

As of ddtrace 1.9.0, CPU Profiling 2.0 is now in opt-in (that is, disabled by default) beta 2. (Click to expand for details)

​You can enable it:

• Using an environment variable by setting DD_PROFILING_FORCE_ENABLE_NEW=true
• Or via code by adding to your Datadog.configure block:
  ​

Datadog.configure do |c|
  # ... existing configuration ...
  c.profiling.advanced.force_enable_new_profiler = true
end

​What to expect from Ruby CPU Profiling 2.0 beta 2?

• Finer-grained profiling data due to our sampling engine rewritten in C+Rust. The profiler will be able to run more often and get more information while keeping the same 2% overhead target you're used to, and with a lower impact on latency. Especially when looking at the "Code Hotspots" panel for a distributed trace, expect more and finer grained profiles.
• Thread id information now includes the operating system thread id for Ruby 3.1+, so you'll be able to correlate your thread information when looking at other system monitoring tools
• Thread names are now collected and you're able to filter your profiles by these names
• Experimental support for capturing CPU and Wall-time spent doing Garbage Collection. This is disabled by default as we're still improving the performance of this feature and fixing a few incompatibilities with Ruby Ractors. You can enable it by adding DD_PROFILING_FORCE_ENABLE_GC=true or c.profiling.advanced.force_enable_gc_profiling = true to the instructions seen above.

...with more and faster improvements to come in early 2023!
​
Give it a try, and we'd love to hear your feedback. Below, you'll find a list of known issues that we're still looking into.
​
Known issues:

• Profiling CPU-time overhead is not shown in flamegraphs (unlike with the existing profiler). We will be fixing this soon!

• Rare incompatibilities with native extensions/libraries.

  Ruby CPU Profiling 2.0 gathers profiling data by sending SIGPROF unix signals to Ruby applications. This is a common approach used by many other profilers, and it may cause system calls performed by native extensions/libraries to be interrupted with an EINTR error code (reference).

  Most native extensions/libraries are unaffected by this issue, but we know of at least one case: when using the mysql2 gem together with versions of libmysqlclient older than 8.0.0 this can lead to failed database requests (reference). The affected libmysqlclient version is known to be present on Ubuntu 18.04, but not 20.04 and later releases.
  ​
  We expect these occurrences to be rare, and will be working to both improve the ecosystem as well as to deploy countermeasures in the profiler itself to avoid triggering these issues.

• Ruby 2.5 and below are missing an API that allows the profiler to detect the currently-active Ruby thread. We have deployed a workaround, but suspect that it may lead to crashes in extremely rare situations. We are still researching a solution for this issue and do not plan on rolling out CPU Profiling 2.0 automatically to Ruby 2.5 and below applications until it is fixed.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection is incompatible with Ractors due to Ruby upstream bugs (https://bugs.ruby-lang.org/issues/18464 and https://bugs.ruby-lang.org/issues/19112). We plan to work with the Ruby developers to incorporate fixes for these issues.

• The disabled-by-default experimental support for capturing CPU and Wall-time spent doing Garbage Collection can cause a lot of overhead in Ruby applications with high object allocation rates. We will be fixing this soon!

Added

• Tracing: Add Stripe instrumentation (#2557)
• Tracing: Add configurable response codes considered as errors for Net/HTTP, httprb and httpclient (#2501, #2576)(@caramcc)
• Tracing: Flexible header matching for HTTP propagator (#2504)
• Tracing: OpenTelemetry Traces support (#2496)
• Tracing: W3C: Propagate unknown values as-is (#2485)
• AppSec: Add event kit API (#2512)
• Profiling: Allow profiler development on arm64 macOS (#2573)
• Core: Add profiling_enabled state to environment logger output (#2541)
• Core: Add 'type' to OptionDefinition (#2493)
• Allow debase-ruby_core_source 3.2.0 to be used (#2526)

Changed

• Profiling: Upgrade to libdatadog to 1.0.1.1.0 (#2530)
• Appsec: Update appsec rules 1.4.3 (#2580)
• Ci: Update CI Visibility metadata extraction (#2586)

Fixed

• Profiling: Fix wrong libdatadog version being picked during profiler build (#2531)
• Tracing: Support PG calls with a block (#2522)
• Ci: Fix error in teamcity env vars (#2562)

Read the full changeset and the release milestone