Add Karafka integration

[nvh0412]

What does this PR do?

Fixed #1660

In this PR, we introduce the Kafka integration for the Karafka gem. Which includes:

1. Distributed tracing by utilizing message.metadata.headers
2. Traces for worker.process and each message executor inside the batch, so we can link them to the origin trace if the distributed tracing is on

Motivation:

We’re integrating Karafka to implement proper distributed tracing in our system with Datadog, as it lacks an official integration. This integration will also enable distributed tracing if the message headers include distributed tracing data.

Distributed tracing will help us create a proper service map, connecting Kafka producers and consumers.

Change log entry

Yes. Add Karafka integration for distributed tracing.

(Added by @ivoanjo)

Additional Notes:

How to test the change?

[drichards-87]

Created a Jira card for Docs Team editorial review.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 80.58252% with 20 lines in your changes missing coverage. Please review.

Project coverage is 97.75%. Comparing base (ce4393e) to head (27e5711).
Report is 41 commits behind head on master.

Files with missing lines	Patch %	Lines
lib/datadog/tracing/contrib/karafka/patcher.rb	51.61%	15 Missing ⚠️
lib/datadog/tracing/contrib/karafka.rb	80.00%	4 Missing ⚠️
lib/datadog/tracing/contrib/karafka/integration.rb	95.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4147      +/-   ##
==========================================
- Coverage   97.78%   97.75%   -0.03%     
==========================================
  Files        1353     1358       +5     
  Lines       81817    81920     +103     
  Branches     4145     4150       +5     
==========================================
+ Hits        80001    80081      +80     
- Misses       1816     1839      +23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mensfeld]

FYI feel free to ping me once remarks are done. I will be happy to help and maybe in the future retire my own instrumentation in favour of the DD one ;)

[drichards-87]

Left a couple of very small suggestions from Docs and approved the PR.

[nvh0412]

Hey @marcotc , do you know how to have these 2 envs?

Neither DATADOG_API_KEY nor DD_API_KEY is in your environment.

re: https://app.circleci.com/pipelines/github/DataDog/dd-trace-rb/19004/workflows/45a01e1d-a2ee-441b-8443-775013999391/jobs/665625

[marcotc]

👋 @nvh0412, thank you so much for your patience.
We are in the middle of a CI migration, so the test runs are a bit confusing now.

One thing I noticed missing is the entry for karafka in the Rakefile, which means tests are not running in CI. It should be added here:

dd-trace-rb/Rakefile

Lines 222 to 269 in 30f6e21

	# Datadog Tracing integrations
	# REMINDER: If adding a new task here, make sure also add it to the `Matrixfile`
	[
	:action_cable,
	:action_mailer,
	:action_pack,
	:action_view,
	:active_model_serializers,
	:active_record,
	:active_support,
	:aws,
	:concurrent_ruby,
	:dalli,
	:delayed_job,
	:elasticsearch,
	:ethon,
	:excon,
	:faraday,
	:grape,
	:grpc,
	:http,
	:httpclient,
	:httprb,
	:kafka,
	:lograge,
	:mongodb,
	:mysql2,
	:opensearch,
	:pg,
	:presto,
	:que,
	:racecar,
	:rack,
	:rake,
	:redis,
	:resque,
	:roda,
	:rest_client,
	:semantic_logger,
	:sequel,
	:shoryuken,
	:sidekiq,
	:sinatra,
	:sneakers,
	:stripe,
	:sucker_punch,
	:suite,
	:trilogy

As well as in the Matrixfile, which you probably copy the current entry for kafka to use for your tests:

dd-trace-rb/Matrixfile

Lines 109 to 111 in 30f6e21

	'kafka' => {
	'activesupport' => '✅ 2.5 / ✅ 2.6 / ✅ 2.7 / ✅ 3.0 / ✅ 3.1 / ✅ 3.2 / ✅ 3.3 / ✅ 3.4 / ✅ jruby'
	},

[nvh0412]

Sacrifice for the linting gods ☠️

This commit msg actually made my day

[TonyCTHsu]

Sorry, the pull request is currently blocked and I am working on

1. With Datadog's Github admin and security team to configure workflows to run properly in order to acquire the required status for merging pull requests.
2. Discussion with the maintainer about the minimum version to target for support.

[TonyCTHsu]

👋 @marco I have prepared a branch tmp/karafka-tracing to circumvent the security measures preventing secret access.

Basically, rebase tmp/karafka-tracing onto the branch of (karafka-tracing) of this pull request and push to trigger the CI. This will trigger our Gitlab pipeline, unit tests and system tests to acquire the required status check to be merged.

Currently, the unit tests are failing because of the minimum version defined subject to test. Static analysis is failing with rubocop and steep .

I did cherrypicked the changes containing autogenerated gemfiles and lockfiles. You might do the same in order to change your dependencies for testing.