Implemented a "group by trace" processor. #1362

jpkrohling · 2020-07-15T10:19:06Z

Signed-off-by: Juraci Paixão Kröhling juraci@kroehling.de

Description: This change adds a new 'groupbytrace' processor, as defined on #1309. This addresses the first item in the action plan.

Link to tracking Issue: #1309

Testing: extensive unit tests have been added

Documentation: sample configuration + readme added

codecov · 2020-07-15T10:24:34Z

Codecov Report

Merging #1362 into master will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1362      +/-   ##
==========================================
+ Coverage   91.35%   91.49%   +0.14%     
==========================================
  Files         240      245       +5     
  Lines       16744    17045     +301     
==========================================
+ Hits        15297    15596     +299     
- Misses       1044     1045       +1     
- Partials      403      404       +1

Impacted Files	Coverage Δ
processor/groupbytraceprocessor/event.go	`100.00% <100.00%> (ø)`
processor/groupbytraceprocessor/factory.go	`100.00% <100.00%> (ø)`
processor/groupbytraceprocessor/processor.go	`100.00% <100.00%> (ø)`
processor/groupbytraceprocessor/ring_buffer.go	`100.00% <100.00%> (ø)`
processor/groupbytraceprocessor/storage_memory.go	`100.00% <100.00%> (ø)`
service/defaultcomponents/defaults.go	`84.90% <100.00%> (+0.29%)`	⬆️
receiver/otlpreceiver/otlp.go	`91.66% <0.00%> (-2.78%)`	⬇️
translator/internaldata/resource_to_oc.go	`87.50% <0.00%> (ø)`
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 11198b1...03995c6. Read the comment docs.

processor/groupbytraceprocessor/factory.go

jpkrohling · 2020-07-16T08:16:56Z

@pjanotti, @bogdandrutu , @tigrannajaryan , would one of you please review this one? I marked it as draft for now, as there might be things to discuss, but from my perspective, this is ready for review.

jpkrohling · 2020-07-21T06:58:59Z

@owais , would you be able to look into this one, please?

jrcamp

Can you expand on what the use case should be in the REAMDE (reading the linked issue looks like it's for tail based sampling).

Is this a common enough use case to go into core or should it go in contrib?

processor/groupbytraceprocessor/README.md

processor/groupbytraceprocessor/config.go

jpkrohling · 2020-07-22T08:09:47Z

Is this a common enough use case to go into core or should it go in contrib?

My idea is that once this processor gets stable enough, the tail-based sampler starts using it instead of having its own similar logic.

jpkrohling · 2020-07-23T13:23:59Z

I just realized that I left a change to service/defaultcomponents/defaults.go out. Will amend this PR later to include this, but the change there is trivial and doesn't block the review.

nilebox

A few initial comments on the design.

Using pdata model without conversion to OTLP might be better, and that will allow you will to use TraceID.String() instead of hashing.

processor/groupbytraceprocessor/config.go

processor/groupbytraceprocessor/factory.go

processor/groupbytraceprocessor/processor.go

jpkrohling · 2020-07-27T11:55:02Z

Using pdata model without conversion to OTLP might be better, and that will allow you will to use TraceID.String() instead of hashing.

Based on your suggestion, the last commit changes the PR to use the internal pdata model instead of OTLP. The performance gain seems to be consistent and relevant:

Before the last commit (f81445e), 3 benchmark runs:

$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      7714 ns/op	   12468 B/op	     159 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	7.779s
$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      8129 ns/op	   13159 B/op	     168 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	8.193s
$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      8736 ns/op	   13946 B/op	     178 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	8.802s

After:

$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      3377 ns/op	    6463 B/op	      54 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	3.438s
$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      2781 ns/op	    5020 B/op	      44 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	2.842s
$ go test -benchmem -run="^$" ./processor/groupbytraceprocessor -bench "^(BenchmarkConsumeTracesCompleteOnFirstBatch)$"
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/collector/processor/groupbytraceprocessor
BenchmarkConsumeTracesCompleteOnFirstBatch-12    	 1000000	      3560 ns/op	    7062 B/op	      59 allocs/op
PASS
ok  	go.opentelemetry.io/collector/processor/groupbytraceprocessor	3.636s

nilebox

I think we should handle the graceful shutdown properly, see relevant comment thread.

processor/groupbytraceprocessor/config.go

processor/groupbytraceprocessor/factory.go

processor/groupbytraceprocessor/processor.go

processor/groupbytraceprocessor/hash.go

processor/groupbytraceprocessor/hash_test.go

processor/groupbytraceprocessor/processor.go

processor/groupbytraceprocessor/storage_memory.go

jpkrohling · 2020-07-29T11:36:14Z

@nilebox, @owais: other than the clarification around the shutdown behavior, is there anything pending for this PR?

nilebox

The overall idea and high-level implementation make sense to me.

However, I personally find the code hard to read with 4 channels and 3 data structures manipulated in more than one place in different ways, but might be just me.
I'd probably prefer something like "event queue" with a few simple steps for each event type.

I would like one of the maintainers (Tigran or Bogdan) to do a review of this code as well.

processor/groupbytraceprocessor/processor.go

processor/groupbytraceprocessor/storage_memory.go

processor/groupbytraceprocessor/processor.go

Need to properly handle spans with different traces

jpkrohling · 2020-07-31T11:54:37Z

The last commit addresses the scenario where resource spans contain spans from multiple traces. There are only two points open: shutdown behavior and nil next consumer.

processor/groupbytraceprocessor/processor.go

nilebox · 2020-07-31T13:20:11Z

The code looks good to me now, but need to check the remaining open questions with maintainers of the repo.
Also not sure if this processor should live in the core repo or contrib.

jpkrohling · 2020-07-31T13:30:37Z

Also not sure if this processor should live in the core repo or contrib.

This has been asked before by @jrcamp, and here's my answer:

My idea is that once this processor gets stable enough, the tail-based sampler starts using it instead of having its own similar logic.

processor/groupbytraceprocessor/processor.go

jpkrohling · 2020-07-31T13:55:52Z

The last commit also removed the nil check on the next consumer, as I agree that it doesn't really make sense for a processor to not have a next consumer.

The only remaining point is about the shutdown then.

@tigrannajaryan, @bogdandrutu, would you please help us on making a decision here? Basically:

what should happen with in-flight traces when the processor is shutting down? Should we just discard them? Should we flush them immediately, blocking until they are consumed, possibly honoring the context's deadline?

jpkrohling · 2020-07-31T14:03:55Z

@nilebox would it be OK if I create an issue to discuss the shutdown behavior, implementing a fix in a follow-up PR?

nilebox · 2020-07-31T14:06:35Z

@jpkrohling sure, I will approve this PR then once you create an issue. Still requires a maintainer to look at it and merge though :)

jpkrohling · 2020-07-31T14:10:41Z

Created #1465

processor/groupbytraceprocessor/factory.go

Closes open-telemetry#1309. Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

jpkrohling · 2020-08-10T12:28:44Z

I addressed @bogdandrutu's comment and squashed all the commits, since this was probably the last change in this PR.

jpkrohling · 2020-08-10T14:31:07Z

@nilebox , @bogdandrutu is there anything extra to be done with this PR? If not, would you mind merging it?

jpkrohling requested review from bogdandrutu, dmitryax, flands, james-bebbington, owais, pjanotti, rghetia and tigrannajaryan as code owners July 15, 2020 10:19

jpkrohling commented Jul 15, 2020

View reviewed changes

processor/groupbytraceprocessor/factory.go Show resolved Hide resolved

jpkrohling marked this pull request as draft July 16, 2020 08:15

jpkrohling marked this pull request as ready for review July 21, 2020 07:08

jrcamp reviewed Jul 21, 2020

View reviewed changes

processor/groupbytraceprocessor/README.md Show resolved Hide resolved

processor/groupbytraceprocessor/config.go Outdated Show resolved Hide resolved

jpkrohling force-pushed the jpkrohling/issue1309 branch from 0258c4f to 18f6be3 Compare July 22, 2020 08:08

jrcamp assigned owais Jul 22, 2020

jpkrohling force-pushed the jpkrohling/issue1309 branch from 29b9c15 to bd80239 Compare July 27, 2020 06:52

jpkrohling requested a review from nilebox as a code owner July 27, 2020 06:52

nilebox suggested changes Jul 27, 2020

View reviewed changes

nilebox reviewed Jul 27, 2020

View reviewed changes

processor/groupbytraceprocessor/processor.go Outdated Show resolved Hide resolved

nilebox reviewed Jul 28, 2020

View reviewed changes

jpkrohling force-pushed the jpkrohling/issue1309 branch from 40e0c8e to 74f8846 Compare July 28, 2020 06:45

nilebox reviewed Jul 30, 2020

View reviewed changes

nilebox self-requested a review July 30, 2020 23:15

nilebox previously approved these changes Jul 30, 2020

View reviewed changes

nilebox assigned nilebox and unassigned owais Jul 31, 2020

jpkrohling force-pushed the jpkrohling/issue1309 branch from a1e5229 to 6c2ac09 Compare July 31, 2020 11:52

nilebox reviewed Jul 31, 2020

View reviewed changes

processor/groupbytraceprocessor/processor.go Outdated Show resolved Hide resolved

nilebox reviewed Jul 31, 2020

View reviewed changes

processor/groupbytraceprocessor/processor.go Outdated Show resolved Hide resolved

jpkrohling mentioned this pull request Jul 31, 2020

Shutdown behavior for groupbyprocessor #1465

Closed

nilebox approved these changes Jul 31, 2020

View reviewed changes

bogdandrutu approved these changes Aug 4, 2020

View reviewed changes

processor/groupbytraceprocessor/factory.go Outdated Show resolved Hide resolved

bogdandrutu added the waiting-for-author label Aug 5, 2020

jpkrohling force-pushed the jpkrohling/issue1309 branch from b557243 to a44919a Compare August 10, 2020 12:27

Implemented a "group by trace" processor.

03995c6

Closes open-telemetry#1309. Signed-off-by: Juraci Paixão Kröhling <juraci@kroehling.de>

jpkrohling force-pushed the jpkrohling/issue1309 branch from a44919a to 03995c6 Compare August 10, 2020 12:28

bogdandrutu merged commit a8a7fcf into open-telemetry:master Aug 11, 2020

chris-smith-zocdoc mentioned this pull request Sep 3, 2020

Separate out aggregation capability from tail based sampling processor open-telemetry/opentelemetry-collector-contrib#16162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented a "group by trace" processor. #1362

Implemented a "group by trace" processor. #1362

jpkrohling commented Jul 15, 2020

codecov bot commented Jul 15, 2020 •

edited

Loading

jpkrohling commented Jul 16, 2020

jpkrohling commented Jul 21, 2020

jrcamp left a comment

jpkrohling commented Jul 22, 2020

jpkrohling commented Jul 23, 2020

nilebox left a comment

jpkrohling commented Jul 27, 2020 •

edited

Loading

nilebox left a comment

jpkrohling commented Jul 29, 2020

nilebox left a comment •

edited

Loading

jpkrohling commented Jul 31, 2020

nilebox commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

nilebox commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Aug 10, 2020

jpkrohling commented Aug 10, 2020

Implemented a "group by trace" processor. #1362

Implemented a "group by trace" processor. #1362

Conversation

jpkrohling commented Jul 15, 2020

codecov bot commented Jul 15, 2020 • edited Loading

Codecov Report

jpkrohling commented Jul 16, 2020

jpkrohling commented Jul 21, 2020

jrcamp left a comment

Choose a reason for hiding this comment

jpkrohling commented Jul 22, 2020

jpkrohling commented Jul 23, 2020

nilebox left a comment

Choose a reason for hiding this comment

jpkrohling commented Jul 27, 2020 • edited Loading

nilebox left a comment

Choose a reason for hiding this comment

jpkrohling commented Jul 29, 2020

nilebox left a comment • edited Loading

Choose a reason for hiding this comment

jpkrohling commented Jul 31, 2020

nilebox commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

nilebox commented Jul 31, 2020

jpkrohling commented Jul 31, 2020

jpkrohling commented Aug 10, 2020

jpkrohling commented Aug 10, 2020

codecov bot commented Jul 15, 2020 •

edited

Loading

jpkrohling commented Jul 27, 2020 •

edited

Loading

nilebox left a comment •

edited

Loading