Skip to content
This repository has been archived by the owner on May 25, 2022. It is now read-only.

Enhanced benchmarking for file_input #166

Merged
merged 1 commit into from
Jun 2, 2021

Conversation

djaglowski
Copy link
Member

This PR extends #165. It reworks the file input operator's benchmarking tests, and adds additional scenarios that we should keep an eye on as we refactor the operator.

@djaglowski djaglowski force-pushed the cleanup-file-input-3 branch from 67947c9 to 354bd1c Compare June 1, 2021 18:41
@djaglowski
Copy link
Member Author

> make bench
go test -benchmem -run=^$ -bench ^* ./...
...
BenchmarkFileInput/Single-8               468019              2152 ns/op             688 B/op         10 allocs/op
BenchmarkFileInput/Glob-8                 110001             11013 ns/op            2762 B/op         42 allocs/op
BenchmarkFileInput/MultiGlob-8            113391             10673 ns/op            2762 B/op         42 allocs/op
BenchmarkFileInput/MaxConcurrent-8         42062             23964 ns/op            2757 B/op         42 allocs/op
BenchmarkFileInput/FngrPrntLarge-8        668179              2124 ns/op             689 B/op         10 allocs/op
BenchmarkFileInput/FngrPrntSmall-8        658712              1890 ns/op             688 B/op         10 allocs/op
PASS
ok      github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/input/file      48.597s

@djaglowski djaglowski marked this pull request as ready for review June 2, 2021 15:11
@djaglowski djaglowski requested a review from a team June 2, 2021 15:11
@jsirianni jsirianni self-requested a review June 2, 2021 15:18
@jsirianni
Copy link
Member

My bench results using a Ryzen 9 3950x (16 cores, 32 thread)

go test -benchmem -run=^$ -bench ^* ./...
...
goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-log-collection/operator/builtin/input/file
cpu: AMD Ryzen 9 3950X 16-Core Processor            
BenchmarkFileInput/Single-32         	  801732	      1347 ns/op	     640 B/op	       7 allocs/op
BenchmarkFileInput/Glob-32           	  335818	      3400 ns/op	    2561 B/op	      30 allocs/op
BenchmarkFileInput/MultiGlob-32      	  333190	      3426 ns/op	    2561 B/op	      30 allocs/op
BenchmarkFileInput/MaxConcurrent-32  	  191235	      5513 ns/op	    2561 B/op	      30 allocs/op
BenchmarkFileInput/FngrPrntLarge-32  	  786253	      1362 ns/op	     640 B/op	       7 allocs/op
BenchmarkFileInput/FngrPrntSmall-32  	  770312	      1376 ns/op	     640 B/op	       7 allocs/op

@djaglowski djaglowski merged commit 6c757ec into open-telemetry:main Jun 2, 2021
@djaglowski djaglowski deleted the cleanup-file-input-3 branch June 2, 2021 16:00
jsirianni pushed a commit to observIQ/stanza that referenced this pull request Jun 25, 2021
jsirianni pushed a commit to observIQ/stanza that referenced this pull request Jun 25, 2021
* reorg test files only, ported from open-telemetry/opentelemetry-log-collection#165

* port otel enhanced file input benchmarking open-telemetry/opentelemetry-log-collection#166

* skip closing files that are already closed
jsirianni pushed a commit to observIQ/stanza that referenced this pull request Jul 8, 2021
* reorg test files only, ported from open-telemetry/opentelemetry-log-collection#165

* port otel enhanced file input benchmarking open-telemetry/opentelemetry-log-collection#166

* skip closing files that are already closed

* port otel file rotation tracking open-telemetry/opentelemetry-log-collection#182

* fix poll()
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants