EMF exporter performance: send EMF logs in batches #2572
Conversation
Codecov Report
@@ Coverage Diff @@
## main #2572 +/- ##
==========================================
- Coverage 91.37% 91.34% -0.03%
==========================================
Files 431 431
Lines 21461 21473 +12
==========================================
+ Hits 19609 19615 +6
- Misses 1387 1391 +4
- Partials 465 467 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
d8c575c to
1e5934d
Compare
| if am.Len() > 0 { | ||
| am.ForEach(func(k string, v pdata.AttributeValue) { | ||
| labels[k] = v.StringVal() | ||
| }) | ||
| } |
There was a problem hiding this comment.
q: why can't we directly take the labels from groupedMetrics.Labels?
There was a problem hiding this comment.
The labels only serves for the purpose of logging. It should be simple and timely. I separated translateOTelToGroupedMetric and label extraction in the latest commit. Another concern is since groupMetrics.Labels contains extra labels from datapoints, they would blow up the info log.
| for _, pusher := range emf.listPushers() { | ||
| returnError := pusher.ForceFlush() | ||
| if returnError != nil { |
There was a problem hiding this comment.
Thanks for catching the performance issue here.
Could you combine this code block with the code in func (emf *emfExporter) Shutdown()?
|
Looks like tests failed due to other components, and there's no way to trigger /retest. |
b8d60d2 to
c775ce3
Compare
Bumps [github.com/golang/snappy](https://github.com/golang/snappy) from 0.0.2 to 0.0.3. - [Release notes](https://github.com/golang/snappy/releases) - [Commits](golang/snappy@v0.0.2...v0.0.3) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…otlpmetric (#2572) * Bump google.golang.org/grpc in /exporters/otlp/otlpmetric Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.43.0 to 1.44.0. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.43.0...v1.44.0) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Auto-fix go.sum changes in dependent modules Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: MrAlias <MrAlias@users.noreply.github.com>
Why do we need it?
This PR is a follow-up PR of #2571. This PR fixes the performance issue described in aws-observability/aws-otel-collector#388.
Instead of pushing logs by every
logEventBatch, it is optimized to push logs in batches. AlogEventBatchwill be sent if 1) it's full or expires 2) the received metrics are all processed.