ES memory write race fix and performance improvement (#67) #91
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
leifmadsen
previously approved these changes
Feb 9, 2022
* Fix Loki readiness check
* Allow multiple API calls to ES
- This patch adds possibility to submit multiple requests to ES in paralel
and hence improving performance.
* Revert "Fix concurrent conflicts in elasticsearch (#53)"
- Unfortunately concurrent.Map is not ready for concurrency,
so we need to avoid using it.
- This reverts commit e61141a.
* Fix concurrency in ES client message buffer
- "fatal error: concurrent map writes"
/go/src/github.com/infrawatch/sg-core/plugins/application/elasticsearch/main.go:95
/go/src/github.com/infrawatch/sg-core/pkg/bus.go:65 calls Application callbacks as a goroutine
which results in concurrency in ReceiveEvent(). I've added a mutex to protect access.
- Co-authored-by: Chris Sibbitt <csibbitt@redhat.com>
* Remove unused package
- concurrent.Map is unused throughout the project and it did not work as expected,
because values could be overwritten when unlocked after returning from method.
* Add amqp1 transport
* Don't transfer whole message through channels
- This patch makes channel synchronization use pointers instead of whole messages,
which improves performance of the elasticsearch app plugin. It also adds mechanism
to flush index buffer in case of remnants of some batch messages are kept
for longer time.
* Integration CI fixes
- simplify rsyslog configuration to work with newer rsyslog
- stop using ELN repo because of rsyslog-omamqp1
(plugin is available in appstream)
- add proper wait time between steps instead of simple sleep
* Update .github/workflows/tests.yml
- Co-authored-by: Leif Madsen <lmadsen@redhat.com>
* Move golint exclude-rules to proper place
(cherry picked from commit 7d46ba5)
a4d2d80 to
44cb460
Compare
|
Rebased on latest stable-1.3 |
leifmadsen
approved these changes
Feb 10, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
and hence improving performance.
so we need to avoid using it.
/go/src/github.com/infrawatch/sg-core/plugins/application/elasticsearch/main.go:95
/go/src/github.com/infrawatch/sg-core/pkg/bus.go:65 calls Application callbacks as a goroutine
which results in concurrency in ReceiveEvent(). I've added a mutex to protect access.
because values could be overwritten when unlocked after returning from method.
which improves performance of the elasticsearch app plugin. It also adds mechanism
to flush index buffer in case of remnants of some batch messages are kept
for longer time.
(plugin is available in appstream)
(cherry picked from commit 7d46ba5)