Add flag monitor_all_broker_highwatermarks, refactor
#4385
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
monitor_all_broker_highwatermarks, refactor
ofek
requested changes
Aug 17, 2019
kafka_consumer/datadog_checks/kafka_consumer/data/conf.yaml.example
Outdated
Show resolved
Hide resolved
ofek
previously approved these changes
Aug 17, 2019
A large refactor, most in preparation for the new version of the check. 1. Stop pointlessly passing groups/topics around. This is much cleaner handled at the object level no point in passing most of this around. 2. Add a flag `monitor_all_broker_highwatermarks` that allows fetching broker offsets for topics that are not part of a consumer's subscription. 3. Cleanup the tests slightly as one test was shoehorned onto another test and the shared state was causing some weird breakage. 4. Streamline the overall code flow so that its more readable. 5. Change the way the context limit is enforced so that it's applied to the sum total, rather than piecemeal. Additionally, because it's applied to the total rather than the pieces, after discussion with @ofek I bumped the context limit to 500. This should be a reasonable default to prevent abuse while still handling most cases. 6. Change so that internal topics like `__consumer_offsets` are never excluded from the broker highwater mark fetch. It's generally pointless to fetch these as users care about the contents of the messages in that topic, not the broker offsets of that particular topic.
Codecov Report
|
ofek
approved these changes
Aug 17, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A large refactor, most in preparation for the new version of the check.
handled at the object level no point in passing most of this around.
monitor_all_broker_highwatermarksthat allows fetchingbroker offsets for topics that are not part of a consumer's
subscription.
test and the shared state was causing some weird breakage.
the sum total, rather than piecemeal. Additionally, because it's applied
to the total rather than the pieces, after discussion with @ofek I
bumped the context limit to 500. This should be a reasonable default to
prevent abuse while still handling most cases.
__consumer_offsetsare neverexcluded from the broker highwater mark fetch. It's generally pointless
to fetch these as users care about the contents of the messages in that
topic, not the broker offsets of that particular topic.