[kafka] discovery of groups, topics and partitions #612
Conversation
The consumer_groups, topics, and partitions arguments are each now optional. Any omitted values will be fetched from Zookeeper. If a value is omitted, the parent value must still be it's expected type (typically a dict). Modern Kafka consumers store their offset in Kafka rather than Zookeeper, but this PR does not add support for this. It's tricky because this check is much simpler if it assumes a single place can be queried for all consumer offsets, but currently there's no easy way to do that. Once KIP-88 is implemented, it will be much easier to add this functionality, although it would only work for Kafka brokers >= 0.10.2.0. In the meantime, you can instrument your individual kafka consumers to submit their offsets to Datadog.
|
Logic to the substantial work by @jeffwidman was reviewed in #271, this adds testing logic, and infrastructure, but the code is essentially untouched. So I'll be merging this. Thank you Jeff! |
|
Thanks for pushing this across the finish line @truthbk, I really, really appreciate it. Sorry I didn't get a chance to get back to it until now. I am a little sad that it was merged via squashing, as I had carefully kept each of the commits as atomic units so it was easier to follow the history. Versus now, with all your commits merged in there as a well it makes It also means I get no credit for the work in the git history. Since I'm spending time at my day job patching this, it's better if my changes are directly attributed to me in Again, I am really happy to see this land, and I appreciate you handling the logic around sending an event, etc. |
What does this PR do?
This PR supersedes #271 - contains @jeffwidman work plus added testing infrastructure and submission of events in certain notable scenarios.
Main goal is to allow collecting all consumer groups, topics, partitions, without the need to include them in the configuration YAML. Note that this will hit zookeeper hard. Please see #271 for more details.
Motivation
@jeffwidman work