Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(inputs.kafka_consumer): sarama debug logs #12304

Merged
merged 3 commits into from
Dec 5, 2022
Merged

feat(inputs.kafka_consumer): sarama debug logs #12304

merged 3 commits into from
Dec 5, 2022

Conversation

trvrnrth
Copy link
Contributor

Required for all PRs

Outputs sarama logs at debug level in the same manner as the kafka output plugin in order to aid troubleshooting.

@telegraf-tiger
Copy link
Contributor

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@telegraf-tiger telegraf-tiger bot added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Nov 29, 2022
@trvrnrth
Copy link
Contributor Author

!signed-cla

@trvrnrth trvrnrth force-pushed the kafka-input-debug-master branch 4 times, most recently from 63fbc06 to c9db6fa Compare November 29, 2022 18:55
@trvrnrth trvrnrth force-pushed the kafka-input-debug-master branch from c9db6fa to 0a41b12 Compare November 29, 2022 18:56
powersj
powersj reviewed Nov 29, 2022
View reviewed changes
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - thanks for the PR. This is identical to what exists in the kafka output today. Would you be willing to add the DebugLogger to the common.kafka so we have this in one place?

@trvrnrth
Copy link
Contributor Author

@powersj Sounds like a good idea. Refactored accordingly.

@trvrnrth trvrnrth requested a review from powersj November 29, 2022 23:11
@trvrnrth trvrnrth force-pushed the kafka-input-debug-master branch 2 times, most recently from baf895b to 9ca4131 Compare November 30, 2022 08:41
powersj
powersj approved these changes Nov 30, 2022
View reviewed changes
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving this into a common section.

One thing I realized when playing with this is that sarama's logger is global, so whatever plugin gets initialized last will get used as part of the output. So if a user has a kafka_consumer + kafka output it will lead to some really confusion log output.

This was already an issue before, so nothing to do for this PR.

@powersj powersj added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Nov 30, 2022
@trvrnrth
Copy link
Contributor Author

Thanks for moving this into a common section.

One thing I realized when playing with this is that sarama's logger is global, so whatever plugin gets initialized last will get used as part of the output. So if a user has a kafka_consumer + kafka output it will lead to some really confusion log output.

This was already an issue before, so nothing to do for this PR.

Yeah, it's not ideal. I imagine it might actually be better to log directly, just tagging as sarama much like the agent does.

@srebhan
Copy link
Member

srebhan commented Dec 1, 2022

@trvrnrth and @powersj how about instantiating a new logger here (as @trvrnrth suggested) with a sarama prefix? I think all it needs is a call to models.NewLogger()...

@trvrnrth
Use a separate logger for sarama
f5da4a7 
As the sarama logger is global it doesn't make sense to output on a
plugin log as this is confusing for the case that both the
kafka_consumer input and kafka output plugins are used.
@trvrnrth
Copy link
Contributor Author

trvrnrth commented Dec 1, 2022

@srebhan @powersj That was straightforward enough so I've gone ahead. I instantiate the Logger directly as NewLogger expects a pluginType which we don't have in this context and I though that a prefix of just [sarama] would probably be clearer than making up a plugin type to end up with [kafka.sarama] or [sarama.debug] or the like.

@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Dec 1, 2022

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz
static_linux_amd64.tar.gz

powersj
powersj approved these changes Dec 1, 2022
View reviewed changes
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better!

srebhan
srebhan approved these changes Dec 5, 2022
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @trvrnrth! Thanks for your effort!

@srebhan srebhan merged commit 70b33c2 into influxdata:master Dec 5, 2022
@trvrnrth trvrnrth deleted the kafka-input-debug-master branch December 5, 2022 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@powersj powersj powersj approved these changes

@srebhan srebhan srebhan approved these changes

Assignees

@srebhan srebhan

Labels
feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@trvrnrth @srebhan @powersj