Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(inputs.execd): read from stdout using ReadLine instead of scanner.Scan to overcome 64kb buffer limit #12935

Merged
merged 7 commits into from
Mar 28, 2023
Merged

fix(inputs.execd): read from stdout using ReadLine instead of scanner.Scan to overcome 64kb buffer limit #12935

merged 7 commits into from
Mar 28, 2023

Conversation

spaghettidba
Copy link
Contributor

  • Updated associated README.md.
  • Wrote appropriate unit tests.
  • Pull request title or commits are in [conventional commit format]

Resolves #12934

@spaghettidba
Changed scan behavior and buffer size
f7034f5 
When reading the process' output stream, scanner.Scan() can fail to process lines longer than the default scan buffer (64Kb). In that case, execd logs the error and stops reading stdout, even if the process is still running.
- Added a parameter for buffer size (in Kb), with default 64Kb
- Changed the code to continue scanning stdout even when the scan errors out
@spaghettidba
New implementation with ReadLine
2abd38f
@telegraf-tiger telegraf-tiger bot added area/execd Issues related to execd or plugins that would be better suited to be used through execd fix pr to fix corresponding bug plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Mar 25, 2023
powersj
powersj reviewed Mar 27, 2023
View reviewed changes
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank for the PR to go along with the issue! Couple requests in line, let me know if you think that is possible.

powersj
powersj approved these changes Mar 27, 2023
View reviewed changes
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

FYI I pushed one change to fix the linter so I could approve

@powersj powersj added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Mar 27, 2023
srebhan
srebhan reviewed Mar 28, 2023
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small comment @spaghettidba.

@spaghettidba @srebhan
Made buffer_size optional in sample.conf
b8eca88 
Co-authored-by: Sven Rebhan <36194019+srebhan@users.noreply.github.com>
@telegraf-tiger
Copy link
Contributor

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

👍 This pull request doesn't change the Telegraf binary size

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

srebhan
srebhan approved these changes Mar 28, 2023
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spaghettidba for this nice fix!

@srebhan srebhan merged commit bfeae49 into influxdata:master Mar 28, 2023
srebhan pushed a commit that referenced this pull request Apr 3, 2023
@spaghettidba @srebhan
fix(inputs.execd): Read from stdout using ReadLine instead of scanner…
4b642e7 
….Scan to overcome 64kb buffer limit (#12935)

(cherry picked from commit bfeae49)
adaszko
adaszko reviewed Jun 20, 2023
View reviewed changes
plugins/inputs/execd/README.md
@@ -50,6 +50,10 @@ See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
## Delay before the process is restarted after an unexpected termination
restart_delay = "10s"

## Buffer size used to read from the command output stream
## Optional parameter. Default is 64 Kib, minimum is 16 bytes
# buffer_size = "64Kib"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

2023-06-20T10:40:29Z I! Loading config: /etc/telegraf/telegraf.conf
2023-06-20T10:40:29Z E! error loading config file /etc/telegraf/telegraf.conf: error parsing execd, line 218: (execd.Execd.BufferSize) units: unknown unit Kib in 128Kib

Getting these errors with Kib. Shouldn’t the unit say KiB?

@srebhan srebhan added this to the v1.26.1 milestone Jun 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adaszko adaszko adaszko left review comments

@powersj powersj powersj approved these changes

@srebhan srebhan srebhan approved these changes

Assignees

@srebhan srebhan

Labels
area/execd Issues related to execd or plugins that would be better suited to be used through execd fix pr to fix corresponding bug plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
v1.26.1
Development

Successfully merging this pull request may close these issues.

inputs.execd: collection stops when the stdout scanner fails
4 participants
@spaghettidba @adaszko @powersj @srebhan