fix(http): Stop plugins from leaking file descriptors on telegraf reload

[nick-kentik]

Summary

On reload, telegraf rebuilds its plugins, which means creating a new *http.Client (and *http.Transport) in each of the files touched in this PR. In general, transports should be long-lived; they carry state and possibly idle HTTP connections.

Without explicitly calling CloseIdleConnections(), in cases where no idle_conn_timeout is set, we leave it up to the go runtime to decide when to drop the connections - and it can easily decide "never", leading to open connections accumulating over time, especially with repeated reloads of the telegraf config in a single process, and eventually telegraf failing with EMFILE problems.

Fix this by telling the Go runtime (by calling CloseIdleConnections()) when we're done with a given transport.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15177

[nick-kentik]

Honestly, I don't like this PR very much, but it does solve the fd leaks.

I feel like the common HTTP client could be reworked so the teardown step is umissable - a few ways to do that. It would also be better if I didn't need all the dummy Start() plugin methods. I wanted to steer clear of big refactorings though, especially those touching telegraf core.

[srebhan]

Thanks for your fix @nick-kentik! Some comments in the code... I generally like the changes in the code (even though you seem to dislike them ;-)). We could think of adding a idle-timeout setting to the common HTTP client and use it (or disable it) where applicable...

[nick-kentik]

Thanks @srebhan , suggestions applied.

I don't particularly like this solution, but I like the file descriptor leaks much less 😅. Plenty of time to improve things once the bugs are squashed.

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[srebhan]

Thanks @nick-kentik! Feel free to submit with a follow up PR to add a config option with a reasonable (long) default for the idle-timeout!

[powersj]

Thanks for the PR. We had internally talked about this yesterday and agreed that this was the way forward as well. So the timing of your PR was perfect.