fix: issues with prometheus kubernetes pod discovery

[gracewehner]

Required for all PRs:

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format (e.g. feat: or fix:)

Resolves #9600

• Get the pod info from the event using event.Object.(*corev1.Pod) so that the pod isn't an empty struct. This is the equivalent of watcher.Next(pod) when the previous client package was used.

• A go routine is already running that keeps calling the watchPod() function whenever it returns. Right now with the watchPod() function is calling its own go routine, the function returns and watchPod is called again. This continuously creates new threads and new watchers and re-registers pods instead of having one watcher running each time. Changed back to the previous functionality with a for loop that returns when ctx.Done()

• Added defer watcher.Stop() which is the equivalent of defer watcher.Close() when the previous client package was used.

I'm not sure any unit tests can be added to detect these issues since the function relies on the kubernetes watch api call and discovering kubernetes pods.

[telegraf-tiger]

Looks like new artifacts were built from this PR. Get them here!

Artifact URLs

• darwin_amd64.tar.gz

• windows_amd64.zip

• windows_i386.zip

• s390x.rpm

• linux_s390x.tar.gz

• s390x.deb

• armel.rpm

• linux_armel.tar.gz

• armel.deb

• ppc64le.rpm

• linux_ppc64le.tar.gz

• ppc64el.deb

• linux_mipsel.tar.gz

• mipsel.deb

• i386.rpm

• freebsd_i386.tar.gz

• linux_i386.tar.gz

• i386.deb

• static_linux_amd64.tar.gz

• armv6hl.rpm

• freebsd_armv7.tar.gz

• linux_armhf.tar.gz

• armhf.deb

• linux_mips.tar.gz

• mips.deb

• aarch64.rpm

• linux_arm64.tar.gz

• arm64.deb

• x86_64.rpm

• freebsd_amd64.tar.gz

• linux_amd64.tar.gz

• amd64.deb

[sspaink]

lgtm! Thank you for fixing this and describing the why/how so clearly.

[reimda]

Looks reasonable to me. I don't have a deep understanding of k8s.io/client-go/kubernetes but it sounds like you have identified valid problems and provided good fixes.