Regression in thanos v0.35.1

[dghubble]

Thanos, Prometheus and Golang version used:

Starting in thanos v0.35.1, thanos-sidecar will hang waiting for Prometheus to start. But Prometheus isn't supposed to start until after thanos-sidecar. We use the Thanos sidecar --reloader.config-file and --reloader.config-envsubst-file features to populate Prometheus's config, which then allows Prometheus to start with a config that has had substitutions applied.

This is one of the few times in recent years I've had to rollback a Thanos release. Rolling back to v0.35.0 works.

Looking at the changelog, https://github.com/thanos-io/thanos/releases/tag/v0.35.1, it seems like #7323 introduced this.

Object Storage Provider:

What happened:

thanos-sidecar waits on Prometheus to start up.

What you expected to happen:

thanos-sidecar is not supposed to wait on Prometheus. It's responsible for templating the shared config.

How to reproduce it (as minimally and precisely as possible):

Start thanos-sidecar without Prometheus running. Observe that it hangs.

Full logs to relevant components:

Anything else we need to know:

[dghubble]

cc @MichaHoffmann @fpetkovski

[MichaHoffmann]

My bad! Ill make the waiting asynchronous and use a channel to ping the servers that they can start, that way the reloader should run concurrently! Let me draft up a PR

[cgibbs35]

just found this when trying to use a chainguard image been fighting with it for a whole day not understanding why it never populated.