Compact: failed deleting non-compaction group directories/files

[goober]

Thanos, Prometheus and Golang version used:

Docker image: thanos/thanos:v0.19.0-rc.1

Command arguments:

compact
  --http-address=0.0.0.0:10902
  --objstore.config-file=/etc/thanos/config/thanos.yaml
  --data-dir=/var/thanos/compact
  --retention.resolution-raw=14d
  --retention.resolution-5m=14d
  --retention.resolution-1h=14d
  --delete-delay=12h
  --no-debug.halt-on-error
  --wait

Object Storage Provider:
Ceph 3.2

What happened:
When we upgraded to v0.19.0-rc.1 from v0.18 to verify that #3795 solves our compact issues (which it did) we have started to notice a lot of warning messages in the thanos-compact logs.

level=warn ts=2021-03-10T23:58:43.986607565Z caller=compact.go:964 msg="failed deleting non-compaction group directories/files, some disk space usage might have leaked. Continuing" err="read dir: open /var/thanos/compact/compact: no such file or directory" dir=/var/thanos/compact/compact

Everything seems to work as expected otherwise.

Could it be relevant to check if the compact folder exists and ignore the following lines if it does not?

https://github.com/thanos-io/thanos/blob/main/pkg/compact/compact.go#L963-L965

[GiedriusS]

This message is harmless but IMHO we should fix this before the final release. Thank you for the report 🤗 I think the correct approach here is to check whether the directory exists before trying to do anything with it. @bwplotka you had suggestions regarding this. Do you agree with my proposed solution here?

[bwplotka]

Thanks, yea I think we are on it with your PR @GiedriusS, so... we are on it (: Nothing to worry about, thanks for reporting!

[wiardvanrij]

This is fixed in the 0.19 release.