tools bucket rewrite: Impossible to have multiple matchers?

[ahurtaud]

Thanos, Prometheus and Golang version used:
thanos v0.19.0-rc.0

Object Storage Provider:
S3

What happened:
I tried to rewrite some blocks in dry-run for now and I am not happy with the output...
I tried to add multiple matchers (either by multiple arrays or multiple label matchers) and for both solution, it is doing an "OR" matcher and not an "AND" matcher like prometheus:

For example if I try:

- matchers: \"{job=\\\"job-to-delete\\\",__name__=\\\"up\\\"}\"

It will delete ALL timeseries of the job "job-to-delete" AND all metrics "up" from all other jobs as well.

Same if I try something like:

- matchers: \"{job=\\\"job-to-delete\\\"}\"
- matchers: \"{__name__=\\\"up\\\"}\"

What you expected to happen:
I would like only the "up" metrics of the job "job-to-delete" to be removed

How to reproduce it (as minimally and precisely as possible):
Dry run a deletion with a matcher for a common metric (up) and a job:

thanos tools bucket rewrite --id=01EZW9YDX86ZGANKC5VY3E8DHZ --objstore.config-file=bucket-prd.yaml --rewrite.to-delete-config="
- matchers: \"{__name__=~\\\"up\\\",job=~\\\"anyjob\\\"}\"
" --dry-run --log.level=debug

cat the change.log and see that all "up" metrics are deleted wether or not part of "anyjob", and some "non up" metrics are deleted from the job "anyjob".

Anything else we need to know:
I would also expect something like:

- matchers: \"{job=\\\"job-to-delete\\\",job=\\\"another-job\\\"}\"

to never match and delete anything.

[yeya24]

Valid requests. This seems a bug to me. We should use AND between matchers instead of OR.
WDYT @bwplotka?

@ahurtaud PR opened at #3886, it would be good if you could help test it and see if it matches your usecase.