Compact: Offline deduplication

[smalldirector]

Changes

As the discussion in #1014, implement a new feature in compactor which provides offline deduplication function for the data from different replicas.

The offline deduplication follow the same design with query deduplication. The user needs to specify replica label by config dedup.replica-label before enable it, and the function uses current query deduplication algorithm(penalty algorithm) to merge data points come from different replicas.

The offline deduplication function is based on bucket level in remote storage, so the user needs to ensure that all replica data write to same bucket.

Verification

• Compared data quality by tsdb read API.
• Compared the graph by Thanos query UI.

Below figures are the comparison for one sample metrics before dedup and after dedup. It defines one interval replica label _agg_replica_ to represent the merged data.

• Tested the dedup function online, and monitoring its metrics(especially memory usage).

For below figure, each of block size is around 3GB and we have two replicas for each block. As it is using streaming read/write way to operate block, so no OOM exception happens.

[bwplotka]

Why it was closed? Also can we ensure the PR is split into smaller bits if possible? It will ensure quicker review for sure (:

[smalldirector]

@bwplotka Closed it as I want to have a clean commit history for the review. Actually I opened another new PR: #1276. Can you please check the codes there?

Agree. Smaller PR will make the review easier. However, those codes are for one new feature dedup function, IMO keep everything in one PR will give a better context to know how it works there.
Do you have any suggestion on how to split it?