Do not query blocks store for time ranges already covered by ingesters #2642
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15fc37c to
4c47d7f
Compare
aa93f40 to
265d2e0
Compare
Signed-off-by: Marco Pracucci <marco@pracucci.com>
265d2e0 to
c733124
Compare
pstibrany
approved these changes
May 29, 2020
There was a problem hiding this comment.
Looks good.
Based on discussion with Marco, chunk storage wouldn't benefit much from this optimization, because reasonable query_store_after parameter needs to be very small (chunks can be flushed to storage soon after series gets stale).
Thanks also for improving tracing!
…-covered-by-ingesters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
When
-querier.query-store-afteris configured and running with the blocks storage, we should be able to safely manipulate the query maxt for series fetched from the store because more recent samples are 100% fetched from ingesters only.This is a quite important optimisation for the blocks storage where we can afford to query ingesters only for the last few hours and thus we could skip querying the most recent non-compacted-yet blocks from the storage.
We tested this change in a large cluster (60 ingesters) running some slow queries keeping any caching disabled (for testing), and we got the following results:
Note to reviewers:
Which issue(s) this PR fixes:
N/A
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]