Max reader algo #1352
Merged
Max reader algo #1352
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
packages/elasticsearch-asset-apis/src/elasticsearch-reader-api/algorithms/idSlicer-refactor.ts
Outdated
Show resolved
Hide resolved
packages/elasticsearch-asset-apis/src/elasticsearch-reader-api/interfaces.ts
Outdated
Show resolved
Hide resolved
|
In the ElasticsearchReaderApi getIndexDate function - i think you can set track_total_hits: false there too |
|
I think in SpacesReaderClient getDataType you can just set track_total_hits: false |
…only for slicer count calls
godber
approved these changes
Feb 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For date and id slicer/reader
two new feature flags for experimentation:
total_optimization: defaults to false, if set to true, will will change the track_total_hits parameter inside the query to not find the exact total count for a given query. This is useful for dealing with slices and reads of significant data density. Setting this to true, you do not gain much benefit at all with therecurse_optimizationflag so its advised not to use them both together.recurse_optimization: defaults to false, if set to true, it attempts to lower the amount of times it has to recount a query to make a slice by make smarter chunk sizes based off the total count instead of its simple splitting logic, only really useful iftotal_optimizationis set to falseFor spaces slicer/reader
The two new feature flags for experimentation are the same except that
total_optimizationdefaults to true as per the current usage of this does so in spaces