Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More granular control of preloading on MMapDirectory. #11929

Merged
merged 11 commits into from
Nov 15, 2022

Conversation

jpountz
Copy link
Contributor

@jpountz jpountz commented Nov 14, 2022

This enables configuring preloading on MMapDirectory based on the file name as well as the IOContext that is used to open the file.

The default codec has a number of small and hot files, that actually used to be
fully loaded in memory before we moved them off-heap. In the general case,
these files are expected to fully fit into the page cache for things to work
well. Should we give control over preloading to codecs? This is what this
commit does for the following files:
 - Terms index (`tip`)
 - Points index (`kdi`)
 - Stored fields index (`fdx`)
 - Terms vector index (`tvx`)

This only has an effect on `MMapDirectory`.
Copy link
Contributor

@uschindler uschindler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great to me! +1

Have you checked with Elasticserach code if this is enough for their purpose?

@jpountz
Copy link
Contributor Author

jpountz commented Nov 15, 2022

Yes it would be enough. 👍

@jpountz jpountz added this to the 9.5.0 milestone Nov 15, 2022
@jpountz jpountz merged commit 69a7cb2 into apache:main Nov 15, 2022
@jpountz jpountz deleted the mmapdirectory_preload_configuration branch November 15, 2022 09:52
jpountz added a commit that referenced this pull request Nov 15, 2022
This enables configuring preloading on MMapDirectory based on the file name as well as the IOContext that is used to open the file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants