[Backport 0.5-nexus] Write metadata cache data to mappings _meta with refresh time update (#805) #840
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pensearch-project#805) * [0.5-nexus] Write mock metadata cache data to mappings _meta (opensearch-project#744) * write mock metadata cache data to mappings _meta Signed-off-by: Sean Kao <seankao@amazon.com> * Enable write to cache by default Signed-off-by: Sean Kao <seankao@amazon.com> * bugfix: _meta.latestId missing when create index Signed-off-by: Sean Kao <seankao@amazon.com> * set and unset config in test suite Signed-off-by: Sean Kao <seankao@amazon.com> * fix: use member flintSparkConf Signed-off-by: Sean Kao <seankao@amazon.com> --------- Signed-off-by: Sean Kao <seankao@amazon.com> * default metadata cache write disabled Signed-off-by: Sean Kao <seankao@amazon.com> * remove string literal "external" in index builder Signed-off-by: Sean Kao <seankao@amazon.com> * track refreshInterval and lastRefreshTime Signed-off-by: Sean Kao <seankao@amazon.com> * add last refresh timestamps to metadata log entry Signed-off-by: Sean Kao <seankao@amazon.com> * update metadata cache test case: should pass Signed-off-by: Sean Kao <seankao@amazon.com> * move to spark package; get refresh interval Signed-off-by: Sean Kao <seankao@amazon.com> * parse refresh interval Signed-off-by: Sean Kao <seankao@amazon.com> * minor syntax fix on FlintSpark.createIndex Signed-off-by: Sean Kao <seankao@amazon.com> * strategize cache writer interface Signed-off-by: Sean Kao <seankao@amazon.com> * update refresh timestamps in FlintSpark Signed-off-by: Sean Kao <seankao@amazon.com> * add test cases Signed-off-by: Sean Kao <seankao@amazon.com> * IT test for refresh timestamp update Signed-off-by: Sean Kao <seankao@amazon.com> * add doc for spark conf Signed-off-by: Sean Kao <seankao@amazon.com> * change mock table name Signed-off-by: Sean Kao <seankao@amazon.com> * add IT test at FlintSpark level Signed-off-by: Sean Kao <seankao@amazon.com> * test with external scheduler Signed-off-by: Sean Kao <seankao@amazon.com> * refactor refreshIndex method; add test for modes Signed-off-by: Sean Kao <seankao@amazon.com> * fix typo Signed-off-by: Sean Kao <seankao@amazon.com> * fix failed test caused by refactoring Signed-off-by: Sean Kao <seankao@amazon.com> * rename method; add comment Signed-off-by: Sean Kao <seankao@amazon.com> --------- Signed-off-by: Sean Kao <seankao@amazon.com> (cherry picked from commit a07f88f)
ykmr1224
approved these changes
Oct 30, 2024
dai-chen
approved these changes
Oct 30, 2024
|
Please check CI failure. |
|
CI failure is the same caused by external scheduler test cases fail in nexus branch |
c3b1da4
into
opensearch-project:0.5-nexus
3 of 4 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Manually backport #805
Related Issues
_metaas read cache for frontend user to access #746By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.