LUCENE-9830: Hunspell: store word length for faster dictionary lookup/enumeration #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Solr references probably should be removed from the checklist above |
donnerpeter
commented
Mar 10, 2021
| @@ -65,6 +65,7 @@ | |||
| TrigramAutomaton automaton = new TrigramAutomaton(word); | |||
|
|
|||
| dictionary.words.processAllWords( | |||
| Math.max(1, word.length() - 4), | |||
There was a problem hiding this comment.
We now pass minLength in addition to maxLength
donnerpeter
commented
Mar 10, 2021
| @@ -63,17 +75,14 @@ | |||
| * <li>VINT: a delta pointer to the entry for the same word without the last character. | |||
| * Precisely, it's the difference of this entry's start and the prefix's entry start. 0 for | |||
| * single-character entries | |||
| * <li>Optional, for non-leaf entries only: | |||
| * <li>(Optional, for hash-colliding entries only) | |||
There was a problem hiding this comment.
I've moved collision info before the forms data to avoid skipping over vInts on mismatches
mikemccand
pushed a commit
to mikemccand/lucene
that referenced
this pull request
Sep 3, 2021
mocobeta
pushed a commit
to mocobeta/lucene
that referenced
this pull request
Dec 2, 2021
Add a utility task to list all existing pacage names
This was referenced Dec 8, 2021
jpountz
pushed a commit
to jpountz/lucene
that referenced
this pull request
Mar 22, 2024
benwtrent
pushed a commit
to benwtrent/lucene
that referenced
this pull request
Jul 26, 2024
Add HNSW building to search tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Word length could be checked before more materializing the whole word
Solution
Use the spare bits in the hash table
intand collisionbyteto encode word length, if it's short enough (almost always).Tests
No new tests, 10-15% speedup in
TestPerformance.de_suggest.Checklist
Please review the following and check all that apply:
masterbranch../gradlew check.