LUCENE-10318: Introduce HNSW merge from graph prototype #11719
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces prototype for initializing new graph during merge from an existing graph. Graph selected for initialization must have 0 deleted docs and have the comparatively the largest number of vectors in it. In order to implement this, capability of inserting out of order nodes into the graph had to be introduced. Additionally, scores for initializer graph had to be recomputed. Signed-off-by: John Mazanec <jmazane@amazon.com>
Avoids using binary search to find the correct position to insert a node into the array that tracks the nodes present in a level. Common cases occur when insertions are in order. Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: John Mazanec <jmazane@amazon.com>
Signed-off-by: John Mazanec <jmazane@amazon.com>
Connects graph to node 0 on merge so that the graph is able to stay connected during merge. This allows nodes outside of the initializer graph to connect to the initializer graph. Comment out test asserting that graphs are equal when building with merge versus without. Signed-off-by: John Mazanec <jmazane@amazon.com>
Refactors initialize from graph. Fixes initializerGraph ordinal to new graph ordinal mapping. Refactoring in HNSWGraphBuilder. Signed-off-by: John Mazanec <jmazane@amazon.com>
Further refactoring on tests and merge implementation. Signed-off-by: John Mazanec <jmazane@amazon.com>
Uses the vectorsCopy in HnswGraphBuilder when recomputing scores of the old graphs so that the ordering of neighbors is correct. Changes test to assert graphs are equal when no skips are provided. Signed-off-by: John Mazanec <jmazane@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Prototype for initializing new graph during merge from an existing graph. Graph selected for initialization must have 0 deleted docs and have the comparatively the largest number of vectors in it.
As next steps, I will run some experiments with this prototype to determine if this improves indexing/merge speed. Ill post results in #11354