Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazily compute similarity score when reuse the old HNSW graph #12236

Closed
zhaih opened this issue Apr 19, 2023 · 0 comments · Fixed by #12480
Closed

Lazily compute similarity score when reuse the old HNSW graph #12236

zhaih opened this issue Apr 19, 2023 · 0 comments · Fixed by #12480

Comments

@zhaih
Copy link
Contributor

zhaih commented Apr 19, 2023

Description

In #12050 we added ability to reuse old graph as an initializer to speed up the merge, but even when we are re-inserting the old graph's node, we still need to calculate a similarity score here so that we can pop-out the worst non-diverse node here based on a sorted order.
But since the score is only used for diversity checking purpose, we probably do not even need them in some cases (like when we never reach the level connection limit). So we probably can first insert those nodes without calculating the score, then when we eventually need to pop out a worst node, we can calculate the score, sort the neighbor array and then do the normal "find the worst node" procedure.

@zhaih zhaih changed the title Lazily compute similarity score when reuse the old graph Lazily compute similarity score when reuse the old HNSW graph Apr 19, 2023
Jackyrie2 added a commit to Jackyrie2/lucene that referenced this issue Jun 14, 2023
@zhaih zhaih linked a pull request Aug 1, 2023 that will close this issue
@zhaih zhaih added this to the 9.8.0 milestone Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants