gh-14127: remove duplicate neighbors when writing HNSW graphs

[msokolov]

should fix #gh-14127 test failures.

I believe there are no back-compat concerns here since this would be a no-op for graphs with no duplicates (I think that is what we were always producing before), and even if we did produce dups before, this will produce a functionally-equivalent graph, just a bit more compact.

I also don't think we need to take any special action w.r.t. to the CheckIndex check that was added since if we never produced dups without reordering then it won't randomly start failing on indexes that checked out OK before. Although there is some chance I'm wrong - it's difficult to prove we could not have produced duplicates before, I think we would have seen problems by now with the CheckIndex check having been out in the wild on main and 10x branch?

[iverase]

I was thinking in a solution more like this: #14159. Just open it for discussion, I am ok if the preferred solution is done at this level.

The only merit of the other solution is that we will still create new connections, not sure if there is a benefit on doing that.

[msokolov]

@iverase I see what you did there ... that would also solve this problem, but I think it is less desirable since it (1) requires extending the HNSW search API in a way I think we wouldn't use elsewhere, and (2) overall seems to do a more work. I don't think we need the extra links, at least not to satisfy the intended use case here which is to guarantee some notion of connectivity (all nodes reachable from an entry point).

[iverase]

Sounds good to me @msokolov, I didn't like to add yet a new parameter in the search api. Thanks for taking the time to review it.