[FEATURE] Support higher vector dimension limit for lucene

[ankitvij7]

Is your feature request related to a problem?
Lucene is the only engine that supports pre-filtering however it has a vector dimension limit of 1024. This kinds of limits the use of lucene with bigger models like OpenAI text embedding models. I do see that recently that support for pre-filtering was added to faiss, but not to nsmlib

What solution would you like?
The lucene dimension limit in is being actively discussed to be configurable in this Draft PR. However, Elastic with this PR overrides this limit to 2048. Can we do something similar for OpenSearch?

[jmazanec15]

Hi @ankitvij7, for licensing reasons, we cannot refer to Elastic PRs, so we cannot look at the one above.

Id prefer to just maintain consistency with Lucene.

We are adding support for pre-filtering in faiss, btw: #903

[ankitvij7]

@jmazanec15 Thanks for getting back. Is there a plan to support pre-filtering in nsmlib?

[navneet1v]

@ankitvij7 we don't have a plan for that. Reason being Nmslib currently doesn't have that feature.

[vamshin]

@ankitvij7 we have pre filter support in faiss in 2.9 version. Did you get chance to try that?

[jmazanec15]

Closing issue - no activity

[sam-herman]

Hey @jmazanec15 , I am also interested in this one. do you mind if I will assign it to myself and take a stab at it?

[sam-herman]

Per our discussion on slack, it seems that this change is probably no longer needed because Lucene recently merged moving the max dimension limit to codec:
apache/lucene#12436

This allows users to customize the config without overriding it ad-hoc via OpenSearch.
I'm ok with closing this one as it seems as a no-op until OpenSearch will migrate to the latest Lucene version with the change.

[heemin32]

We can keep this thread open for now. Once lucene is updated, we can incorporate the changes in knn repo to support higher dimension.