SOLR-16675: dense vector function queries

[eliaporciani]

https://issues.apache.org/jira/browse/SOLR-16675

Description

Add function queries for dense vector that can be used a rerank time.

Solution

Use the latest changes in LUCENE for apache/lucene#12253 I added a parser in ValueSourceParser:

• vectorSimilarity(vectorEncoding, similarityFunction, valueSource1, valueSource2)

vectorEncoding: FLOAT32 or BYTE
similarityFunction: COSINE, DOT_PRODUCT, EUCLIDEAN
valueSources: here it is accepted or a const vector (e.g. [1,2,3...]) or a fieldName of a DenseVectorField.

Tests

Unit tests for the vector parsing in FunctionQParser:
Integration tests:

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide

[alessandrobenedetti]

I am just wondering if the vector parsing can be done differently (re-using existent code).
I'll investigate and discuss with @eliaporciani , but in the meantime, any other review is welcome!

[alessandrobenedetti]

Also basic documentation and changes is missing

[alessandrobenedetti]

This:
https://solr.apache.org/guide/solr/latest/query-guide/function-queries.html#dist-function
does a similar thing but I don't like the syntax much.

Not that I want to modify the old one, I believe the proposed syntax for the new similarity function is better, but wanted to list the other existing function query for coherence and completeness.

I don't think it's a big deal to have the "distance" function with a syntax and the "similarity" function with a different one for vectors.
Also the code used is completely different i.e. the distance Lucene code is not used and ad hoc code in Solr is used for the distance function query.
My suspicion is that the "dist" between vectors has been there for a while, much earlier than when dense vector fields were introduced in both Lucene and Solr, and no one really updated the code there.

[uschindler]

When updating Lucene, please also fix the startup scripts to pass vector incubator module in command line if Java version is exactly Java 20 or 21. Otherwise Lucene prints warnings and people will not get optimal performance.

So I would really make the Lucene upgrade a separate issue/PR. All documentation should be updated, all startup scripts fixed,....

[alessandrobenedetti]

Yes, it's a separate one, as soon as it's merged we'll proceed with this one

…

On Thu, 6 Jul 2023, 17:44 Uwe Schindler, ***@***.***> wrote: When updating Lucene, please also fix the startup scripts to pass vector incubator module in command line if Java version is exactly Java 20 or 21. Otherwise Lucene prints warnings and people will not get optimal performance. So I would really make the Lucene upgrade a separate issue/PR. All documentation should be updated, all startup scripts fixed,.... — Reply to this email directly, view it on GitHub <#1750 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD5JK22OQ3ORNO6KWYXV7DXO3MPTANCNFSM6AAAAAAZ44PWNY> . You are receiving this because you commented.Message ID: ***@***.***>

[stillalex]

@uschindler the Lucene upgrade is tracked under #1749 I am looking at what needs to be added to the startup script now

[HoustonPutman]

@alessandrobenedetti there are some issues with the string formatting, namely you provide args but don't use "%s". You can probably just replace with a string concatenation.

[alessandrobenedetti]

I agree, not sure why locally the precommit succeeded. Anyway it's just one param to print so I'll change Elia's pr and make it simple,

…

On Fri, 7 Jul 2023, 19:21 Houston Putman, ***@***.***> wrote: @alessandrobenedetti <https://github.com/alessandrobenedetti> there are some issues with the string formatting, namely you provide args but don't use "%s". You can probably just replace with a string concatenation. — Reply to this email directly, view it on GitHub <#1750 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD5JK4G7KYRP5AB64GSDNDXPBAP7ANCNFSM6AAAAAAZ44PWNY> . You are receiving this because you were mentioned.Message ID: ***@***.***>