Add filter function for NeuralQueryBuilder and HybridQueryBuilder and… #1206

chloewqg · 2025-03-03T08:01:00Z

… modify fromXContent function in HybridQueryBuilder to support filter field.

Description

Add filter function to NeuralQueryBuilder and HybridQueryBuilder, which allows to push down non null filter. One exception is that when HybridQueryBuilder has a nested HybridQueryBuilder, then calling the filter function will cause UnsupportedOperationException

Related Issues

Resolves #1206
Related: #282, #1135

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java

heemin32 · 2025-03-03T22:24:30Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

+        if (!validateFilterParams(filter)) {
+            return this;
+        }
+        HybridQueryBuilder compoundQueryBuilder = new HybridQueryBuilder();


Why do we have to create a new query builder instead of returning this?

I cannot find validateFilterParams method. Besides that the naming is confusing, if you conclude validity of the object based on return value that needs to be in the name, something like isValidFilterParams. Also change negation to an explicit == false

The method is added in core. opensearch-project/OpenSearch#17409
Should we have to use that method? Can we just do null check explicitly here instead of relying on the method?

Hi the validateFilterParams is in the core package, inside the AbstractQueryBuilder class. https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/index/query/AbstractQueryBuilder.java#L94

Why do we have to create a new query builder instead of returning this?

Because I saw the the queries of HybridQueryBuilder is marked as final. https://github.com/opensearch-project/neural-search/blob/main/src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java#L56

So I think as we need to reassign a list of queries to the queries variable inside the HybridQueryBuilder. So we can only create a new HybridQueryBuilder to solve this?

Wouldn't this double the filters sizes? Like if there are two filters, when we add(query.filter(filter)), wouldn't this create duplicate filters?

Thanks Martin for the comment! Actually the function name validateFilterParams was suggested by Daniel in the pull request of core package. opensearch-project/OpenSearch#17409 , is it okay to leave as what it is for now?

if method is already there let's utilize it. Please use == false syntax instead of negation, I see you've used it in other places in your PR

Yep. Updated in the newer commit

Quote reply

I think the filter input check is common across all the query builders. Although current check is simple of whether it equals to null, adding a function here can allow complicated logic changes in the future

I think you can do something like queries.set(i, queries.get(i).filter(filter));

martin-gaievski

there is a separate issue and the whole design/RFC for hybrid query. Is this a building block for that design or this is a conflicting issue?
Ref: #282, #1135

martin-gaievski · 2025-03-04T16:38:21Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

+        if (!validateFilterParams(filter)) {
+            return this;
+        }
+        HybridQueryBuilder compoundQueryBuilder = new HybridQueryBuilder();


I cannot find validateFilterParams method. Besides that the naming is confusing, if you conclude validity of the object based on return value that needs to be in the name, something like isValidFilterParams. Also change negation to an explicit == false

chloewqg · 2025-03-04T17:59:23Z

there is a separate issue and the whole design/RFC for hybrid query. Is this a building block for that design or this is a conflicting issue? Ref: #282, #1135

Hi Martin, this is a building block for that design. cc @bzhangam

martin-gaievski · 2025-03-04T18:36:12Z

there is a separate issue and the whole design/RFC for hybrid query. Is this a building block for that design or this is a conflicting issue? Ref: #282, #1135

Hi Martin, this is a building block for that design. cc @bzhangam

Got it, makes sense then. Can you please mention same in the PR description and also update the link to the feature request issue under "Related issues"? If the filtering feature requires multiple PRs/building blocks I suggest you create a high-level meta issue and host smaller PRs under that meta issue.

codecov · 2025-03-04T18:44:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.90%. Comparing base (5f25d6c) to head (cb7dff7).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1206      +/-   ##
============================================
+ Coverage     81.80%   81.90%   +0.09%     
+ Complexity     2606     1312    -1294     
============================================
  Files           190       95      -95     
  Lines          8922     4482    -4440     
  Branches       1520      765     -755     
============================================
- Hits           7299     3671    -3628     
+ Misses         1032      514     -518     
+ Partials        591      297     -294

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

chloewqg · 2025-03-04T22:01:29Z

there is a separate issue and the whole design/RFC for hybrid query. Is this a building block for that design or this is a conflicting issue? Ref: #282, #1135

Hi Martin, this is a building block for that design. cc @bzhangam

Got it, makes sense then. Can you please mention same in the PR description and also update the link to the feature request issue under "Related issues"? If the filtering feature requires multiple PRs/building blocks I suggest you create a high-level meta issue and host smaller PRs under that meta issue.

Gotcha. Thanks!

bzhangam · 2025-03-04T22:15:04Z

src/test/java/org/opensearch/neuralsearch/query/NeuralQueryIT.java

+     * }
+     */
+    @SneakyThrows
+    public void testQueryWithBoostAndFilterApplied() {


Why we add this IT? I think the filter function is an existing feature for neural query?

Here we are testing whether adding another filter on NeuralQuery would be function as expected.

bzhangam · 2025-03-04T22:17:34Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryPostFilterIT.java

+        }
+    }*/
+    @SneakyThrows
+    private void testRangeQueryAsFilter(String indexName) {


Why we add this IT in PostFilterIT? Filter is different from post filter. I think we should either add it to the HybridQueryIT or create a new HybridQueryFilterIT.

Do you plan to raise another PR to add tests for BWC? BWC tests are under the qa folder.

Why we add this IT in PostFilterIT? Filter is different from post filter. I think we should either add it to the HybridQueryIT or create a new HybridQueryFilterIT.

+1, better add new class HybridQueryFilterIT

Do you plan to raise another PR to add tests for BWC? BWC tests are under the qa folder.

I think the current bwc tests is blocking the merge. So I will do the bwc tests here in the same PR

Why we add this IT in PostFilterIT? Filter is different from post filter. I think we should either add it to the HybridQueryIT or create a new HybridQueryFilterIT.

Sure. Will create a new HybridQueryFilterIT

… modify fromXContent function in HybridQueryBuilder to support filter field. Signed-off-by: Chloe Gao <chloewq@amazon.com>

chloewqg force-pushed the local branch from d05a418 to 1cd7801 Compare March 3, 2025 08:04

chloewqg marked this pull request as ready for review March 3, 2025 18:43

chloewqg requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse, zane-neo, vibrantvarun, zhichao-aws, yuye-aws and minalsha as code owners March 3, 2025 18:43

junqiu-lei reviewed Mar 3, 2025

View reviewed changes

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java Outdated Show resolved Hide resolved

src/main/java/org/opensearch/neuralsearch/query/NeuralQueryBuilder.java Show resolved Hide resolved

heemin32 reviewed Mar 3, 2025

View reviewed changes

martin-gaievski reviewed Mar 4, 2025

View reviewed changes

chloewqg force-pushed the local branch from 1cd7801 to 21f0595 Compare March 4, 2025 17:58

bzhangam reviewed Mar 4, 2025

View reviewed changes

This was referenced Mar 5, 2025

[FEATURE] Add top level filter clause to hybrid query clause to avoid duplication of filters in sub query clause of hybrid search #282

Open

[RFC] Common Filter Support for Hybrid Query Sub-Queries #1135

Open

chloewqg force-pushed the local branch from 21f0595 to 4ae66e1 Compare March 6, 2025 18:27

Add filter function for NeuralQueryBuilder and HybridQueryBuilder and…

cb7dff7

… modify fromXContent function in HybridQueryBuilder to support filter field. Signed-off-by: Chloe Gao <chloewq@amazon.com>

chloewqg force-pushed the local branch from 4ae66e1 to cb7dff7 Compare March 10, 2025 01:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filter function for NeuralQueryBuilder and HybridQueryBuilder and… #1206

Add filter function for NeuralQueryBuilder and HybridQueryBuilder and… #1206

chloewqg commented Mar 3, 2025 •

edited

Loading

heemin32 Mar 3, 2025

martin-gaievski Mar 4, 2025

heemin32 Mar 4, 2025

chloewqg Mar 4, 2025

chloewqg Mar 4, 2025

chloewqg Mar 4, 2025

chloewqg Mar 4, 2025

chloewqg Mar 4, 2025

bzhangam Mar 4, 2025

chloewqg Mar 5, 2025

martin-gaievski left a comment

martin-gaievski Mar 4, 2025

chloewqg commented Mar 4, 2025

martin-gaievski commented Mar 4, 2025

codecov bot commented Mar 4, 2025 •

edited

Loading

chloewqg commented Mar 4, 2025

bzhangam Mar 4, 2025

chloewqg Mar 5, 2025

bzhangam Mar 4, 2025

bzhangam Mar 4, 2025

martin-gaievski Mar 4, 2025

chloewqg Mar 5, 2025

chloewqg Mar 5, 2025

Add filter function for NeuralQueryBuilder and HybridQueryBuilder and… #1206

Are you sure you want to change the base?

Add filter function for NeuralQueryBuilder and HybridQueryBuilder and… #1206

Conversation

chloewqg commented Mar 3, 2025 • edited Loading

Description

Related Issues

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martin-gaievski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chloewqg commented Mar 4, 2025

martin-gaievski commented Mar 4, 2025

codecov bot commented Mar 4, 2025 • edited Loading

Codecov Report

chloewqg commented Mar 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chloewqg commented Mar 3, 2025 •

edited

Loading

codecov bot commented Mar 4, 2025 •

edited

Loading