Upgrade BWC version and fix neural search build (#868)

Signed-off-by: Varun Jain <varunudr@amazon.com>

Author: vibrantvarun

.github/workflows/backwards_compatibility_tests_workflow.yml

@@ -15,7 +15,7 @@ jobs:
       matrix:
         java: [ 21 ]
         os: [ubuntu-latest,windows-latest]
-        bwc_version : ["2.9.0","2.10.0","2.11.0","2.12.0","2.13.0","2.14.0","2.15.0","2.16.0-SNAPSHOT"]
+        bwc_version : ["2.9.0","2.10.0","2.11.0","2.12.0","2.13.0","2.14.0","2.15.0","2.16.0","2.17.0-SNAPSHOT"]
         opensearch_version : [ "3.0.0-SNAPSHOT" ]
 
     name: NeuralSearch Restart-Upgrade BWC Tests
@@ -42,7 +42,7 @@ jobs:
       matrix:
         java: [ 21 ]
         os: [ubuntu-latest,windows-latest]
-        bwc_version: [ "2.16.0-SNAPSHOT" ]
+        bwc_version: [ "2.17.0-SNAPSHOT" ]
         opensearch_version: [ "3.0.0-SNAPSHOT" ]
 
     name: NeuralSearch Rolling-Upgrade BWC Tests

TRIAGING.md

@@ -1,8 +1,8 @@
 <img src="https://opensearch.org/assets/img/opensearch-logo-themed.svg" height="64px">
 
-The maintainers of the k-NN/neural-search Repo's seek to promote an inclusive and engaged community of contributors. In 
-order to facilitate this, bi-weekly triage meetings are open-to-all and attendance is encouraged for anyone who hopes to 
-contribute, discuss an issue, or learn more about the project. To learn more about contributing to the 
+The maintainers of the k-NN/neural-search Repo's seek to promote an inclusive and engaged community of contributors. In
+order to facilitate this, bi-weekly triage meetings are open-to-all and attendance is encouraged for anyone who hopes to
+contribute, discuss an issue, or learn more about the project. To learn more about contributing to the
 k-NN/neural-search Repo visit the [Contributing](./CONTRIBUTING.md) documentation.
 
 ### Do I need to attend for my issue to be addressed/triaged?
@@ -11,19 +11,19 @@ Attendance is not required for your issue to be triaged or addressed. All new is
 
 ### What happens if my issue does not get covered this time?
 
-Each meeting we seek to  address all new issues. However, should we run out of time before your issue is discussed, you 
+Each meeting we seek to  address all new issues. However, should we run out of time before your issue is discussed, you
 are always welcome to attend the next meeting or to follow up on the issue post itself.
 
 ### How do I join the Backlog & Triage meeting?
 
-Meetings are hosted regularly at 5 PM Pacific Time on Tuesdays bi-weekly and can be joined via the links posted on the 
-[OpenSearch Meetup Group](https://www.meetup.com/opensearch/events/) list of events. The event will be titled 
+Meetings are hosted regularly at 5 PM Pacific Time on Tuesdays bi-weekly and can be joined via the links posted on the
+[OpenSearch Meetup Group](https://www.meetup.com/opensearch/events/) list of events. The event will be titled
 `Development Backlog & Triage Meeting - k-NN/neural-search`.
 
-After joining the Chime meeting, you can enable your video / voice to join the discussion.  If you do not have a webcam 
+After joining the Chime meeting, you can enable your video / voice to join the discussion.  If you do not have a webcam
 or microphone available, you can still join in via the text chat.
 
-If you have an issue you'd like to bring forth please consider getting a link to the issue so it can be presented to 
+If you have an issue you'd like to bring forth please consider getting a link to the issue so it can be presented to
 everyone in the meeting.
 
 ### Is there an agenda for each week?
@@ -48,13 +48,13 @@ No, all are welcome and encouraged to attend. Attending the Backlog & Triage mee
 
 ### What if I have an issue that is almost a duplicate, should I open a new one to be triaged?
 
-You can always open an issue including one that you think may be a duplicate. However, in cases where you believe there 
-is an important distinction to be made between an existing issue and your newly created one, you are encouraged to 
+You can always open an issue including one that you think may be a duplicate. However, in cases where you believe there
+is an important distinction to be made between an existing issue and your newly created one, you are encouraged to
 attend the triaging meeting to explain.
 
 ### What if I have follow-up questions on an issue?
 
-If you have an existing issue you would like to discuss, you can always comment on the issue itself. Alternatively, you 
+If you have an existing issue you would like to discuss, you can always comment on the issue itself. Alternatively, you
 are welcome to come to the triage meeting to discuss.
 
 ### Is this meeting a good place to get help setting up k-NN/neural-search features on my OpenSearch instance?

gradle.properties

@@ -7,8 +7,8 @@
 # https://github.com/opensearch-project/OpenSearch/blob/main/libs/core/src/main/java/org/opensearch/Version.java .
 # Wired compatibility of OpenSearch works like 3.x version is compatible with 2.(latest-major) version.
 # Therefore, to run rolling-upgrade BWC Test on local machine the BWC version here should be set 2.(latest-major).
-systemProp.bwc.version=2.16.0-SNAPSHOT
-systemProp.bwc.bundle.version=2.16.0
+systemProp.bwc.version=2.17.0-SNAPSHOT
+systemProp.bwc.bundle.version=2.17.0
 
 # For fixing Spotless check with Java 17
 org.gradle.jvmargs=--add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \

src/test/java/org/opensearch/neuralsearch/ValidateDependentPluginInstallationIT.java

@@ -75,7 +75,8 @@ private void createBasicKnnIndex() throws IOException {
             .endObject()
             .toString();
         mapping = mapping.substring(1, mapping.length() - 1);
-        createIndex(KNN_INDEX_NAME, Settings.EMPTY, mapping);
+        Settings settings = Settings.builder().put("index.knn", true).build();
+        createIndex(KNN_INDEX_NAME, settings, mapping);
     }
 
     private Set<String> getAllInstalledPlugins() throws IOException {

src/test/java/org/opensearch/neuralsearch/query/HybridQueryBuilderTests.java

@@ -17,18 +17,27 @@
 import static org.opensearch.neuralsearch.query.NeuralQueryBuilder.MODEL_ID_FIELD;
 import static org.opensearch.neuralsearch.query.NeuralQueryBuilder.QUERY_TEXT_FIELD;
 
+import java.util.HashSet;
 import java.util.Iterator;
 import java.util.List;
 import java.util.Map;
 import java.util.ArrayList;
+import java.util.Optional;
+import java.util.Set;
 import java.util.function.Supplier;
+import java.util.stream.Collectors;
 
 import org.apache.lucene.search.MatchNoDocsQuery;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.TermQuery;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
 import org.opensearch.Version;
 import org.opensearch.cluster.service.ClusterService;
 import org.opensearch.common.io.stream.BytesStreamOutput;
+import org.opensearch.common.settings.ClusterSettings;
+import org.opensearch.common.settings.Setting;
+import org.opensearch.common.settings.Settings;
 import org.opensearch.common.xcontent.XContentFactory;
 import org.opensearch.core.ParseField;
 import org.opensearch.core.common.ParsingException;
@@ -47,8 +56,13 @@
 import org.opensearch.index.query.QueryBuilders;
 import org.opensearch.index.query.QueryShardContext;
 import org.opensearch.index.query.TermQueryBuilder;
+import org.opensearch.knn.index.KNNSettings;
 import org.opensearch.knn.index.SpaceType;
 import org.opensearch.knn.index.VectorDataType;
+import org.opensearch.knn.index.engine.KNNEngine;
+import org.opensearch.knn.index.engine.KNNMethodContext;
+import org.opensearch.knn.index.engine.MethodComponentContext;
+import org.opensearch.knn.index.mapper.KNNMappingConfig;
 import org.opensearch.knn.index.mapper.KNNVectorFieldType;
 import org.opensearch.knn.index.query.KNNQuery;
 import org.opensearch.knn.index.query.KNNQueryBuilder;
@@ -69,6 +83,26 @@ public class HybridQueryBuilderTests extends OpenSearchQueryTestCase {
     static final float BOOST = 1.8f;
     static final Supplier<float[]> TEST_VECTOR_SUPPLIER = () -> new float[4];
     static final QueryBuilder TEST_FILTER = new MatchAllQueryBuilder();
+    @Mock
+    private ClusterService clusterService;
+    private AutoCloseable openMocks;
+
+    @Override
+    public void setUp() throws Exception {
+        super.setUp();
+        openMocks = MockitoAnnotations.openMocks(this);
+        // This is required to make sure that before every test we are initializing the KNNSettings. Not doing this
+        // leads to failures of unit tests cases when a unit test is run separately. Try running this test:
+        // ./gradlew ':test' --tests "org.opensearch.knn.training.TrainingJobTests.testRun_success" and see it fails
+        // but if run along with other tests this test passes.
+        initKNNSettings();
+    }
+
+    @Override
+    public void tearDown() throws Exception {
+        super.tearDown();
+        openMocks.close();
+    }
 
     @SneakyThrows
     public void testDoToQuery_whenNoSubqueries_thenBuildSuccessfully() {
@@ -86,11 +120,14 @@ public void testDoToQuery_whenOneSubquery_thenBuildSuccessfully() {
         Index dummyIndex = new Index("dummy", "dummy");
         QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
         KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
+        KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
+        KNNMethodContext knnMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
+        when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
+        when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.of(knnMethodContext));
         when(mockQueryShardContext.index()).thenReturn(dummyIndex);
-        when(mockKNNVectorField.getDimension()).thenReturn(4);
+        when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
         when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
         when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
-        when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);
 
         NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
             .queryText(QUERY_TEXT)
@@ -116,10 +153,13 @@ public void testDoToQuery_whenMultipleSubqueries_thenBuildSuccessfully() {
         Index dummyIndex = new Index("dummy", "dummy");
         QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
         KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
+        KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
+        KNNMethodContext knnMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
+        when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
+        when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.of(knnMethodContext));
         when(mockQueryShardContext.index()).thenReturn(dummyIndex);
-        when(mockKNNVectorField.getDimension()).thenReturn(4);
+        when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
         when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
-        when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);
         when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
 
         NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
@@ -367,8 +407,10 @@ public void testToXContent_whenIncomingJsonIsCorrect_thenSuccessful() {
         Index dummyIndex = new Index("dummy", "dummy");
         QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
         KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
+        KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
+        when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
         when(mockQueryShardContext.index()).thenReturn(dummyIndex);
-        when(mockKNNVectorField.getDimension()).thenReturn(4);
+        when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
         when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
 
         NeuralQueryBuilder neuralQueryBuilder = new NeuralQueryBuilder().fieldName(VECTOR_FIELD_NAME)
@@ -584,9 +626,11 @@ public void testRewrite_whenMultipleSubQueries_thenReturnBuilderForEachSubQuery(
 
         QueryShardContext mockQueryShardContext = mock(QueryShardContext.class);
         KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
+        KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
+        when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
         Index dummyIndex = new Index("dummy", "dummy");
         when(mockQueryShardContext.index()).thenReturn(dummyIndex);
-        when(mockKNNVectorField.getDimension()).thenReturn(4);
+        when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
         when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
 
         TextFieldMapper.TextFieldType fieldType = (TextFieldMapper.TextFieldType) createMapperService().fieldType(TEXT_FIELD_NAME);
@@ -737,4 +781,17 @@ private void setUpClusterService() {
         ClusterService clusterService = NeuralSearchClusterTestUtils.mockClusterService(Version.CURRENT);
         NeuralSearchClusterUtil.instance().initialize(clusterService);
     }
+
+    private void initKNNSettings() {
+        Set<Setting<?>> defaultClusterSettings = new HashSet<>(ClusterSettings.BUILT_IN_CLUSTER_SETTINGS);
+        defaultClusterSettings.addAll(
+            KNNSettings.state()
+                .getSettings()
+                .stream()
+                .filter(s -> s.getProperties().contains(Setting.Property.NodeScope))
+                .collect(Collectors.toList())
+        );
+        when(clusterService.getClusterSettings()).thenReturn(new ClusterSettings(Settings.EMPTY, defaultClusterSettings));
+        KNNSettings.state().setClusterService(clusterService);
+    }
 }

src/test/java/org/opensearch/neuralsearch/query/HybridQueryTests.java

@@ -13,8 +13,11 @@
 
 import java.io.IOException;
 import java.util.Arrays;
+import java.util.HashSet;
 import java.util.Iterator;
 import java.util.List;
+import java.util.Optional;
+import java.util.Set;
 import java.util.stream.Collectors;
 
 import org.apache.lucene.document.FieldType;
@@ -33,13 +36,24 @@
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.tests.analysis.MockAnalyzer;
 import org.apache.lucene.tests.search.QueryUtils;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+import org.opensearch.cluster.service.ClusterService;
+import org.opensearch.common.settings.ClusterSettings;
+import org.opensearch.common.settings.Setting;
+import org.opensearch.common.settings.Settings;
 import org.opensearch.core.index.Index;
 import org.opensearch.index.mapper.TextFieldMapper;
 import org.opensearch.index.query.BoolQueryBuilder;
 import org.opensearch.index.query.QueryBuilders;
 import org.opensearch.index.query.QueryShardContext;
+import org.opensearch.knn.index.KNNSettings;
 import org.opensearch.knn.index.SpaceType;
 import org.opensearch.knn.index.VectorDataType;
+import org.opensearch.knn.index.engine.KNNEngine;
+import org.opensearch.knn.index.engine.KNNMethodContext;
+import org.opensearch.knn.index.engine.MethodComponentContext;
+import org.opensearch.knn.index.mapper.KNNMappingConfig;
 import org.opensearch.knn.index.mapper.KNNVectorFieldType;
 import org.opensearch.knn.index.query.KNNQueryBuilder;
 
@@ -54,6 +68,26 @@ public class HybridQueryTests extends OpenSearchQueryTestCase {
     static final String TERM_ANOTHER_QUERY_TEXT = "anotherkeyword";
     static final float[] VECTOR_QUERY = new float[] { 1.0f, 2.0f, 2.1f, 0.6f };
     static final int K = 2;
+    @Mock
+    protected ClusterService clusterService;
+    private AutoCloseable openMocks;
+
+    @Override
+    public void setUp() throws Exception {
+        super.setUp();
+        openMocks = MockitoAnnotations.openMocks(this);
+        // This is required to make sure that before every test we are initializing the KNNSettings. Not doing this
+        // leads to failures of unit tests cases when a unit test is run separately. Try running this test:
+        // ./gradlew ':test' --tests "org.opensearch.knn.training.TrainingJobTests.testRun_success" and see it fails
+        // but if run along with other tests this test passes.
+        initKNNSettings();
+    }
+
+    @Override
+    public void tearDown() throws Exception {
+        super.tearDown();
+        openMocks.close();
+    }
 
     @SneakyThrows
     public void testQueryBasics_whenMultipleDifferentQueries_thenSuccessful() {
@@ -116,10 +150,13 @@ public void testRewrite_whenRewriteQuery_thenSuccessful() {
 
         Index dummyIndex = new Index("dummy", "dummy");
         KNNVectorFieldType mockKNNVectorField = mock(KNNVectorFieldType.class);
+        KNNMappingConfig mockKNNMappingConfig = mock(KNNMappingConfig.class);
+        KNNMethodContext mockKNNMethodContext = new KNNMethodContext(KNNEngine.FAISS, SpaceType.L2, MethodComponentContext.EMPTY);
+        when(mockKNNVectorField.getKnnMappingConfig()).thenReturn(mockKNNMappingConfig);
+        when(mockKNNMappingConfig.getKnnMethodContext()).thenReturn(Optional.ofNullable(mockKNNMethodContext));
         when(mockQueryShardContext.index()).thenReturn(dummyIndex);
-        when(mockKNNVectorField.getDimension()).thenReturn(4);
+        when(mockKNNVectorField.getKnnMappingConfig().getDimension()).thenReturn(4);
         when(mockQueryShardContext.fieldMapper(eq(VECTOR_FIELD_NAME))).thenReturn(mockKNNVectorField);
-        when(mockKNNVectorField.getSpaceType()).thenReturn(SpaceType.L2);
         when(mockKNNVectorField.getVectorDataType()).thenReturn(VectorDataType.FLOAT);
         KNNQueryBuilder knnQueryBuilder = new KNNQueryBuilder(VECTOR_FIELD_NAME, VECTOR_QUERY, K);
         Query knnQuery = knnQueryBuilder.toQuery(mockQueryShardContext);
@@ -316,4 +353,17 @@ public void testFilter_whenSubQueriesWithFilterPassed_thenSuccessful() {
         }
         assertEquals(2, countOfQueries);
     }
+
+    private void initKNNSettings() {
+        Set<Setting<?>> defaultClusterSettings = new HashSet<>(ClusterSettings.BUILT_IN_CLUSTER_SETTINGS);
+        defaultClusterSettings.addAll(
+            KNNSettings.state()
+                .getSettings()
+                .stream()
+                .filter(s -> s.getProperties().contains(Setting.Property.NodeScope))
+                .collect(Collectors.toList())
+        );
+        when(clusterService.getClusterSettings()).thenReturn(new ClusterSettings(Settings.EMPTY, defaultClusterSettings));
+        KNNSettings.state().setClusterService(clusterService);
+    }
 }