Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remote Vector Index Build] Introduce RemoteIndexClient skeleton and Build Request construction #2560

Merged
merged 18 commits into from
Mar 4, 2025
Merged

[Remote Vector Index Build] Introduce RemoteIndexClient skeleton and Build Request construction #2560

merged 18 commits into from
Mar 4, 2025

Conversation

owenhalpert
Copy link
Contributor

@owenhalpert owenhalpert commented Feb 25, 2025 •
edited
Loading

Description

First PR for #2518 (Client Skeleton, Build Request)

This PR introduces the initial implementation of the Remote Index client.
In this PR, I have implemented:

  • Client interface + HTTP implementation skeleton, showing overview of the client creation, code flow, etc
    • Updated code flow in RemoteIndexBuildStrategy based on finalized method parameters/return types
  • Programmatic build request construction
  • All client-related settings
  • Extendable RemoteBuildRequest interface and class
  • Client retry strategy
  • Unit tests

In the next PR's, I will implement:

  • Polling mechanism
  • Metric collection
  • Complete unit testing

Related Issues

#2391 Meta Issue
#2393 HLD by @jed326
#2518 My LLD

Check List

  • New functionality includes testing.
    - [ ] New functionality has been documented.
    - [ ] API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
    - [ ] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@owenhalpert owenhalpert changed the title Introduce RemoteIndexClient [Remote Vector Index Build] Introduce RemoteIndexClient Feb 25, 2025
@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch from 3fb30f9 to 2095bba Compare February 25, 2025 21:16
@owenhalpert owenhalpert marked this pull request as ready for review February 25, 2025 21:16
@jed326 jed326 mentioned this pull request Feb 25, 2025
Open
19 tasks
jed326
jed326 reviewed Feb 25, 2025
View reviewed changes
@jed326 jed326 added the v3.0.0 label Feb 25, 2025
@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch 2 times, most recently from 8d43195 to 1f8b36d Compare February 26, 2025 01:27
jed326
jed326 reviewed Feb 26, 2025
View reviewed changes
Copy link
Contributor

@jed326 jed326 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @owenhalpert , the approach here LGTM, think we're just missing some tests related to submitVectorBuild and the settings.

@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch 4 times, most recently from 7076818 to a8a4400 Compare February 26, 2025 23:05
@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch from 2b069b8 to 5176e70 Compare February 27, 2025 05:11
owenhalpert added 17 commits March 3, 2025 18:47
@owenhalpert
Introduce RemoteIndexClient
847698c 
Add RemoteIndexClient initial implementation, its accompanying dependencies, and Build Request, Retry Strategy, and test files

Signed-off-by: owenhalpert <ohalpert@gmail.com>

# Conflicts:
#	CHANGELOG.md
#	src/main/java/org/opensearch/knn/index/KNNSettings.java
#	src/main/java/org/opensearch/knn/index/codec/nativeindex/NativeIndexBuildStrategyFactory.java
#	src/main/java/org/opensearch/knn/index/codec/nativeindex/remote/RemoteIndexBuildStrategy.java
#	src/test/java/org/opensearch/knn/index/codec/nativeindex/remote/RemoteIndexBuildStrategyTests.java

# Conflicts:
#	CHANGELOG.md
#	src/main/java/org/opensearch/knn/index/codec/nativeindex/remote/RemoteIndexBuildStrategy.java

# Conflicts:
#	CHANGELOG.md
@owenhalpert
Initial revisions
bcad396 
Extract interface, remove out of scope methods, move build request to client, refactors

Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Add more build request work
2da5883 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Add more build request work, Javadoc
160793b 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Add submitVectorBuild unit test
6829a05 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Remove authentication until further testing
edec7d3 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Add dimension
335d99e 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Remove auth settings for now
e3db43b 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Add secure settings, access control logic for http request
deaeba4 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Test full build request
e77a835 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Remove fp16 vs. fp32 logic, other initial revisions
0731388 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Refactor, add remote build request interface, complete FAISS request …
7b78062 
…parameters, secure setting testing

Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Drop Jackson dependency for OpenSearch XContent
158004c 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Refactoring
1cee707 
Refactor, further delineate responsibilities of client vs. build request, add more abstraction

Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Move common constants to Remote constants class, simplify RemoteBuild…
82a50cb 
…Request using serialization logic (XContent)

Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Rehaul index parameter logic, minor edits, settings change
aef03d0 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert
Remove TODO and test helper through caller
3e5746e 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch 5 times, most recently from 1dc54cd to 6db364c Compare March 4, 2025 06:42
@owenhalpert
Add new unit tests, move some tests out of HTTPClient tests
75b1afa 
Signed-off-by: owenhalpert <ohalpert@gmail.com>
@owenhalpert owenhalpert force-pushed the vector-client-skeleton branch from 6db364c to 75b1afa Compare March 4, 2025 06:42
jmazanec15
jmazanec15 approved these changes Mar 4, 2025
View reviewed changes
Copy link
Member

@jmazanec15 jmazanec15 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jmazanec15 jmazanec15 merged commit b77b6b6 into opensearch-project:main Mar 4, 2025
35 checks passed
This was referenced Mar 4, 2025
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@navneet1v navneet1v navneet1v approved these changes

@jmazanec15 jmazanec15 jmazanec15 approved these changes

@jed326 jed326 jed326 approved these changes

@Vikasht34 Vikasht34 Vikasht34 left review comments

@heemin32 heemin32 Awaiting requested review from heemin32 heemin32 is a code owner

@VijayanB VijayanB Awaiting requested review from VijayanB VijayanB is a code owner

@vamshin vamshin Awaiting requested review from vamshin vamshin is a code owner

@naveentatikonda naveentatikonda Awaiting requested review from naveentatikonda naveentatikonda is a code owner

@junqiu-lei junqiu-lei Awaiting requested review from junqiu-lei junqiu-lei is a code owner

@martin-gaievski martin-gaievski Awaiting requested review from martin-gaievski martin-gaievski is a code owner

@ryanbogan ryanbogan Awaiting requested review from ryanbogan ryanbogan is a code owner

@luyuncheng luyuncheng Awaiting requested review from luyuncheng luyuncheng is a code owner

@shatejas shatejas Awaiting requested review from shatejas shatejas is a code owner

@0ctopus13prime 0ctopus13prime Awaiting requested review from 0ctopus13prime 0ctopus13prime is a code owner

Assignees
No one assigned
Labels
v3.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@owenhalpert @navneet1v @jmazanec15 @jed326 @Vikasht34