[FEATURE] Enables /score endpoint for embedding models #12846

gmarinho2 · 2025-02-06T18:26:40Z

Enables /score endpoint for all embedding models via API.

Issue: [FEATURE] Enables offline /score for embedding models (2/2)

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

github-actions · 2025-02-06T18:26:51Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

As stated in the PR that introduced the /rerank API, the code was 90% the same. With this refactoring it also becomes possible to use the /rerank API with embedding models Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

maxdebayser · 2025-02-14T13:43:29Z

@DarkLight1337 and @K-Mistele , this PR is ready for review. The changes are:

Refactor OpenAIServingScores to enable the /score endpoint for embedding models. In this case the cosine similarity between the embedding pairs is returned.
Reuse OpenAIServingScores to also serve the /rerank endpoints. As a result of this refactoring, the /rerank endpoint now also support embedding models.
Improve /score unit tests to compare the results with the reference implementation (sentence-transformers)
Add an embedding model to the /score unit tests
Update the documentation.

DarkLight1337

Thanks for adding this! Some initial comments.

vllm/entrypoints/openai/serving_score.py

tests/entrypoints/openai/test_rerank.py

vllm/entrypoints/openai/serving_score.py

vllm/entrypoints/openai/api_server.py

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

DarkLight1337

Thanks, LGTM now. @K-Mistele can you take another pass as this as well?

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

gmarinho2 · 2025-02-20T12:41:56Z

@joerunde

K-Mistele · 2025-02-20T17:45:11Z

Thanks for the ping @DarkLight1337 - looks good to me.

DarkLight1337 · 2025-02-21T06:09:41Z

Since main is broken, I'm force-merging this - test_score and test_rerank passed prior to the latest merge.

…12846)

…12846) Signed-off-by: Michael Glass <mrglass@us.ibm.com>

gmarinho2 and others added 18 commits January 21, 2025 11:18

[FEATURE] Enables offline /score for embedding models

9b0ce65

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Merge branch 'upstream_main'

74cd2dd

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Changes variable name and uses tuple instead of list.

590ab4d

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Separates scoring logic and makes small ajustments

f219024

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Separates scoring logic and makes small ajustments

47211e5

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Moves scoring functions declaration to llm class

1a89033

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Completes embedding_score function signature

db7919b

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Passes new parameters to self.encode() in embeddind_score

41b11be

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Adds type annotations for the parameters

b57e01a

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Minor adjustments

4956eae

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Minor adjustments in embedding_score

f8b8d8c

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

trigger ci

cd835de

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

trigger ci

0e95ead

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Merge branch 'vllm-project:main' into main

bba2ea6

Merge branch 'vllm-project:main' into main

635b8e8

first implementation of embedding scores via api

7851b44

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

second version of api scoring

12383b2

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

makes separate functions for cross-encoder score and embedding score

12ef932

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

mergify bot added the frontend label Feb 6, 2025

fixes pre-commit errors

945280f

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

gmarinho2 force-pushed the scoring-openai branch from 6d9077b to 945280f Compare February 7, 2025 13:49

adapts union sintax to python 3.9

053bdbb

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

gmarinho2 force-pushed the scoring-openai branch 3 times, most recently from 3599577 to 8b0b41f Compare February 7, 2025 18:53

fixes alternating response bug

cfbc4b9

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

gmarinho2 force-pushed the scoring-openai branch 2 times, most recently from c4762c7 to 2d44144 Compare February 7, 2025 23:53

fixes pre-commit errors

a9b3a0d

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

gmarinho2 requested review from DarkLight1337, robertgshaw2-redhat and simon-mo as code owners February 13, 2025 16:07

adds documentation

7e9478a

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

mergify bot added the documentation Improvements or additions to documentation label Feb 13, 2025

maxdebayser and others added 4 commits February 13, 2025 14:20

Refactor /rerank to reuse code from /score

056f1be

As stated in the PR that introduced the /rerank API, the code was 90% the same. With this refactoring it also becomes possible to use the /rerank API with embedding models Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Merge branch 'max-scoring-openai' into scoring-openai

6a6e45b

fixes union syntax

5c0495d

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

adds documentation

7b2891b

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

DarkLight1337 reviewed Feb 14, 2025

View reviewed changes

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

K-Mistele reviewed Feb 14, 2025

View reviewed changes

tests/entrypoints/openai/test_rerank.py Show resolved Hide resolved

vllm/entrypoints/openai/serving_score.py Outdated Show resolved Hide resolved

vllm/entrypoints/openai/api_server.py Outdated Show resolved Hide resolved

gmarinho2 added 2 commits February 18, 2025 09:48

fixing mypy errors and refactoring

6218cc3

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

Puts embedding score code in score_utils to avoid duplicated code

5b54495

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

gmarinho2 force-pushed the scoring-openai branch from 6492787 to 5b54495 Compare February 18, 2025 14:18

changes variable name for clarity

17a2960

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

DarkLight1337 approved these changes Feb 18, 2025

View reviewed changes

gmarinho2 and others added 3 commits February 18, 2025 12:51

refactoring serving_score

2a082f3

Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>

factor out common code

077cbae

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

remove extra code lines

1fa73a8

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Merge branch 'vllm-project:main' into scoring-openai

c9a7240

DarkLight1337 enabled auto-merge (squash) February 21, 2025 03:36

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 21, 2025

simon-mo merged commit 1c3c975 into vllm-project:main Feb 21, 2025
40 of 58 checks passed

JenZhao pushed a commit to JenZhao/vllm that referenced this pull request Feb 21, 2025

[FEATURE] Enables /score endpoint for embedding models (vllm-project#…

6415b9d

…12846)

gmarinho2 deleted the scoring-openai branch February 21, 2025 18:17

michaelrglass pushed a commit to michaelrglass/vllm that referenced this pull request Feb 21, 2025

[FEATURE] Enables /score endpoint for embedding models (vllm-project#…

cda36bc

…12846) Signed-off-by: Michael Glass <mrglass@us.ibm.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Enables /score endpoint for embedding models #12846

[FEATURE] Enables /score endpoint for embedding models #12846

gmarinho2 commented Feb 6, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Feb 6, 2025

maxdebayser commented Feb 14, 2025

DarkLight1337 left a comment

DarkLight1337 left a comment

gmarinho2 commented Feb 20, 2025

K-Mistele commented Feb 20, 2025

DarkLight1337 commented Feb 21, 2025

[FEATURE] Enables /score endpoint for embedding models #12846

[FEATURE] Enables /score endpoint for embedding models #12846

Conversation

gmarinho2 commented Feb 6, 2025 • edited by github-actions bot Loading

github-actions bot commented Feb 6, 2025

maxdebayser commented Feb 14, 2025

DarkLight1337 left a comment

Choose a reason for hiding this comment

DarkLight1337 left a comment

Choose a reason for hiding this comment

gmarinho2 commented Feb 20, 2025

K-Mistele commented Feb 20, 2025

DarkLight1337 commented Feb 21, 2025

gmarinho2 commented Feb 6, 2025 •

edited by github-actions bot

Loading