Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local vllm support #854

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Local vllm support #854

wants to merge 3 commits into from

Conversation

Bilokin
Copy link
Contributor

@Bilokin Bilokin commented Mar 3, 2025 •
edited
Loading

Hello,

I would like to add a local vllm support to smolagents.
This is needed on servers where it is impossible to launch a vllm server instance and the performance of transformers library is limited.
Tested on CodeAgent and ToolCallingAgent.
Please tell me if there are any problems in the code, which unit-tests do I need to add and where should I put additional docs.
Closes #767
Closes #582, because vllm doesn't have this issue

@albertvillanova
Copy link
Member

Duplicate of:

@albertvillanova albertvillanova added the duplicate This issue or pull request already exists label Mar 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
duplicate This issue or pull request already exists
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Agents with vLLM framework Agent will get stuck after a few iterations if used in a loop
2 participants
@Bilokin @albertvillanova