Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add example for multi-host serving of llama 70B on GPUs #124

Closed
Tracked by #159
ahg-g opened this issue Apr 29, 2024 · 2 comments · Fixed by #133
Closed
Tracked by #159

Add example for multi-host serving of llama 70B on GPUs #124

ahg-g opened this issue Apr 29, 2024 · 2 comments · Fixed by #133
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@ahg-g
Copy link
Contributor

ahg-g commented Apr 29, 2024

Add an example for running llama2 70B on GPUs using LWS; we can use the model server that comes with llama itself see meta-llama/llama#594
@ahg-g ahg-g added the kind/feature Categorizes issue or PR as related to a new feature. label Apr 29, 2024
@liurupeng liurupeng mentioned this issue Apr 30, 2024
Closed
7 tasks
@Edwinhr716
Copy link
Contributor

/assign

@Edwinhr716 Edwinhr716 removed their assignment May 2, 2024
@ahg-g
Copy link
Contributor Author

ahg-g commented May 5, 2024

Another one from vllm: https://docs.vllm.ai/en/latest/serving/distributed_serving.html

@gujingit gujingit mentioned this issue May 8, 2024
Merged
@liurupeng liurupeng mentioned this issue Jun 4, 2024
Closed
20 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add vllm distributed inference examples gujingit/lws
2 participants
@ahg-g @Edwinhr716