Skip to content

Commit

Permalink
Update doc about models' TP support
Browse files Browse the repository at this point in the history
  • Loading branch information
kwen2501 committed Jan 29, 2025
1 parent b5aaf87 commit b4dd359
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions docs/source/en/perf_infer_gpu_multi.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ torchrun --nproc-per-node 4 demo.py

PyTorch tensor parallel is currently supported for the following models:
* [Llama](https://huggingface.co/docs/transformers/model_doc/llama#transformers.LlamaModel)
* [Gemma](https://huggingface.co/docs/transformers/en/model_doc/gemma), [Gemma2](https://huggingface.co/docs/transformers/en/model_doc/gemma2)
* [Granite](https://huggingface.co/docs/transformers/en/model_doc/granite)
* [Mistral](https://huggingface.co/docs/transformers/en/model_doc/mistral)
* [Qwen2](https://huggingface.co/docs/transformers/en/model_doc/qwen2), [Qwen2MoE](https://huggingface.co/docs/transformers/en/model_doc/qwen2_moe), [Qwen2-VL](https://huggingface.co/docs/transformers/v4.48.0/en/model_doc/qwen2_vl)
* [Starcoder2](https://huggingface.co/docs/transformers/en/model_doc/starcoder2)
* [Cohere](https://huggingface.co/docs/transformers/en/model_doc/cohere), [Cohere2](https://huggingface.co/docs/transformers/en/model_doc/cohere2)
* [GLM](https://huggingface.co/docs/transformers/en/model_doc/glm)
* [Mixtral](https://huggingface.co/docs/transformers/en/model_doc/mixtral)
* [OLMo](https://huggingface.co/docs/transformers/en/model_doc/olmo), [OLMo2](https://huggingface.co/docs/transformers/en/model_doc/olmo2)
* [Phi](https://huggingface.co/docs/transformers/en/model_doc/phi), [Phi-3](https://huggingface.co/docs/transformers/en/model_doc/phi3)

You can request to add tensor parallel support for another model by opening a GitHub Issue or Pull Request.

Expand Down

0 comments on commit b4dd359

Please sign in to comment.