Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Attention] Deepseek v3 MLA support with FP8 compute #12601

Merged
merged 41 commits into from
Feb 1, 2025
Merged

[Attention] Deepseek v3 MLA support with FP8 compute #12601

merged 41 commits into from
Feb 1, 2025

Conversation

LucasWilkinson
Copy link
Collaborator

Based off of: #12528 that needs to land first

LucasWilkinson and others added 21 commits January 30, 2025 16:57
@LucasWilkinson @WoosukKwon @simon-mo
squashed commits
27ad92c 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
fix VLLM_MLA_PERFORM_MATRIX_ABSORPTION=0
c34e5ca 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
more cleanups
f2cac91 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @mgoin
Update utils.py
068e672 
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @mgoin
Update vllm/attention/backends/mla/utils.py
31b802c 
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
review comments
634eee6 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
renaming for consistency
7487429 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @zhuohan123
Update vllm/config.py
d27826d 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
review comments
8bdc14a 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
review comments
09d814c 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @tlrmchlsmth
Update vllm/attention/backends/mla/utils.py
4a46014 
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
disable MLA for v3 for now
0881475 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
fix failing test
37e39f4 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
fix mypy
cfb2d26 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
fix mypy
5afc1bf 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
add cuda graph support
54ba87d 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
ci fix
31c34bf 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
Revert "add cuda graph support"
433322b 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @alexm-redhat
Fix TP > 1 cuda graphs
f2b2500 
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson @alexm-redhat
cleanup
2d61054 
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
cleanup
645622c 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
filter compressed tensor models better
db2c583 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
Merge remote-tracking branch 'origin/main' into mla-fp8
fac827f
@LucasWilkinson
simplification
5002734 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
mgoin
mgoin approved these changes Feb 1, 2025
View reviewed changes
simon-mo and others added 2 commits February 1, 2025 00:56
@simon-mo @mgoin @LucasWilkinson
Update loader.py
0d66687 
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
format
5fe1d1d 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
reduce split kv amount
5d5071c 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson
fix none type error
7ac6f52 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@mgoin mgoin mentioned this pull request Feb 1, 2025
Closed
@LucasWilkinson
ci fix
dc0e2af 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
@LucasWilkinson LucasWilkinson mentioned this pull request Feb 1, 2025
Merged
4 tasks
@simon-mo simon-mo merged commit baeded2 into vllm-project:main Feb 1, 2025
42 of 44 checks passed
yangw1234 pushed a commit to yangw1234/habana-vllm-fork that referenced this pull request Feb 2, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @yangw1234
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
2be4805 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Isotr0py pushed a commit to Isotr0py/vllm that referenced this pull request Feb 2, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @Isotr0py
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
c22f65d 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
youngkent pushed a commit to youngkent/vllm that referenced this pull request Feb 3, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @youngkent
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
f09c81a 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
srikanthsrnvs pushed a commit to srikanthsrnvs/vllm that referenced this pull request Feb 3, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @srikanthsrnvs
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
bb94260 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Signed-off-by: Srikanth Srinivas <srikanth@astrum.ai>
@Syst3m1cAn0maly Syst3m1cAn0maly mentioned this pull request Feb 3, 2025
Open
1 task
sahelib25 pushed a commit to krai/vllm that referenced this pull request Feb 3, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @sahelib25
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
06f14ab 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
xuechendi pushed a commit to yangw1234/habana-vllm-fork that referenced this pull request Feb 3, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @xuechendi
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
baf04c8 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
@houseroad houseroad mentioned this pull request Feb 4, 2025
Open
fxmarty-amd pushed a commit to fxmarty-amd/vllm that referenced this pull request Feb 7, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @fxmarty-amd
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
62c73f2 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Signed-off-by: Felix Marty <felmarty@amd.com>
NickLucche pushed a commit to NickLucche/vllm that referenced this pull request Feb 7, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @NickLucche
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
6bb84bb 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
ShangmingCai pushed a commit to ShangmingCai/vllm that referenced this pull request Feb 10, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @ShangmingCai
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
2962ca4 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
GWS0428 pushed a commit to GWS0428/VARserve that referenced this pull request Feb 12, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @GWS0428
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
bd83b50 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
gshtras
gshtras reviewed Feb 14, 2025
View reviewed changes
vllm/attention/backends/mla/utils.py
def get_scale_group_shapes_for_fp8(layer: LinearBase) -> \
Tuple[Tuple[int, int], Tuple[int, int]]:
if isinstance(layer.quant_method, Fp8LinearMethod):
if layer.quant_method.block_quant is not None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fp8LinearMethod.block_quant is a boolean, is there meant to be a check for False instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is a bug, I fixed it here #13181

panf2333 pushed a commit to yottalabsai/vllm that referenced this pull request Feb 18, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @panf2333
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
58dcea2 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
kerthcet pushed a commit to kerthcet/vllm that referenced this pull request Feb 21, 2025
@LucasWilkinson @WoosukKwon
@simon-mo @mgoin @zhuohan123 @tlrmchlsmth @alexm-redhat @kerthcet
[Attention] Deepseek v3 MLA support with FP8 compute (vllm-project#12601
86c6cbc 
)

This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
@LucasWilkinson LucasWilkinson mentioned this pull request Feb 25, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mgoin mgoin mgoin approved these changes

@simon-mo simon-mo simon-mo left review comments

@gshtras gshtras gshtras left review comments

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123

@youkaichao youkaichao Awaiting requested review from youkaichao

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat

@comaniac comaniac Awaiting requested review from comaniac

@njhill njhill Awaiting requested review from njhill

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@LucasWilkinson @robertgshaw2-redhat @simon-mo @tlrmchlsmth @mgoin @gshtras