Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Paged attention new #82

Merged
merged 62 commits into from
Jan 31, 2025
Merged

Paged attention new #82

merged 62 commits into from
Jan 31, 2025

Conversation

goliaro
Copy link
Contributor

@goliaro goliaro commented Jan 9, 2025

Description of changes:

Related Issues:

Linked Issues:

  • Issue #

Issues closed by this PR:

  • Closes #

Bob-Chen222 and others added 30 commits October 3, 2024 15:51
@Bob-Chen222
add page_manager and request_manager functions
71d8a7b
@Bob-Chen222
add batch_config
0eaca39
@Bob-Chen222
request manager h and request manger cc to be continued
151872f
@chenzhuofu
Merge remote-tracking branch 'origin/specscheduler' into paged_attent…
904364d 
…ion_new
@chenzhuofu
refactored the interface of block manager but may not be bug free
e3abef8
@chenzhuofu
ckpt before build
73dc699
@Bob-Chen222
some fix
de0b803
@Bob-Chen222
ready for sanity check
0e405c1
@Bob-Chen222
Merge remote-tracking branch 'origin/specscheduler' into paged_attent…
dec2266 
…ion_new
@Bob-Chen222
fix last commit index
6b4777e
@Bob-Chen222
fix request id error
8394f15
@chenzhuofu
fix spec token num
2ec8b5b
@chenzhuofu
fix small error in free_multiple_blocks
b12df8c
@Bob-Chen222
ckpt single request
6298f2a
@Bob-Chen222
add cleanup
c00ddec
@Bob-Chen222
ckpt before index error in prepare_parameters
b1ff323
@Bob-Chen222
fix token error in prepare_batch_config
8a3975a
@Bob-Chen222
ckpt, something wrong in the prefilling
f4e73ea
@Bob-Chen222
ckpt
4eeb021
@Bob-Chen222
Merge remote-tracking branch 'origin/specscheduler' into paged_attent…
12fafa3 
…ion_new
@Bob-Chen222
update
3ad0ca5
@Bob-Chen222
Merge remote-tracking branch 'origin/specscheduler' into paged_attent…
945dee9 
…ion_new
@Bob-Chen222
add some docuementation and delete print
19e41d6
@Bob-Chen222
add additional flag max-kv-cache-size
b1793fb
@Bob-Chen222
Merge remote-tracking branch 'origin/specscheduler' into paged_attent…
fd65a90 
…ion_new
@Bob-Chen222
fix for merge
832f5cb
@Bob-Chen222
init page manager at request manager init and clean the format
4a7162f
@Bob-Chen222
ckpt
6b74f93
@Bob-Chen222
refactor and add kv cache flag via page manager
20cb714
@Bob-Chen222
ckpt for performance issue
311c450
@lockshaw lockshaw mentioned this pull request Jan 9, 2025
Closed
@chenzhuofu chenzhuofu changed the base branch from inference to specscheduler January 17, 2025 16:11
@chenzhuofu
Copy link
Collaborator

Now the pending bugs are all fixed! You could check if page manager design makes sense. @jiazhihao @zikun-li

@chenzhuofu chenzhuofu marked this pull request as ready for review January 27, 2025 08:53
@chenzhuofu chenzhuofu merged commit 918356d into specscheduler Jan 31, 2025
26 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jiazhihao jiazhihao Awaiting requested review from jiazhihao

@zikun-li zikun-li Awaiting requested review from zikun-li

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@goliaro @chenzhuofu @Bob-Chen222