Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add --mmap in llama-bench #5257

Merged
merged 7 commits into from
Feb 1, 2024
Merged

add --mmap in llama-bench #5257

merged 7 commits into from
Feb 1, 2024

Conversation

NeoZhangJianyu
Copy link
Collaborator

add --no-mmap: in some case, mmap will lead to hang issue on SYCL backend.
show "SYCL" backend in result table.
update get_gpu_info() for SYCL.

slaren
slaren reviewed Feb 1, 2024
View reviewed changes
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
examples/llama-bench/llama-bench.cpp Outdated Show resolved Hide resolved
@NeoZhangJianyu
Copy link
Collaborator Author

@airMeng , @luoyu-intel , @abhilash1910, @ggerganov Invite you to review if you are idle.

slaren
slaren requested changes Feb 1, 2024
View reviewed changes
Copy link
Collaborator

@slaren slaren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Change the mmap parameter name from no-mmap to mmap. Set it by default to 1.
  • Allow multiple values
  • Print the value in the markdown printer if it is not the default.

@NeoZhangJianyu
Copy link
Collaborator Author

  • Change the mmap parameter name from no-mmap to mmap. Set it by default to 1.
  • Allow multiple values
  • Print the value in the markdown printer if it is not the default.

OK, I will update according to comments.

@NeoZhangJianyu NeoZhangJianyu requested a review from slaren February 1, 2024 16:03
@slaren
Copy link
Collaborator

slaren commented Feb 1, 2024

The changes to llama-bench are good, should we merge this now, or wait for a review of the ggml-sycl changes?

@slaren slaren merged commit 128dcbd into ggerganov:master Feb 1, 2024
53 checks passed
@NeoZhangJianyu NeoZhangJianyu deleted the update_bench branch February 2, 2024 01:04
@NeoZhangJianyu NeoZhangJianyu changed the title add --no-mmap in llama-bench add --mmap in llama-bench Feb 2, 2024
jordankanter pushed a commit to jordankanter/llama.cpp that referenced this pull request Feb 3, 2024
@NeoZhangJianyu @jordankanter
add --no-mmap in llama-bench (ggerganov#5257)
5451395 
* add --no-mmap, show sycl backend

* fix conflict

* fix code format, change print for --no-mmap

* ren no_mmap to mmap, show mmap when not default value in printer

* update guide for mmap

* mv position to reduce model reload
hodlen pushed a commit to hodlen/llama.cpp that referenced this pull request Apr 1, 2024
@NeoZhangJianyu @hodlen
add --no-mmap in llama-bench (ggerganov#5257)
c4f4bff 
* add --no-mmap, show sycl backend

* fix conflict

* fix code format, change print for --no-mmap

* ren no_mmap to mmap, show mmap when not default value in printer

* update guide for mmap

* mv position to reduce model reload
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slaren slaren slaren approved these changes

@abhilash1910 abhilash1910 abhilash1910 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@NeoZhangJianyu @slaren @abhilash1910