Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic Vulkan Multi-GPU implementation #5321

Merged
merged 8 commits into from
Feb 7, 2024
Merged

Basic Vulkan Multi-GPU implementation #5321

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ca8110c
Initial Vulkan multi-gpu implementation
0cc4m Feb 4, 2024
5a1ad8c
Add names to backend device functions
0cc4m Feb 4, 2024
a1f9c00
Add further missing cleanup code
0cc4m Feb 4, 2024
c71316f
Reduce code duplication in tensor split layer assignment
0cc4m Feb 4, 2024
daa6a9c
generalize LLAMA_SPLIT_LAYER for all backends, do not expose device c…
slaren Feb 5, 2024
087ae64
Only do device info print in the beginning and initialize one backend…
0cc4m Feb 5, 2024
fd8351b
Rework backend memory management to make sure devices and buffers get…
0cc4m Feb 6, 2024
b22e925
Rename cpu assist free function
0cc4m Feb 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions common/common.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,10 @@
#define GGML_USE_CUBLAS_SYCL
#endif

#if (defined(GGML_USE_CUBLAS) || defined(GGML_USE_SYCL)) || defined(GGML_USE_VULKAN)
#define GGML_USE_CUBLAS_SYCL_VULKAN
#endif

int32_t get_num_physical_cores() {
#ifdef __linux__
// enumerate the set of thread siblings, num entries is num cores
Expand Down Expand Up @@ -648,8 +652,8 @@ bool gpt_params_parse_ex(int argc, char ** argv, gpt_params & params) {
params.tensor_split[i] = 0.0f;
}
}
#ifndef GGML_USE_CUBLAS_SYCL
fprintf(stderr, "warning: llama.cpp was compiled without cuBLAS/SYCL. Setting a tensor split has no effect.\n");
#ifndef GGML_USE_CUBLAS_SYCL_VULKAN
fprintf(stderr, "warning: llama.cpp was compiled without cuBLAS/SYCL/Vulkan. Setting a tensor split has no effect.\n");
#endif // GGML_USE_CUBLAS_SYCL
} else if (arg == "--no-mmap") {
params.use_mmap = false;
Expand Down
Loading
Loading