Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h #5377

bmtwl · 2024-02-06T22:50:30Z

Attempt number two.

Removed sched.h from ggml.h, moved ggml_get_numa_affinity out of the public API and purely into ggml.c, removed trailing whitespace and fixed up a few inconsistent variables

More info: #5358 (comment)

… for a new mirror mode that will require numa.h

…, removed trailing whitespace and fixed up a few inconsistent variables

… for a new mirror mode that will require numa.h

ggml.h

llama.h

cebtenzzre · 2024-02-07T18:59:33Z

If mirror mode isn't implemented yet, the user should be shown a warning or error if they try to use it - "Mirror Mode Enabled" doesn't communicate that.

bmtwl · 2024-02-07T19:13:15Z

If mirror mode isn't implemented yet, the user should be shown a warning or error if they try to use it - "Mirror Mode Enabled" doesn't communicate that.

I figured that hiding it behind the #ifdef would be enough, but I can add a warning in for sure

…ies. Added a note about mirror mode note being implemented yet

cebtenzzre · 2024-02-07T20:03:55Z

I figured that hiding it behind the #ifdef would be enough, but I can add a warning in for sure

If there is currently no use for #defining GGML_NUMA_MIRROR, then the code that depends on it shouldn't be committed yet.

examples/server/server.cpp

ggml.c

…finity and making it static

bmtwl · 2024-02-08T17:04:52Z

I have fixed the errors in the last test. I also fixed a few related errors in the "examples" folder
All tests now pass in both make and cmake :

100% tests passed, 0 tests failed out of 22

common/common.cpp

ggml.c

…blem with master

bmtwl · 2024-02-14T18:19:29Z

I'm currently installing VS on a Windows box to do local regression testing and clear up these errors before requesting this be re-run

bmtwl · 2024-02-14T23:54:09Z

I'm trying to troubleshoot the build errors on Android and Vulkan under Windows.
I have a Windows build env going, so I'm hopeful I can get to the bottom of that, but the error on Android seems to point to a lingering ggml_backend_init(bool numa) that I can't find in the codebase for the life of me, and I don't have android to test against (or a cross-compiling environment set up)
Can someone with more experience on the automatic regression testing tools or the Android dev stuff help me to troubleshoot and get the final tests green?

slaren · 2024-02-15T00:05:28Z

The Android example fetches llama.cpp from the master branch, so it breaks when the API changes, you can ignore that error.

llama.cpp/examples/llama.android/app/src/main/cpp/CMakeLists.txt

Lines 16 to 20 in 594fca3

    
           FetchContent_Declare( 
        
                   llama 
        
                   GIT_REPOSITORY https://github.com/ggerganov/llama.cpp 
        
                   GIT_TAG        master 
        
           )

The Windows error also seems unrelated to this PR, the Vulkan build is broken at the moment.

bmtwl · 2024-02-15T00:14:30Z

Thanks @slaren
Is there anything else for me to do before this is committed?

slaren · 2024-02-15T00:16:02Z

Looks good to me, but let's wait for @ggerganov review.

llama.h

ggml.h

ggml.c

examples/server/server.cpp

common/common.cpp

Align enum values Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Remove whitespace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

align paremeters Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…p example

ggml.c

simplified return for platforms without NUMA support Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

bmtwl · 2024-02-15T20:01:49Z

I've made the final proposed code changes, brought the branch in to sync with current, built and run the regression tests locally on both Linux and Windows.

common/common.cpp

@rankaiyx

* Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h * Reverted Makefile * Fixed include * Removed sched.h from ggml.h, moved ggml_get_numa_affinity into ggml.c, removed trailing whitespace and fixed up a few inconsistent variables * removed trailing whitespace * Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h * Reverting Makefile * Fixed a number of issues with the move from BOOL to ggml_numa_strategies. Added a note about mirror mode note being implemented yet * Removing MIRROR_MODE code for this PR * Removing last bit of MIRROR_MODE code for this PR * Removing unneeded branch in server.cpp example and moving get_numa_affinity and making it static * Fixed lingering init_llama_backend() bool calls in tests and examples * Remote enum llama_numa_strategies * Revert bad merge with dynatemp flags * add missing enum ggml_numa_strategies declaration and revert sync problem with master * add missing enum ggml_numa_strategies declaration * fixed ggml_init_numa variable * Update ggml.h Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> * Update READMEs with info about numa flags, change INTERLEAVE strategy name to DISTRIBUTE everywhere, implement the improved distribution strategy from @rankaiyx, fix a spelling mistake and un-merge some bad merges * split numa init out from llama_backend_init and created llama_numa_init. Updated all code paths and samples * Fix up some boolean vs enum comparisons * Added #ifdefs for non-Linux OS that don't have cpu_set_t datatype * Update ggml.h Align enum values Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update ggml.c Remove whitespace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update ggml.c align paremeters Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update examples/server/server.cpp remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update common/common.cpp Remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * unified ggml_numa_strategy enum and fixed text alignment in server.cpp example * Update ggml.c simplified return for platforms without NUMA support Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> * removed redundant else from cli argument processing of --numa * whitespace --------- Co-authored-by: root <root@nenya.lothlorien.ca> Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Jared Van Bortel <jared@nomic.ai>

@rankaiyx

* Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h * Reverted Makefile * Fixed include * Removed sched.h from ggml.h, moved ggml_get_numa_affinity into ggml.c, removed trailing whitespace and fixed up a few inconsistent variables * removed trailing whitespace * Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h * Reverting Makefile * Fixed a number of issues with the move from BOOL to ggml_numa_strategies. Added a note about mirror mode note being implemented yet * Removing MIRROR_MODE code for this PR * Removing last bit of MIRROR_MODE code for this PR * Removing unneeded branch in server.cpp example and moving get_numa_affinity and making it static * Fixed lingering init_llama_backend() bool calls in tests and examples * Remote enum llama_numa_strategies * Revert bad merge with dynatemp flags * add missing enum ggml_numa_strategies declaration and revert sync problem with master * add missing enum ggml_numa_strategies declaration * fixed ggml_init_numa variable * Update ggml.h Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> * Update READMEs with info about numa flags, change INTERLEAVE strategy name to DISTRIBUTE everywhere, implement the improved distribution strategy from @rankaiyx, fix a spelling mistake and un-merge some bad merges * split numa init out from llama_backend_init and created llama_numa_init. Updated all code paths and samples * Fix up some boolean vs enum comparisons * Added #ifdefs for non-Linux OS that don't have cpu_set_t datatype * Update ggml.h Align enum values Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update ggml.c Remove whitespace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update ggml.c align paremeters Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update examples/server/server.cpp remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update common/common.cpp Remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * unified ggml_numa_strategy enum and fixed text alignment in server.cpp example * Update ggml.c simplified return for platforms without NUMA support Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> * removed redundant else from cli argument processing of --numa * whitespace --------- Co-authored-by: root <root@nenya.lothlorien.ca> Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Jared Van Bortel <jared@nomic.ai>

bartowski1182 · 2024-07-28T14:52:26Z

Sorry to necro this @bmtwl but I'm wondering if you happen to know what the appropriate option is for a single 7702. I believe it has NUMA in a single socket, so wondering what options if any I should be using or how to test it

root added 7 commits February 6, 2024 08:54

Added numa options to allow finer grained control as well as plumbing…

d919c6d

… for a new mirror mode that will require numa.h

Reverted Makefile

65792fa

Fixed include

592e451

Removed sched.h from ggml.h, moved ggml_get_numa_affinity into ggml.c…

a69d6e2

…, removed trailing whitespace and fixed up a few inconsistent variables

removed trailing whitespace

60b80b0

Added numa options to allow finer grained control as well as plumbing…

7aa974d

… for a new mirror mode that will require numa.h

Reverting Makefile

12789eb

slaren reviewed Feb 7, 2024

View reviewed changes

ggml.h Outdated Show resolved Hide resolved

slaren reviewed Feb 7, 2024

View reviewed changes

llama.h Outdated Show resolved Hide resolved

Fixed a number of issues with the move from BOOL to ggml_numa_strateg…

c43808c

…ies. Added a note about mirror mode note being implemented yet

root added 2 commits February 7, 2024 21:36

Syncing to pr

3eccea1

Removing MIRROR_MODE code for this PR

61c37ba

cebtenzzre reviewed Feb 7, 2024

View reviewed changes

examples/server/server.cpp Outdated Show resolved Hide resolved

examples/server/server.cpp Outdated Show resolved Hide resolved

ggml.c Outdated Show resolved Hide resolved

ggml.c Outdated Show resolved Hide resolved

root and others added 5 commits February 7, 2024 22:02

Removing last bit of MIRROR_MODE code for this PR

d47f232

Removing unneeded branch in server.cpp example and moving get_numa_af…

783b7ca

…finity and making it static

Merge branch 'ggerganov:master' into master

f156112

Fixed lingering init_llama_backend() bool calls in tests and examples

12c23b6

Merge branch 'ggerganov:master' into master

18fb9a5

bmtwl and others added 2 commits February 8, 2024 09:17

Merge branch 'ggerganov:master' into master

90668fb

Remote enum llama_numa_strategies

b65c863

slaren reviewed Feb 8, 2024

View reviewed changes

common/common.cpp Outdated Show resolved Hide resolved

Revert bad merge with dynatemp flags

7bbe511

slaren reviewed Feb 8, 2024

View reviewed changes

ggml.c Outdated Show resolved Hide resolved

root and others added 3 commits February 8, 2024 19:55

add missing enum ggml_numa_strategies declaration and revert sync pro…

314174d

…blem with master

add missing enum ggml_numa_strategies declaration

c2c3166

Merge branch 'ggerganov:master' into master

fecd66a

Fix up some boolean vs enum comparisons

7fb5427

slaren approved these changes Feb 14, 2024

View reviewed changes

Added #ifdefs for non-Linux OS that don't have cpu_set_t datatype

e237527

ggerganov approved these changes Feb 15, 2024

View reviewed changes

bmtwl and others added 7 commits February 15, 2024 07:11

Update ggml.h

dc828c4

Align enum values Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Update ggml.c

4ffe18e

Remove whitespace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Update ggml.c

1585fec

align paremeters Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Update examples/server/server.cpp

c847828

remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Update common/common.cpp

377b58f

Remove whitespace and align brace Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Merge branch 'ggerganov:master' into master

5de34f5

unified ggml_numa_strategy enum and fixed text alignment in server.cp…

da65211

…p example

cebtenzzre reviewed Feb 15, 2024

View reviewed changes

ggml.c Outdated Show resolved Hide resolved

bmtwl and others added 2 commits February 15, 2024 09:39

Update ggml.c

7d1f026

simplified return for platforms without NUMA support Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>

Merge branch 'ggerganov:master' into master

a5c9a5d

cebtenzzre reviewed Feb 15, 2024

View reviewed changes

common/common.cpp Outdated Show resolved Hide resolved

removed redundant else from cli argument processing of --numa

a3cf7bf

cebtenzzre approved these changes Feb 15, 2024

View reviewed changes

whitespace

26ea983

ggerganov merged commit f486f6e into ggerganov:master Feb 16, 2024
51 of 54 checks passed

USBhost mentioned this pull request Feb 18, 2024

error: call to undeclared function 'pthread_getaffinity_np'; ISO C99 and later do not support implicit function declarations #5565

Closed

Zhenzhong1 mentioned this pull request Feb 19, 2024

error: implicit declaration of function ‘getcpu’ #5537

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h #5377

Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h #5377

bmtwl commented Feb 6, 2024 •

edited by ggerganov

Loading

cebtenzzre commented Feb 7, 2024

bmtwl commented Feb 7, 2024

cebtenzzre commented Feb 7, 2024

bmtwl commented Feb 8, 2024

bmtwl commented Feb 14, 2024

bmtwl commented Feb 14, 2024

slaren commented Feb 15, 2024 •

edited

Loading

bmtwl commented Feb 15, 2024

slaren commented Feb 15, 2024

bmtwl commented Feb 15, 2024

bartowski1182 commented Jul 28, 2024

Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h #5377

Added numa options to allow finer grained control as well as plumbing for a new mirror mode that will require numa.h #5377

Conversation

bmtwl commented Feb 6, 2024 • edited by ggerganov Loading

cebtenzzre commented Feb 7, 2024

bmtwl commented Feb 7, 2024

cebtenzzre commented Feb 7, 2024

bmtwl commented Feb 8, 2024

bmtwl commented Feb 14, 2024

bmtwl commented Feb 14, 2024

slaren commented Feb 15, 2024 • edited Loading

bmtwl commented Feb 15, 2024

slaren commented Feb 15, 2024

bmtwl commented Feb 15, 2024

bartowski1182 commented Jul 28, 2024

bmtwl commented Feb 6, 2024 •

edited by ggerganov

Loading

slaren commented Feb 15, 2024 •

edited

Loading