Segfaulting on libamdhip64.so

[aschereT]

Running into an odd segfault:

 ❯ python3 koboldcpp.py --contextsize 4096 --gpulayers 1 --usecublas --model models/athena-v4.Q5_K_M.gguf                                                   [00:14:28] tall and link a BLAS library. Set LLAMA_OPENBLAS=1 to compile with OpenBLAS support or LLAMA_CLBLAST=1 to compile with ClBlast support. This is just a reminder,
***
Welcome to KoboldCpp - Version 1.61.2.yr0-ROCm
Set AMD HSA_OVERRIDE_GFX_VERSION to 10.3.0
Attempting to use hipBLAS library for faster prompt ingestion. A compatible AMD GPU will be required.
Initializing dynamic library: koboldcpp_hipblas.so
==========
Namespace(model='models/athena-v4.Q5_K_M.gguf', model_param='models/athena-v4.Q5_K_M.gguf', port=5001, port_param=5001, host='', launch=False, config=None, threads=11, usecublas=[], usevulkan=None, useclblast=None, noblas=False, gpulayers=1, tensor_split=None, contextsize=4096, ropeconfig=[0.0, 10000.0], blasbatchsize=512, blasthreads=11, lora=None, smartcontext=False, noshift=False, bantokens=None, forceversion=0, nommap=False, usemlock=False, noavx2=False, debugmode=0, skiplauncher=False, hordeconfig=None, onready='', benchmark=None, multiuser=0, remotetunnel=False, highpriority=False, foreground=False, preloadstory='', quiet=False, checkforupdates=False, ssl=None, nocertify=False, sdconfig=None, mmproj='', password=None, ignoremissing=False)                                                                                                  ------------------------------------
==========
Loading model: /home/user/repos/koboldcpp-rocm/models/athena-v4.Q5_K_M.gguf 
[Threads: 11, BlasThreads: 11, SmartContext: False, ContextShift: True]
 cachyos-extra-v3                                                                                                                                3.2 MiB  3.79 MiB/s 00:01 [--------------------------------------------------
The reported GGUF Arch is: llama
 cachyos                                                                                                                                       135.9 KiB   254 KiB/s 00:01 [--------------------------------------------------
---
Identified as GGUF model: (ver 6)
Attempting to Load...
---
Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | 
ggml_init_cublas: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6800 XT, compute capability 10.3, VMM: no
llama_model_loader: loaded meta data with 20 key-value pairs and 363 tensors from /home/user/repos/koboldcpp-rocm/models/athena-v4.Q5_K_M.gguf (version GGUF V2)
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V2
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_head           = 40
llm_load_print_meta: n_head_kv        = 40
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 5120
llm_load_print_meta: n_embd_v_gqa     = 5120
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 13824
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attm      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = all F32
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 8.60 GiB (5.67 BPW) 
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.32 MiB
llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/41 layers to GPU
llm_load_tensors:      ROCm0 buffer size =   220.29 MiB
llm_load_tensors:        CPU buffer size =  8801.63 MiB
...................................................................................................
Automatic RoPE Scaling: Using (scale:1.000, base:10000.0).
llama_new_context_with_model: n_ctx      = 4176
llama_new_context_with_model: n_batch    = 512
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
[1]    67488 segmentation fault  python3 koboldcpp.py --contextsize 4096 --gpulayers 1 --usecublas --model 

With the dmesg entry

[36852.652364] traps: python3[67488] general protection fault ip:7afc3fb303d8 sp:7ffe0ce60100 error:0 in libamdhip64.so.6.0.32831-bf7e2c6[7afc3f810000+37b000]

Only happens when using gpulayers > 0. Any ideas on what went wrong or how to fix it?

This is on 9c1707d.

rocminfo:

 ❯ rocminfo                                                                                                                                                                                                        [00:18:35]
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 9 5900X 12-Core Processor
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 9 5900X 12-Core Processor
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3700                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            24                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    65757828(0x3eb6284) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    65757828(0x3eb6284) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    65757828(0x3eb6284) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx1030                            
  Uuid:                    GPU-eae62a4d4e095f74               
  Marketing Name:          AMD Radeon RX 6800 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      4096(0x1000) KB                    
    L3:                      131072(0x20000) KB                 
  Chip ID:                 29631(0x73bf)                      
  ASIC Revision:           1(0x1)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   2575                               
  BDFID:                   2304                               
  Internal Node ID:        1                                  
  Compute Unit:            72                                 
  SIMDs per CU:            2                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   2                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          32(0x20)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        32(0x20)                           
  Max Work-item Per CU:    1024(0x400)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 116                                
  SDMA engine uCode::      83                                 
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    16760832(0xffc000) KB              
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    16760832(0xffc000) KB              
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx1030         
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***           

GDB stacktrace:

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00007ffff15397d0 in ?? () from /opt/rocm/lib/libamdhip64.so.6
(gdb) bt
#0  0x00007ffff15397d0 in ?? () from /opt/rocm/lib/libamdhip64.so.6
#1  0x00007ffff156c2a1 in ?? () from /opt/rocm/lib/libamdhip64.so.6
#2  0x00007ffff15359f0 in ?? () from /opt/rocm/lib/libamdhip64.so.6
#3  0x00007ffff1535ce7 in ?? () from /opt/rocm/lib/libamdhip64.so.6
#4  0x00007ffff150b3a1 in ?? () from /opt/rocm/lib/libamdhip64.so.6
#5  0x00007ffff13cb53c in ?? () from /opt/rocm/lib/libamdhip64.so.6
#6  0x00007ffff13cf83d in hipMemset () from /opt/rocm/lib/libamdhip64.so.6
#7  0x00007ffff53536e2 in ?? () from /home/user/repos/koboldcpp-rocm/koboldcpp_hipblas.so
#8  0x00007ffff513aeaa in llama_new_context_with_model () from /home/user/repos/koboldcpp-rocm/koboldcpp_hipblas.so
#9  0x00007ffff5192146 in gpttype_load_model(load_model_inputs, FileFormat, FileFormatExtraMeta) () from /home/user/repos/koboldcpp-rocm/koboldcpp_hipblas.so
#10 0x00007ffff50ed327 in load_model () from /home/user/repos/koboldcpp-rocm/koboldcpp_hipblas.so
#11 0x00007ffff7f9512e in ?? () from /usr/lib/libffi.so.8
#12 0x00007ffff7f91147 in ?? () from /usr/lib/libffi.so.8
#13 0x00007ffff7f945ce in ffi_call () from /usr/lib/libffi.so.8
#14 0x00007ffff7f36a02 in ?? () from /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#15 0x00007ffff7f35744 in ?? () from /usr/lib/python3.11/lib-dynload/_ctypes.cpython-311-x86_64-linux-gnu.so
#16 0x00007ffff7ad1e8c in ?? () from /usr/lib/libpython3.11.so.1.0
#17 0x00007ffff7ae0063 in _PyEval_EvalFrameDefault () from /usr/lib/libpython3.11.so.1.0
#18 0x00007ffff7bc7594 in ?? () from /usr/lib/libpython3.11.so.1.0
#19 0x00007ffff7bc6c1f in ?? () from /usr/lib/libpython3.11.so.1.0
#20 0x00007ffff7bf012f in ?? () from /usr/lib/libpython3.11.so.1.0
#21 0x00007ffff7beae82 in ?? () from /usr/lib/libpython3.11.so.1.0
#22 0x00007ffff7c08e83 in ?? () from /usr/lib/libpython3.11.so.1.0
#23 0x00007ffff7c08165 in ?? () from /usr/lib/libpython3.11.so.1.0
#24 0x00007ffff7c07320 in ?? () from /usr/lib/libpython3.11.so.1.0
#25 0x00007ffff7bff62e in ?? () from /usr/lib/libpython3.11.so.1.0
#26 0x00007ffff7bb4dbb in ?? () from /usr/lib/libpython3.11.so.1.0
#27 0x00007ffff7625d90 in ?? () from /usr/lib/libc.so.6
#28 0x00007ffff7625e4a in __libc_start_main () from /usr/lib/libc.so.6
#29 0x0000000000401045 in ?? ()

[YellowRoseCx]

I haven't seen that error before, have you tried running the program with Python 3.10? Have you added the group permissions for ROCm? https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/prerequisites.html#setting-permissions-for-groups

Which Linux Distro are you on?

[aschereT]

Hi, sorry for the late reply.

I'm running on CachyOS (arch derivative). I have the group permissions. From what I can tell, this was actually a bug with ROCm 6.0. The recent 6.1 release should fix it, and I'll report back once that hits the repos.