Kobold-cpp broken on linux in latest Main branch #51

EtereosDawn · 2024-06-07T05:58:40Z

EtereosDawn
Jun 7, 2024

Earlier today I ran git pull as usual and recompiled rocm, my models no longer load. I'm using rocm 6.0.2.


# System details report
---

## Report details
- **Date it was generated:** 2024-06-07 02:55:07

## Hardware information:
- **Hardware model:** SOYO SY-YL B550M
- **Memory:** 32.0 GiB
- **Processor:** AMD Ryzen™ 5 5600X × 12
- **Graphics:** AMD Radeon™ RX 6750 XT
- **Disk capacity:** 1.7 TB

## Software information:
- **Firmware version:** 5.17
- **OS Name:** Fedora Linux 40 (Workstation Edition)
- **OS Build:** (null)
- **OS Type:** 64-bit
- **GNOME Version:** 46
- **Window system:** Wayland
- **Kernel check:** Linux 6.8.11-300.fc40.x86_64

python koboldcpp.py --threads 6 --blasthreads 6 --usecublas mmq lowvram --gpulayers 18 --blasbatchsize 256 --contextsize 8192 --model
***
Welcome to KoboldCpp - Version 1.67.yr0-ROCm
Set AMD HSA_OVERRIDE_GFX_VERSION to 10.3.0
For command line arguments, please refer to --help
***
Attempting to use hipBLAS library for faster prompt ingestion. A compatible AMD GPU will be required.
Initializing dynamic library: koboldcpp_hipblas.so
==========
Namespace(model=None, model_param='/mnt/DiscoLocal/MachineLearnig/mythomax-l2-kimiko-v2-13b.Q5_K_M.gguf', port=5001, port_param=5001, host='', launch=False, config=None, threads=6, usecublas=['mmq', 'lowvram'], usevulkan=None, useclblast=None, noblas=False, contextsize=8192, gpulayers=18, tensor_split=None, checkforupdates=False, ropeconfig=[0.0, 10000.0], blasbatchsize=256, blasthreads=6, lora=None, noshift=False, nommap=False, usemlock=False, noavx2=False, debugmode=0, skiplauncher=False, onready='', benchmark=None, multiuser=0, remotetunnel=False, highpriority=False, foreground=False, preloadstory='', quiet=False, ssl=None, nocertify=False, mmproj='', password=None, ignoremissing=False, chatcompletionsadapter='', flashattention=False, quantkv=0, forceversion=0, smartcontext=False, hordemodelname='', hordeworkername='', hordekey='', hordemaxctx=0, hordegenlen=0, sdmodel='', sdthreads=0, sdclamped=False, sdvae='', sdvaeauto=False, sdquant=False, sdlora='', sdloramult=1.0, whispermodel='', hordeconfig=None, sdconfig=None)
==========
Loading model: /mnt/DiscoLocal/MachineLearnig/mythomax-l2-kimiko-v2-13b.Q5_K_M.gguf

The reported GGUF Arch is: llama

---
Identified as GGUF model: (ver 6)
Attempting to Load...
---
Using automatic RoPE scaling. If the model has customized RoPE settings, they will be used directly instead!
System Info: AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
llama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from /mnt/DiscoLocal/MachineLearnig/mythomax-l2-kimiko-v2-13b.Q5_K_M.gguf (version GGUF V2)
llm_load_vocab: special tokens cache size = 259
llm_load_vocab: token to piece cache size = 0.1684 MB
llm_load_print_meta: format           = GGUF V2
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 5120
llm_load_print_meta: n_head           = 40
llm_load_print_meta: n_head_kv        = 40
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 5120
llm_load_print_meta: n_embd_v_gqa     = 5120
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 13824
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 13B
llm_load_print_meta: model ftype      = unknown, may not work
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 8.60 GiB (5.67 BPW) 
llm_load_print_meta: general.name     = LLaMA v2
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6750 XT, compute capability 10.3, VMM: no
llm_load_tensors: ggml ctx size =    0.43 MiB
llm_load_tensors: offloading 18 repeating layers to GPU
llm_load_tensors: offloaded 18/41 layers to GPU
llm_load_tensors:      ROCm0 buffer size =  3866.99 MiB
llm_load_tensors:        CPU buffer size =  8801.63 MiB
...................................................................................................
Automatic RoPE Scaling: Using (scale:1.000, base:32768.0).
llama_new_context_with_model: n_ctx      = 8288
llama_new_context_with_model: n_batch    = 256
llama_new_context_with_model: n_ubatch   = 256
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 32768.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  ROCm_Host KV buffer size =  6475.00 MiB
llama_new_context_with_model: KV self size  = 6475.00 MiB, K (f16): 3237.50 MiB, V (f16): 3237.50 MiB
llama_new_context_with_model:  ROCm_Host  output buffer size =     0.12 MiB
llama_new_context_with_model:      ROCm0 compute buffer size =   164.42 MiB
llama_new_context_with_model:  ROCm_Host compute buffer size =   346.85 MiB
llama_new_context_with_model: graph nodes  = 1286
llama_new_context_with_model: graph splits = 260
ggml_cuda_compute_forward: ADD failed
CUDA error: shared object initialization failed
  current device: 0, in function ggml_cuda_compute_forward at ggml-cuda.cu:2366
  err
GGML_ASSERT: ggml-cuda.cu:102: !"CUDA error"
[New LWP 199031]
[New LWP 199290]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fe814113e03 in wait4 () from /lib64/libc.so.6
#0  0x00007fe814113e03 in wait4 () from /lib64/libc.so.6
#1  0x00007fe8020d5e9b in ggml_print_backtrace () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#2  0x00007fe80241b2f7 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#3  0x00007fe80241fc6e in ?? () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#4  0x00007fe80241a9a7 in ggml_backend_sched_graph_compute_async () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#5  0x00007fe802249588 in llama_decode () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#6  0x00007fe8022af8df in gpttype_load_model(load_model_inputs, FileFormat, FileFormatExtraMeta) () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#7  0x00007fe8021d4ded in load_model () from /home/madrugada/koboldcpp-rocm/koboldcpp_hipblas.so
#8  0x00007fe814855056 in ffi_call_unix64 () from /lib64/libffi.so.8
#9  0x00007fe8148516a0 in ffi_call_int.lto_priv () from /lib64/libffi.so.8
#10 0x00007fe8148544ee in ffi_call () from /lib64/libffi.so.8
#11 0x00007fe8060e9393 in _ctypes_callproc.cold () from /usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so
#12 0x00007fe8060f3472 in PyCFuncPtr_call () from /usr/lib64/python3.12/lib-dynload/_ctypes.cpython-312-x86_64-linux-gnu.so
#13 0x00007fe814368cc6 in _PyObject_MakeTpCall () from /lib64/libpython3.12.so.1.0
#14 0x00007fe8143716a4 in _PyEval_EvalFrameDefault () from /lib64/libpython3.12.so.1.0
#15 0x00007fe8143fbe24 in PyEval_EvalCode () from /lib64/libpython3.12.so.1.0
#16 0x00007fe8144206da in run_eval_code_obj () from /lib64/libpython3.12.so.1.0
#17 0x00007fe81441adee in run_mod () from /lib64/libpython3.12.so.1.0
#18 0x00007fe814435503 in pyrun_file () from /lib64/libpython3.12.so.1.0
#19 0x00007fe81443481c in _PyRun_SimpleFileObject () from /lib64/libpython3.12.so.1.0
#20 0x00007fe81443443f in _PyRun_AnyFileObject () from /lib64/libpython3.12.so.1.0
#21 0x00007fe81442cc73 in Py_RunMain () from /lib64/libpython3.12.so.1.0
#22 0x00007fe8143e403c in Py_BytesMain () from /lib64/libpython3.12.so.1.0
#23 0x00007fe81403d088 in __libc_start_call_main () from /lib64/libc.so.6
#24 0x00007fe81403d14b in __libc_start_main_impl () from /lib64/libc.so.6
#25 0x000055f109a7b095 in _start ()
[Inferior 1 (process 199025) detached]
Abortado (imagem do núcleo gravada)

YellowRoseCx · 2024-06-22T19:11:11Z

YellowRoseCx
Jun 22, 2024
Maintainer

did this end up getting resolved?

1 reply

EtereosDawn Jun 23, 2024
Author

unfortunately the problem persists in latest branch

DeepIndigo677 · 2024-06-22T19:16:33Z

DeepIndigo677
Jun 22, 2024

I've been having this error since 1.66.1

0 replies

TsunamicBug · 2024-06-24T17:16:23Z

TsunamicBug
Jun 24, 2024

For me on fedora 40 it just doesn't compile rocm properly. It'll just say:

clang: warning: argument unused during compilation: '-s' [-Wunused-command-line-argument]
In file included from ggml-cuda.cu:7:
./ggml-cuda/common.cuh:30:10: fatal error: 'rocblas/rocblas.h' file not found
30 | #include "rocblas/rocblas.h"
| ^~~~~~~~~~~~~~~~~~~

2 replies

EtereosDawn Jul 2, 2024
Author

install all packages

sudo dnf install rocm* hip*

EtereosDawn Jul 3, 2024
Author

Ops! the true fix for this problem is this

sudo dnf install rocblas-devel.x86_64 rocblas.x86_64

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kobold-cpp broken on linux in latest Main branch #51

{{title}}

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Kobold-cpp broken on linux in latest Main branch #51

EtereosDawn Jun 7, 2024

Replies: 3 comments · 3 replies

YellowRoseCx Jun 22, 2024 Maintainer

EtereosDawn Jun 23, 2024 Author

DeepIndigo677 Jun 22, 2024

TsunamicBug Jun 24, 2024

EtereosDawn Jul 2, 2024 Author

EtereosDawn Jul 3, 2024 Author

EtereosDawn
Jun 7, 2024

Replies: 3 comments 3 replies

YellowRoseCx
Jun 22, 2024
Maintainer

EtereosDawn Jun 23, 2024
Author

DeepIndigo677
Jun 22, 2024

TsunamicBug
Jun 24, 2024

EtereosDawn Jul 2, 2024
Author

EtereosDawn Jul 3, 2024
Author