Cannot load Bloom-7b1 ggml model in GPU #3697

zolastro · 2023-10-20T12:08:36Z

I used the convert-bloom-hf-to-gguf.py file to convert the Huggingface bigscience/bloom-7b1 to a ggml model with f16 successfully:

python convert-bloom-hf-to-gguf.py models/bloom-7b1/ 1

This gives me a model ggml-model-f16.gguf that correctly loads and run in CPU. However, when I try to offload a layer on the GPU, I get the following error:

GGML_ASSERT: /llama.cpp/ggml-cuda.cu:6115: false

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

Physical (or virtual) hardware you are using, e.g. for Linux:

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             64
On-line CPU(s) list:                0-63
Thread(s) per core:                 2
Core(s) per socket:                 16
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              79
Model name:                         Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
Stepping:                           1
CPU MHz:                            1197.469
CPU max MHz:                        3000.0000
CPU min MHz:                        1200.0000
BogoMIPS:                           4190.27
Virtualization:                     VT-x
L1d cache:                          1 MiB
L1i cache:                          1 MiB
L2 cache:                           8 MiB
L3 cache:                           80 MiB
NUMA node0 CPU(s):                  0-15,32-47
NUMA node1 CPU(s):                  16-31,48-63
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        KVM: Mitigation: Split huge pages
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT vulnerable
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp l
                                    m constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3
                                     sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fa
                                    ult epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle 
                                    avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln p
                                    ts md_clear flush_l1d

Operating System, e.g. for Linux:

Linux nemo 5.4.0-165-generic #182-Ubuntu SMP Mon Oct 2 19:43:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

SDK version, e.g. for Linux:

Python 3.10.13
GNU Make 4.2.1
g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

Clone bloom7b1 from Huggingface (https://huggingface.co/bigscience/bloom-7b1)
Use the convert-bloom-hf-to-gguf.py to convert to f16 ggml.
Try to load the model on GPU:

./build/bin/main -m models/bloom-7b1/ggml-model-f16.gguf -n 256 -b 512 -c 512  -f ../prompt.txt --threads 32 --temp 0.1 --top-p 0.75 --top-k 40 -cb -ngl 33

Failure Logs

./build/bin/main -m models/bloom-7b1/ggml-model-f16.gguf -n 256 -b 512 -c 512  -f ../prompt.txt --threads 32 --temp 0.1 --top-p 0.75 --top-k 40 -cb -ngl 33
Log start
main: build = 1399 (004797f)
main: built with cc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 for x86_64-linux-gnu
main: seed  = 1697803662
ggml_init_cublas: found 6 CUDA devices:
  Device 0: NVIDIA TITAN RTX, compute capability 7.5
  Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5
  Device 2: NVIDIA TITAN Xp, compute capability 6.1
  Device 3: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1
  Device 4: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1
  Device 5: NVIDIA GeForce GTX 1080 Ti, compute capability 6.1
llama_model_loader: loaded meta data with 19 key-value pairs and 366 tensors from models/bloom-7b1/ggml-model-f16.gguf (version GGUF V2 (latest))
llama_model_loader: - tensor    0:                token_embd.weight f16      [  4096, 250880,     1,     1 ]
llama_model_loader: - tensor    1:                    output.weight f16      [  4096, 250880,     1,     1 ]
llama_model_loader: - tensor    2:           token_embd_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor    3:             token_embd_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor    4:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor    5:             blk.0.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor    6:            blk.0.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor    7:              blk.0.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor    8:         blk.0.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor    9:           blk.0.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   10:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   11:              blk.0.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   12:              blk.0.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   13:                blk.0.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   14:            blk.0.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   15:              blk.0.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   16:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   17:             blk.1.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   18:            blk.1.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   19:              blk.1.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   20:         blk.1.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   21:           blk.1.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   22:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   23:              blk.1.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   24:              blk.1.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   25:                blk.1.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   26:            blk.1.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   27:              blk.1.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   28:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   29:             blk.2.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   30:            blk.2.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   31:              blk.2.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   32:         blk.2.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   33:           blk.2.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   34:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   35:              blk.2.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   36:              blk.2.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   37:                blk.2.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   38:            blk.2.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   39:              blk.2.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   40:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   41:             blk.3.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   42:            blk.3.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   43:              blk.3.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   44:         blk.3.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   45:           blk.3.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   46:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   47:              blk.3.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   48:              blk.3.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   49:                blk.3.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   50:            blk.3.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   51:              blk.3.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   52:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   53:             blk.4.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   54:            blk.4.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   55:              blk.4.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   56:         blk.4.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   57:           blk.4.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   58:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   59:              blk.4.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   60:              blk.4.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   61:                blk.4.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   62:            blk.4.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   63:              blk.4.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   64:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   65:             blk.5.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   66:            blk.5.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   67:              blk.5.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   68:         blk.5.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   69:           blk.5.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   70:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   71:              blk.5.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   72:              blk.5.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   73:                blk.5.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   74:            blk.5.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   75:              blk.5.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   76:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   77:             blk.6.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   78:            blk.6.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   79:              blk.6.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   80:         blk.6.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   81:           blk.6.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   82:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   83:              blk.6.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   84:              blk.6.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   85:                blk.6.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   86:            blk.6.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   87:              blk.6.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   88:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   89:             blk.7.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   90:            blk.7.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor   91:              blk.7.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor   92:         blk.7.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   93:           blk.7.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   94:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   95:              blk.7.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   96:              blk.7.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor   97:                blk.7.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor   98:            blk.7.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor   99:              blk.7.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  100:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  101:             blk.8.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  102:            blk.8.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  103:              blk.8.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  104:         blk.8.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  105:           blk.8.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  106:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  107:              blk.8.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  108:              blk.8.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  109:                blk.8.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  110:            blk.8.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  111:              blk.8.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  112:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  113:             blk.9.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  114:            blk.9.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  115:              blk.9.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  116:         blk.9.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  117:           blk.9.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  118:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  119:              blk.9.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  120:              blk.9.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  121:                blk.9.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  122:            blk.9.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  123:              blk.9.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  124:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  125:            blk.10.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  126:           blk.10.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  127:             blk.10.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  128:        blk.10.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  129:          blk.10.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  130:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  131:             blk.10.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  132:             blk.10.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  133:               blk.10.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  134:           blk.10.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  135:             blk.10.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  136:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  137:            blk.11.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  138:           blk.11.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  139:             blk.11.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  140:        blk.11.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  141:          blk.11.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  142:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  143:             blk.11.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  144:             blk.11.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  145:               blk.11.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  146:           blk.11.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  147:             blk.11.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  148:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  149:            blk.12.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  150:           blk.12.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  151:             blk.12.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  152:        blk.12.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  153:          blk.12.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  154:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  155:             blk.12.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  156:             blk.12.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  157:               blk.12.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  158:           blk.12.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  159:             blk.12.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  160:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  161:            blk.13.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  162:           blk.13.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  163:             blk.13.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  164:        blk.13.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  165:          blk.13.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  166:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  167:             blk.13.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  168:             blk.13.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  169:               blk.13.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  170:           blk.13.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  171:             blk.13.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  172:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  173:            blk.14.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  174:           blk.14.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  175:             blk.14.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  176:        blk.14.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  177:          blk.14.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  178:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  179:             blk.14.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  180:             blk.14.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  181:               blk.14.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  182:           blk.14.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  183:             blk.14.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  184:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  185:            blk.15.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  186:           blk.15.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  187:             blk.15.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  188:        blk.15.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  189:          blk.15.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  190:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  191:             blk.15.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  192:             blk.15.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  193:               blk.15.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  194:           blk.15.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  195:             blk.15.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  196:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  197:            blk.16.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  198:           blk.16.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  199:             blk.16.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  200:        blk.16.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  201:          blk.16.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  202:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  203:             blk.16.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  204:             blk.16.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  205:               blk.16.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  206:           blk.16.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  207:             blk.16.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  208:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  209:            blk.17.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  210:           blk.17.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  211:             blk.17.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  212:        blk.17.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  213:          blk.17.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  214:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  215:             blk.17.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  216:             blk.17.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  217:               blk.17.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  218:           blk.17.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  219:             blk.17.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  220:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  221:            blk.18.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  222:           blk.18.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  223:             blk.18.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  224:        blk.18.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  225:          blk.18.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  226:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  227:             blk.18.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  228:             blk.18.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  229:               blk.18.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  230:           blk.18.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  231:             blk.18.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  232:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  233:            blk.19.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  234:           blk.19.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  235:             blk.19.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  236:        blk.19.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  237:          blk.19.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  238:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  239:             blk.19.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  240:             blk.19.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  241:               blk.19.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  242:           blk.19.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  243:             blk.19.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  244:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  245:            blk.20.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  246:           blk.20.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  247:             blk.20.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  248:        blk.20.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  249:          blk.20.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  250:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  251:             blk.20.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  252:             blk.20.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  253:               blk.20.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  254:           blk.20.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  255:             blk.20.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  256:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  257:            blk.21.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  258:           blk.21.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  259:             blk.21.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  260:        blk.21.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  261:          blk.21.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  262:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  263:             blk.21.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  264:             blk.21.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  265:               blk.21.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  266:           blk.21.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  267:             blk.21.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  268:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  269:            blk.22.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  270:           blk.22.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  271:             blk.22.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  272:        blk.22.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  273:          blk.22.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  274:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  275:             blk.22.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  276:             blk.22.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  277:               blk.22.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  278:           blk.22.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  279:             blk.22.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  280:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  281:            blk.23.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  282:           blk.23.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  283:             blk.23.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  284:        blk.23.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  285:          blk.23.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  286:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  287:             blk.23.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  288:             blk.23.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  289:               blk.23.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  290:           blk.23.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  291:             blk.23.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  292:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  293:            blk.24.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  294:           blk.24.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  295:             blk.24.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  296:        blk.24.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  297:          blk.24.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  298:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  299:             blk.24.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  300:             blk.24.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  301:               blk.24.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  302:           blk.24.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  303:             blk.24.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  304:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  305:            blk.25.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  306:           blk.25.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  307:             blk.25.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  308:        blk.25.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  309:          blk.25.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  310:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  311:             blk.25.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  312:             blk.25.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  313:               blk.25.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  314:           blk.25.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  315:             blk.25.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  316:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  317:            blk.26.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  318:           blk.26.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  319:             blk.26.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  320:        blk.26.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  321:          blk.26.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  322:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  323:             blk.26.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  324:             blk.26.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  325:               blk.26.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  326:           blk.26.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  327:             blk.26.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  328:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  329:            blk.27.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  330:           blk.27.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  331:             blk.27.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  332:        blk.27.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  333:          blk.27.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  334:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  335:             blk.27.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  336:             blk.27.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  337:               blk.27.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  338:           blk.27.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  339:             blk.27.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  340:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  341:            blk.28.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  342:           blk.28.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  343:             blk.28.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  344:        blk.28.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  345:          blk.28.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  346:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  347:             blk.28.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  348:             blk.28.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  349:               blk.28.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  350:           blk.28.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  351:             blk.28.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  352:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  353:            blk.29.attn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  354:           blk.29.attn_qkv.weight f16      [  4096, 12288,     1,     1 ]
llama_model_loader: - tensor  355:             blk.29.attn_qkv.bias f32      [ 12288,     1,     1,     1 ]
llama_model_loader: - tensor  356:        blk.29.attn_output.weight f16      [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  357:          blk.29.attn_output.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  358:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  359:             blk.29.ffn_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  360:             blk.29.ffn_up.weight f16      [  4096, 16384,     1,     1 ]
llama_model_loader: - tensor  361:               blk.29.ffn_up.bias f32      [ 16384,     1,     1,     1 ]
llama_model_loader: - tensor  362:           blk.29.ffn_down.weight f16      [ 16384,  4096,     1,     1 ]
llama_model_loader: - tensor  363:             blk.29.ffn_down.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  364:               output_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  365:                 output_norm.bias f32      [  4096,     1,     1,     1 ]
llama_model_loader: - kv   0:                       general.architecture str     
llama_model_loader: - kv   1:                               general.name str     
llama_model_loader: - kv   2:                       bloom.context_length u32     
llama_model_loader: - kv   3:                     bloom.embedding_length u32     
llama_model_loader: - kv   4:                  bloom.feed_forward_length u32     
llama_model_loader: - kv   5:                          bloom.block_count u32     
llama_model_loader: - kv   6:                 bloom.attention.head_count u32     
llama_model_loader: - kv   7:              bloom.attention.head_count_kv u32     
llama_model_loader: - kv   8:         bloom.attention.layer_norm_epsilon f32     
llama_model_loader: - kv   9:                          general.file_type u32     
llama_model_loader: - kv  10:                       tokenizer.ggml.model str     
llama_model_loader: - kv  11:                      tokenizer.ggml.tokens arr     
llama_model_loader: - kv  12:                      tokenizer.ggml.scores arr     
llama_model_loader: - kv  13:                  tokenizer.ggml.token_type arr     
llama_model_loader: - kv  14:                      tokenizer.ggml.merges arr     
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32     
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32     
llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32     
llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32     
llama_model_loader: - type  f32:  244 tensors
llama_model_loader: - type  f16:  122 tensors
llm_load_vocab: mismatch in special tokens definition ( 203/250880 vs 0/250880 ).
llm_load_print_meta: format           = GGUF V2 (latest)
llm_load_print_meta: arch             = bloom
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 250880
llm_load_print_meta: n_merges         = 250434
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 30
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: f_norm_eps       = 1.0e-05
llm_load_print_meta: f_norm_rms_eps   = 0.0e+00
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 16384
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = mostly F16
llm_load_print_meta: model params     = 8.10 B
llm_load_print_meta: model size       = 15.08 GiB (16.00 BPW) 
llm_load_print_meta: general.name   = Bloom
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: PAD token = 3 '<pad>'
llm_load_print_meta: LF token  = 130 'Ä'
llm_load_tensors: ggml ctx size =    0.12 MB
llm_load_tensors: using CUDA for GPU acceleration
ggml_cuda_set_main_device: using device 0 (NVIDIA TITAN RTX) as main device
llm_load_tensors: mem required  = 1960.15 MB
llm_load_tensors: offloading 30 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors: VRAM used: 13486.12 MB
...GGML_ASSERT: /llama.cpp/ggml-cuda.cu:6115: false
Aborted (core dumped)

The text was updated successfully, but these errors were encountered:

QueryType · 2023-10-20T18:01:51Z

I am not able to load llava model ggml-model-q4_k.gguf, mmproj mmproj-model-f16.gguf, into GPU too. (main) llama v2 is working fine.
`llama.cpp % ./run_llava.sh /Volumes/d/shm/output/2023-10-20/00000-3438987449-swapped.png "describe the image"
clip_model_load: model name: openai/clip-vit-large-patch14-336
clip_model_load: description: image encoder for LLaVA
clip_model_load: GGUF version: 2
clip_model_load: alignment: 32
clip_model_load: n_tensors: 377
clip_model_load: n_kv: 18
clip_model_load: ftype: f16

clip_model_load: text_encoder: 0
clip_model_load: vision_encoder: 1
clip_model_load: llava_projector: clip_model_load: model size: clip_model_load: metadata size: clip_model_load: total allocated llama_model_loader: loaded llama_model_loader: - tensor 0: llama_model_loader: - tensor 1: llama_model_loader: - tensor 2: llama_model_loader: - tensor 3: llama_model_loader: - tensor 4: llama_model_loader: - tensor 5: llama_model_loader: - tensor 6: llama_model_loader: - tensor 7: llama_model_loader: - tensor 8: llama_model_loader: - tensor 9: llama_model_loader: - tensor 10: llama_model_loader: - tensor 11: llama_model_loader: - tensor 12: llama_model_loader: - tensor 13: llama_model_loader: - tensor 14: llama_model_loader: - tensor 15: llama_model_loader: - tensor 16: llama_model_loader: - tensor 17: llama_model_loader: - tensor 18: llama_model_loader: - tensor 19: llama_model_loader: - tensor 20: llama_model_loader: - tensor 21: llama_model_loader: - tensor 22: llama_model_loader: - tensor 23: llama_model_loader: - tensor 24: llama_model_loader: - tensor 25: llama_model_loader: - tensor 26: llama_model_loader: - tensor 27: llama_model_loader: - tensor 28: llama_model_loader: - tensor 29: llama_model_loader: - tensor 30: llama_model_loader: - tensor 31: llama_model_loader: - tensor 32: llama_model_loader: - tensor 33: llama_model_loader: - tensor 34: llama_model_loader: - tensor 35: llama_model_loader: - tensor 36: llama_model_loader: - tensor 37: llama_model_loader: - tensor 38: llama_model_loader: - tensor 39: llama_model_loader: - tensor 40: llama_model_loader: - tensor 41: llama_model_loader: - tensor 42: llama_model_loader: - tensor 43: llama_model_loader: - tensor 44: llama_model_loader: - tensor 45: llama_model_loader: - tensor 46: llama_model_loader: - tensor 47: llama_model_loader: - tensor 48: llama_model_loader: - tensor 49: llama_model_loader: - tensor 50: llama_model_loader: - tensor 51: llama_model_loader: - tensor 52: llama_model_loader: - tensor 53: llama_model_loader: - tensor 54: llama_model_loader: - tensor 55: llama_model_loader: - tensor 56: llama_model_loader: - tensor 57: llama_model_loader: - tensor 58: llama_model_loader: - tensor 59: llama_model_loader: - tensor 60: llama_model_loader: - tensor 61: llama_model_loader: - tensor 62: llama_model_loader: - tensor 63: llama_model_loader: - tensor 64: llama_model_loader: - tensor 65: llama_model_loader: - tensor 66: llama_model_loader: - tensor 67: llama_model_loader: - tensor 68: llama_model_loader: - tensor 69: llama_model_loader: - tensor 70: llama_model_loader: - tensor 71: llama_model_loader: - tensor 72: llama_model_loader: - tensor 73: llama_model_loader: - tensor 74: llama_model_loader: - tensor 75: llama_model_loader: - tensor 76: llama_model_loader: - tensor 77: llama_model_loader: - tensor 78: llama_model_loader: - tensor 79: llama_model_loader: - tensor 80: llama_model_loader: - tensor 81: llama_model_loader: - tensor 82: llama_model_loader: - tensor 83: llama_model_loader: - tensor 84: llama_model_loader: - tensor 85: llama_model_loader: - tensor 86: llama_model_loader: - tensor 87: llama_model_loader: - tensor 88: llama_model_loader: - tensor 89: llama_model_loader: - tensor 90: llama_model_loader: - tensor 91: llama_model_loader: - tensor 92: llama_model_loader: - tensor 93: llama_model_loader: - tensor 94: llama_model_loader: - tensor 95: llama_model_loader: - tensor 96: llama_model_loader: - tensor 97: llama_model_loader: - tensor 98: llama_model_loader: - tensor 99: llama_model_loader: - tensor 100: llama_model_loader: - tensor 101: llama_model_loader: - tensor 102: llama_model_loader: - tensor 103: llama_model_loader: - tensor 104: llama_model_loader: - tensor 105: llama_model_loader: - tensor 106: llama_model_loader: - tensor 107: llama_model_loader: - tensor 108: llama_model_loader: - tensor 109: llama_model_loader: - tensor 110: llama_model_loader: - tensor 111: llama_model_loader: - tensor 112: llama_model_loader: - tensor 113: llama_model_loader: - tensor 114: llama_model_loader: - tensor 115: llama_model_loader: - tensor 116: llama_model_loader: - tensor 117: llama_model_loader: - tensor 118: llama_model_loader: - tensor 119: llama_model_loader: - tensor 120: llama_model_loader: - tensor 121: llama_model_loader: - tensor 122: llama_model_loader: - tensor 123: llama_model_loader: - tensor 124: llama_model_loader: - tensor 125: llama_model_loader: - tensor 126: llama_model_loader: - tensor 127: llama_model_loader: - tensor 128: llama_model_loader: - tensor 129: llama_model_loader: - tensor 130: llama_model_loader: - tensor 131: llama_model_loader: - tensor 132: llama_model_loader: - tensor 133: llama_model_loader: - tensor 134: llama_model_loader: - tensor 135: llama_model_loader: - tensor 136: llama_model_loader: - tensor 137: llama_model_loader: - tensor 138: llama_model_loader: - tensor 139: llama_model_loader: - tensor 140: llama_model_loader: - tensor 141: llama_model_loader: - tensor 142: llama_model_loader: - tensor 143: llama_model_loader: - tensor 144: llama_model_loader: - tensor 145: llama_model_loader: - tensor 146: llama_model_loader: - tensor 147: llama_model_loader: - tensor 148: llama_model_loader: - tensor 149: llama_model_loader: - tensor 150: llama_model_loader: - tensor 151: llama_model_loader: - tensor 152: llama_model_loader: - tensor 153: llama_model_loader: - tensor 154: llama_model_loader: - tensor 155: llama_model_loader: - tensor 156: llama_model_loader: - tensor 157: llama_model_loader: - tensor 158: llama_model_loader: - tensor 159: llama_model_loader: - tensor 160: llama_model_loader: - tensor 161: llama_model_loader: - tensor 162: llama_model_loader: - tensor 163: llama_model_loader: - tensor 164: llama_model_loader: - tensor 165: llama_model_loader: - tensor 166: llama_model_loader: - tensor 167: llama_model_loader: - tensor 168: llama_model_loader: - tensor 169: llama_model_loader: - tensor 170: llama_model_loader: - tensor 171: llama_model_loader: - tensor 172: llama_model_loader: - tensor 173: llama_model_loader: - tensor 174: llama_model_loader: - tensor 175: llama_model_loader: - tensor 176: llama_model_loader: - tensor 177: llama_model_loader: - tensor 178: llama_model_loader: - tensor 179: llama_model_loader: - tensor 180: llama_model_loader: - tensor 181: llama_model_loader: - tensor 182: llama_model_loader: - tensor 183: llama_model_loader: - tensor 184: llama_model_loader: - tensor 185: llama_model_loader: - tensor 186: llama_model_loader: - tensor 187: llama_model_loader: - tensor 188: llama_model_loader: - tensor 189: llama_model_loader: - tensor 190: llama_model_loader: - tensor 191: llama_model_loader: - tensor 192: llama_model_loader: - tensor 193: llama_model_loader: - tensor 194: llama_model_loader: - tensor 195: llama_model_loader: - tensor 196: llama_model_loader: - tensor 197: llama_model_loader: - tensor 198: llama_model_loader: - tensor 199: llama_model_loader: - tensor 200: llama_model_loader: - tensor 201: llama_model_loader: - tensor 202: llama_model_loader: - tensor 203: llama_model_loader: - tensor 204: llama_model_loader: - tensor 205: llama_model_loader: - tensor 206: llama_model_loader: - tensor 207: llama_model_loader: - tensor 208: llama_model_loader: - tensor 209: llama_model_loader: - tensor 210: llama_model_loader: - tensor 211: llama_model_loader: - tensor 212: llama_model_loader: - tensor 213: llama_model_loader: - tensor 214: llama_model_loader: - tensor 215: llama_model_loader: - tensor 216: llama_model_loader: - tensor 217: llama_model_loader: - tensor 218: llama_model_loader: - tensor 219: llama_model_loader: - tensor 220: llama_model_loader: - tensor 221: llama_model_loader: - tensor 222: llama_model_loader: - tensor 223: llama_model_loader: - tensor 224: llama_model_loader: - tensor 225: llama_model_loader: - tensor 226: llama_model_loader: - tensor 227: llama_model_loader: - tensor 228: llama_model_loader: - tensor 229: llama_model_loader: - tensor 230: llama_model_loader: - tensor 231: llama_model_loader: - tensor 232: llama_model_loader: - tensor 233: llama_model_loader: - tensor 234: llama_model_loader: - tensor 235: llama_model_loader: - tensor 236: llama_model_loader: - tensor 237: llama_model_loader: - tensor 238: llama_model_loader: - tensor 239: llama_model_loader: - tensor 240: llama_model_loader: - tensor 241: llama_model_loader: - tensor 242: llama_model_loader: - tensor 243: llama_model_loader: - tensor 244: llama_model_loader: - tensor 245: llama_model_loader: - tensor 246: llama_model_loader: - tensor 247: llama_model_loader: - tensor 248: llama_model_loader: - tensor 249: llama_model_loader: - tensor 250: llama_model_loader: - tensor 251: llama_model_loader: - tensor 252: llama_model_loader: - tensor 253: llama_model_loader: - tensor 254: llama_model_loader: - tensor 255: llama_model_loader: - tensor 256: llama_model_loader: - tensor 257: llama_model_loader: - tensor 258: llama_model_loader: - tensor 259: llama_model_loader: - tensor 260: llama_model_loader: - tensor 261: llama_model_loader: - tensor 262: llama_model_loader: - tensor 263: llama_model_loader: - tensor 264: llama_model_loader: - tensor 265: llama_model_loader: - tensor 266: llama_model_loader: - tensor 267: llama_model_loader: - tensor 268: llama_model_loader: - tensor 269: llama_model_loader: - tensor 270: llama_model_loader: - tensor 271: llama_model_loader: - tensor 272: llama_model_loader: - tensor 273: llama_model_loader: - tensor 274: llama_model_loader: - tensor 275: llama_model_loader: - tensor 276: llama_model_loader: - tensor 277: llama_model_loader: - tensor 278: llama_model_loader: - tensor 279: llama_model_loader: - tensor 280: llama_model_loader: - tensor 281: llama_model_loader: - tensor 282: llama_model_loader: - tensor 283: llama_model_loader: - tensor 284: llama_model_loader: - tensor 285: llama_model_loader: - tensor 286: llama_model_loader: - tensor 287: llama_model_loader: - tensor 288: llama_model_loader: - tensor 289: llama_model_loader: - tensor 290: llama_model_loader: - kv 0: llama_model_loader: - kv 1: llama_model_loader: - kv 2: llama_model_loader: - kv 3: llama_model_loader: - kv 4: llama_model_loader: - kv 5: llama_model_loader: - kv 6: llama_model_loader: - kv 7: llama_model_loader: - kv 8: llama_model_loader: - kv 9: llama_model_loader: - kv 10: llama_model_loader: - kv 11: llama_model_loader: - kv 12: llama_model_loader: - kv 13: llama_model_loader: - kv 14: llama_model_loader: - kv 15: llama_model_loader: - kv 16: llama_model_loader: - kv 17: llama_model_loader: - kv 18: llama_model_loader: - type f32: llama_model_loader: - type q4_K: llama_model_loader: - type q6_K: llm_load_vocab: special tokens llm_load_print_meta: format llm_load_print_meta: arch llm_load_print_meta: vocab type llm_load_print_meta: n_vocab llm_load_print_meta: n_merges llm_load_print_meta: n_ctx_train llm_load_print_meta: n_embd llm_load_print_meta: n_head llm_load_print_meta: n_head_kv llm_load_print_meta: n_layer llm_load_print_meta: n_rot llm_load_print_meta: n_gqa llm_load_print_meta: f_norm_eps llm_load_print_meta: f_norm_rms_eps llm_load_print_meta: f_clamp_kqv llm_load_print_meta: f_max_alibi_bias llm_load_print_meta: n_ff llm_load_print_meta: freq_base_train llm_load_print_meta: freq_scale_train llm_load_print_meta: model type llm_load_print_meta: model ftype llm_load_print_meta: model params llm_load_print_meta: model size llm_load_print_meta: general.name llm_load_print_meta: BOS llm_load_print_meta: EOS llm_load_print_meta: UNK llm_load_print_meta: PAD llm_load_print_meta: LF token llm_load_tensors: ggml ctx size = llm_load_tensors: mem required .................................. llama_new_context_with_model: n_ctx llama_new_context_with_model: freq_base llama_new_context_with_model: llama_new_context_with_model: kv self size llama_new_context_with_model: 1
595.61 MB
0.13 MB
memory: 201.27 MB
meta data with 19 key-value pairs and 291 tensors from /Volumes/d/apps/aimodels/llama2/hf_models/llava/ggml-model-q4_k.gguf (version GGUF V2 (latest))
token_embd.weight q4_K [ 4096, 32000, 1, 1 ]
blk.0.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.0.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.0.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.0.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.0.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.1.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.1.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.1.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.1.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.2.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.2.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.2.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.2.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.3.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.3.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.3.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.3.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.4.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.4.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.4.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.5.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.5.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.5.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.6.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.6.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.6.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.6.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.7.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.7.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.7.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.8.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.8.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.8.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.9.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.9.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.9.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.9.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.10.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.10.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.10.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.11.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.11.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.11.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.12.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.12.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.12.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.12.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.13.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.13.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.13.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.14.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.14.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.14.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.15.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.15.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.15.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.15.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.16.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.16.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.16.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.17.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.17.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.17.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.18.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.18.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.18.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.18.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.19.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.19.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.19.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.20.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.20.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.20.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.21.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.21.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.21.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.21.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.22.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.22.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.22.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.23.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.23.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.23.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.24.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.24.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.24.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.24.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.25.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.25.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.25.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_v.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.26.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.26.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.26.ffn_down.weight q4_K [ 11008, 4096, 1, 1 ]
blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.27.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.27.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.27.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.27.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.28.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.28.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.28.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.28.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.29.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.29.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.29.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.29.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.30.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.30.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.30.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.30.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.attn_q.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.attn_k.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.attn_v.weight q6_K [ 4096, 4096, 1, 1 ]
blk.31.attn_output.weight q4_K [ 4096, 4096, 1, 1 ]
blk.31.ffn_gate.weight q4_K [ 4096, 11008, 1, 1 ]
blk.31.ffn_up.weight q4_K [ 4096, 11008, 1, 1 ]
blk.31.ffn_down.weight q6_K [ 11008, 4096, 1, 1 ]
blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ]
blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ]
output_norm.weight f32 [ 4096, 1, 1, 1 ]
output.weight q6_K [ 4096, 32000, 1, 1 ]
general.architecture str
general.name str
llama.context_length u32
llama.embedding_length u32
llama.block_count u32
llama.feed_forward_length u32
llama.rope.dimension_count u32
llama.attention.head_count u32
llama.attention.head_count_kv u32
llama.attention.layer_norm_rms_epsilon f32
general.file_type u32
tokenizer.ggml.model str
tokenizer.ggml.tokens arr
tokenizer.ggml.scores arr
tokenizer.ggml.token_type arr
tokenizer.ggml.bos_token_id u32
tokenizer.ggml.eos_token_id u32
tokenizer.ggml.padding_token_id u32
general.quantization_version u32
65 tensors
193 tensors
33 tensors
definition check successful ( 259/32000 ).
= GGUF V2 (latest)
= llama
= SPM
= 32000
= 0
= 4096
= 4096
= 32
= 32
= 32
= 128
= 1
= 0.0e+00
= 1.0e-05
= 0.0e+00
= 0.0e+00
= 11008
= 10000.0
= 1
= 7B
= mostly Q4_K - Medium
= 6.74 B
= 3.80 GiB (4.84 BPW)
= LLaMA v2
token = 1 ''
token = 2 ''
token = 0 ''
token = 0 ''
= 13 '<0x0A>'
0.10 MB
= 3891.34 MB
................................................................
= 2048
= 10000.0
freq_scale = 1
= 1024.00 MB
compute buffer total size = 162.13 MB

prompt: 'describe the image'`

nlpcat · 2023-11-02T05:02:57Z

is there any solution for this? I found that models with alibi all seem to have this issue on nvidia gpu. it can run successfully on metal.

ggerganov · 2023-11-02T08:06:00Z

As a temporary workaround, you can add LLAMA_CUDA_MMV_Y=4 to your build and it should work on master.
See the discussion in #3740 (comment)
We still need a proper fix though

slaren · 2023-11-03T11:18:55Z

Fixed in #3921

ggerganov added a commit that referenced this issue Nov 2, 2023

llm : prevent from 1-D tensors being GPU split (#3697)

1efae9b

ggerganov mentioned this issue Nov 3, 2023

ggml-cuda : move row numbers to x grid dim in mul mat vec kernels #3921

Merged

slaren closed this as completed Nov 3, 2023

rvandernoort mentioned this issue Nov 10, 2023

Pytorch converted model returns .GGML_ASSERT: ggml-cuda.cu:6115: false #4017

Closed

4 tasks

olexiyb pushed a commit to Sanctum-AI/llama.cpp that referenced this issue Nov 23, 2023

llm : prevent from 1-D tensors being GPU split (ggerganov#3697)

c692a79

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot load Bloom-7b1 ggml model in GPU #3697

Cannot load Bloom-7b1 ggml model in GPU #3697

zolastro commented Oct 20, 2023

QueryType commented Oct 20, 2023

nlpcat commented Nov 2, 2023

ggerganov commented Nov 2, 2023

slaren commented Nov 3, 2023

Cannot load Bloom-7b1 ggml model in GPU #3697

Cannot load Bloom-7b1 ggml model in GPU #3697

Comments

zolastro commented Oct 20, 2023

Environment and Context

Steps to Reproduce

Failure Logs

QueryType commented Oct 20, 2023

nlpcat commented Nov 2, 2023

ggerganov commented Nov 2, 2023

slaren commented Nov 3, 2023