Getting error: undefined symbol: cget_col_row_stats

[webdevtodayjason]

Here is the error:
AttributeError: /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol:
cget_col_row_stats

I am running on a Ubuntu 22.04 WHL Subsystem using conda.
I am starting this with:
python generate.py --base_model=h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v3 --load_8bit=True --score_model=None --langchain_mode='UserData' --user_path=user_path

How can I resolve this?

[pseudotensor]

Similar here: #409

You are using Linux x86_64 right? Seems like your cuda is not working correctly.

Try running:

conda install cudatoolkit-dev

and then see if PEFT finds the _gpu.so file instead of going to CPU. As for why CPU says that, that's yet another issue I have not seen, but is also mentioned at that link: bitsandbytes-foundation/bitsandbytes#156 (comment)

[webdevtodayjason]

Yea it is a Linux Sub System in Windows 11.
I have this installed and still get this:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/jason/h2ogpt/generate.py:16 in <module>                                                    │
│                                                                                                  │
│   13                                                                                             │
│   14                                                                                             │
│   15 if __name__ == "__main__":                                                                  │
│ ❱ 16 │   entrypoint_main()                                                                       │
│   17                                                                                             │
│                                                                                                  │
│ /home/jason/h2ogpt/generate.py:12 in entrypoint_main                                             │
│                                                                                                  │
│    9                                                                                             │
│   10                                                                                             │
│   11 def entrypoint_main():                                                                      │
│ ❱ 12 │   fire.Fire(main)                                                                         │
│   13                                                                                             │
│   14                                                                                             │
│   15 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py:141 in Fire         │
│                                                                                                  │
│   138 │   context.update(caller_globals)                                                         │
│   139 │   context.update(caller_locals)                                                          │
│   140                                                                                            │
│ ❱ 141   component_trace = _Fire(component, args, parsed_flag_args, context, name)                │
│   142                                                                                            │
│   143   if component_trace.HasError():                                                           │
│   144 │   _DisplayError(component_trace)                                                         │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py:475 in _Fire        │
│                                                                                                  │
│   472 │     is_class = inspect.isclass(component)                                                │
│   473 │                                                                                          │
│   474 │     try:                                                                                 │
│ ❱ 475 │   │   component, remaining_args = _CallAndUpdateTrace(                                   │
│   476 │   │   │   component,                                                                     │
│   477 │   │   │   remaining_args,                                                                │
│   478 │   │   │   component_trace,                                                               │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/fire/core.py:691 in              │
│ _CallAndUpdateTrace                                                                              │
│                                                                                                  │
│   688 │   loop = asyncio.get_event_loop()                                                        │
│   689 │   component = loop.run_until_complete(fn(*varargs, **kwargs))                            │
│   690   else:                                                                                    │
│ ❱ 691 │   component = fn(*varargs, **kwargs)                                                     │
│   692                                                                                            │
│   693   if treatment == 'class':                                                                 │
│   694 │   action = trace.INSTANTIATED_CLASS                                                      │
│                                                                                                  │
│ /home/jason/h2ogpt/src/gen.py:626 in main                                                        │
│                                                                                                  │
│    623 │   │   │   all_kwargs.update(dict(base_model=base_model1, tokenizer_base_model=tokenize  │
│    624 │   │   │   │   │   │   │   │      lora_weights=lora_weights1, inference_server=inferenc  │
│    625 │   │   │   if base_model1 and not login_mode_if_model0:                                  │
│ ❱  626 │   │   │   │   model0, tokenizer0, device = get_model(reward_type=False,                 │
│    627 │   │   │   │   │   │   │   │   │   │   │   │   │      **get_kwargs(get_model, exclude_n  │
│    628 │   │   │   │   │   │   │   │   │   │   │   │   │   │   │   │   │   **all_kwargs))        │
│    629 │   │   │   else:                                                                         │
│                                                                                                  │
│ /home/jason/h2ogpt/src/gen.py:961 in get_model                                                   │
│                                                                                                  │
│    958 │   │   return model, tokenizer, device                                                   │
│    959 │                                                                                         │
│    960 │   # get local torch-HF model                                                            │
│ ❱  961 │   return get_hf_model(load_8bit=load_8bit,                                              │
│    962 │   │   │   │   │   │   load_4bit=load_4bit,                                              │
│    963 │   │   │   │   │   │   load_half=load_half,                                              │
│    964 │   │   │   │   │   │   load_gptq=load_gptq,                                              │
│                                                                                                  │
│ /home/jason/h2ogpt/src/gen.py:1081 in get_hf_model                                               │
│                                                                                                  │
│   1078 │   │   │   │                                                                             │
│   1079 │   │   │   │   if use_gpu_id:                                                            │
│   1080 │   │   │   │   │   config, model = get_config(base_model, return_model=True, raise_exce  │
│ ❱ 1081 │   │   │   │   │   model = get_non_lora_model(base_model, model_loader, load_half, load  │
│   1082 │   │   │   │   │   │   │   │   │   │   │      model_kwargs, reward_type,                 │
│   1083 │   │   │   │   │   │   │   │   │   │   │      config, model,                             │
│   1084 │   │   │   │   │   │   │   │   │   │   │      gpu_id=gpu_id,                             │
│                                                                                                  │
│ /home/jason/h2ogpt/src/gen.py:786 in get_non_lora_model                                          │
│                                                                                                  │
│    783 │   │   │   **model_kwargs,                                                               │
│    784 │   │   )                                                                                 │
│    785 │   elif load_in_8bit or load_in_4bit or not load_half:                                   │
│ ❱  786 │   │   model = model_loader(                                                             │
│    787 │   │   │   base_model,                                                                   │
│    788 │   │   │   config=config,                                                                │
│    789 │   │   │   **model_kwargs,                                                               │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/models/auto/auto_fa │
│ ctory.py:479 in from_pretrained                                                                  │
│                                                                                                  │
│   476 │   │   │   │   class_ref, pretrained_model_name_or_path, **hub_kwargs, **kwargs           │
│   477 │   │   │   )                                                                              │
│   478 │   │   │   _ = hub_kwargs.pop("code_revision", None)                                      │
│ ❱ 479 │   │   │   return model_class.from_pretrained(                                            │
│   480 │   │   │   │   pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs,   │
│   481 │   │   │   )                                                                              │
│   482 │   │   elif type(config) in cls._model_mapping.keys():                                    │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/modeling_utils.py:2 │
│ 881 in from_pretrained                                                                           │
│                                                                                                  │
│   2878 │   │   │   │   mismatched_keys,                                                          │
│   2879 │   │   │   │   offload_index,                                                            │
│   2880 │   │   │   │   error_msgs,                                                               │
│ ❱ 2881 │   │   │   ) = cls._load_pretrained_model(                                               │
│   2882 │   │   │   │   model,                                                                    │
│   2883 │   │   │   │   state_dict,                                                               │
│   2884 │   │   │   │   loaded_state_dict_keys,  # XXX: rename?                                   │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/modeling_utils.py:3 │
│ 228 in _load_pretrained_model                                                                    │
│                                                                                                  │
│   3225 │   │   │   │   )                                                                         │
│   3226 │   │   │   │                                                                             │
│   3227 │   │   │   │   if low_cpu_mem_usage:                                                     │
│ ❱ 3228 │   │   │   │   │   new_error_msgs, offload_index, state_dict_index = _load_state_dict_i  │
│   3229 │   │   │   │   │   │   model_to_load,                                                    │
│   3230 │   │   │   │   │   │   state_dict,                                                       │
│   3231 │   │   │   │   │   │   loaded_keys,                                                      │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/modeling_utils.py:7 │
│ 28 in _load_state_dict_into_meta_model                                                           │
│                                                                                                  │
│    725 │   │   │   │   fp16_statistics = None                                                    │
│    726 │   │   │                                                                                 │
│    727 │   │   │   if "SCB" not in param_name:                                                   │
│ ❱  728 │   │   │   │   set_module_quantized_tensor_to_device(                                    │
│    729 │   │   │   │   │   model, param_name, param_device, value=param, fp16_statistics=fp16_s  │
│    730 │   │   │   │   )                                                                         │
│    731                                                                                           │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/utils/bitsandbytes. │
│ py:89 in set_module_quantized_tensor_to_device                                                   │
│                                                                                                  │
│    86 │   │   │                                                                                  │
│    87 │   │   │   kwargs = old_value.__dict__                                                    │
│    88 │   │   │   if is_8bit:                                                                    │
│ ❱  89 │   │   │   │   new_value = bnb.nn.Int8Params(new_value, requires_grad=False, **kwargs).   │
│    90 │   │   │   elif is_4bit:                                                                  │
│    91 │   │   │   │   new_value = bnb.nn.Params4bit(new_value, requires_grad=False, **kwargs).   │
│    92                                                                                            │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py:294   │
│ in to                                                                                            │
│                                                                                                  │
│   291 │   │   │   and device.type == "cuda"                                                      │
│   292 │   │   │   and self.data.device.type == "cpu"                                             │
│   293 │   │   ):                                                                                 │
│ ❱ 294 │   │   │   return self.cuda(device)                                                       │
│   295 │   │   else:                                                                              │
│   296 │   │   │   new_param = Int8Params(                                                        │
│   297 │   │   │   │   super().to(                                                                │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/nn/modules.py:258   │
│ in cuda                                                                                          │
│                                                                                                  │
│   255 │   │   │   # we store the 8-bit rows-major weight                                         │
│   256 │   │   │   # we convert this weight to the turning/ampere weight during the first infer   │
│   257 │   │   │   B = self.data.contiguous().half().cuda(device)                                 │
│ ❱ 258 │   │   │   CB, CBt, SCB, SCBt, coo_tensorB = bnb.functional.double_quant(B)               │
│   259 │   │   │   del CBt                                                                        │
│   260 │   │   │   del SCBt                                                                       │
│   261 │   │   │   self.data = CB                                                                 │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/functional.py:1987  │
│ in double_quant                                                                                  │
│                                                                                                  │
│   1984 │   │   rows = A.shape[0]                                                                 │
│   1985 │                                                                                         │
│   1986 │   if row_stats is None or col_stats is None:                                            │
│ ❱ 1987 │   │   row_stats, col_stats, nnz_row_ptr = get_colrow_absmax(                            │
│   1988 │   │   │   A, threshold=threshold                                                        │
│   1989 │   │   )                                                                                 │
│   1990                                                                                           │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/functional.py:1876  │
│ in get_colrow_absmax                                                                             │
│                                                                                                  │
│   1873 │                                                                                         │
│   1874 │   prev_device = pre_call(A.device)                                                      │
│   1875 │   is_on_gpu([A, row_stats, col_stats, nnz_block_ptr])                                   │
│ ❱ 1876 │   lib.cget_col_row_stats(ptrA, ptrRowStats, ptrColStats, ptrNnzrows, ct.c_float(thresh  │
│   1877 │   post_call(prev_device)                                                                │
│   1878 │                                                                                         │
│   1879 │   if threshold > 0.0:                                                                   │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/ctypes/__init__.py:387 in __getattr__          │
│                                                                                                  │
│   384 │   def __getattr__(self, name):                                                           │
│   385 │   │   if name.startswith('__') and name.endswith('__'):                                  │
│   386 │   │   │   raise AttributeError(name)                                                     │
│ ❱ 387 │   │   func = self.__getitem__(name)                                                      │
│   388 │   │   setattr(self, name, func)                                                          │
│   389 │   │   return func                                                                        │
│   390                                                                                            │
│                                                                                                  │
│ /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/ctypes/__init__.py:392 in __getitem__          │
│                                                                                                  │
│   389 │   │   return func                                                                        │
│   390 │                                                                                          │
│   391 │   def __getitem__(self, name_or_ordinal):                                                │
│ ❱ 392 │   │   func = self._FuncPtr((name_or_ordinal, self))                                      │
│   393 │   │   if not isinstance(name_or_ordinal, int):                                           │
│   394 │   │   │   func.__name__ = name_or_ordinal                                                │
│   395 │   │   return func                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: /home/jason/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol:
cget_col_row_stats

[pseudotensor]

I see, I wasn't aware you were running in linux in windows. The bitsandbytes are quite specific to the underlying hardware/OS.

Maybe try AutoGPTQ instead of bitsandbytes. i.e. pip uninstall bitsandbytes -y

Or try installation closer to windows instructions when it comes to bitsandbytes. I'm unsure if any existing bitsandbytes works on linux with underlying windows OS.

[pseudotensor]

I revamped and fully tested the docs for windows, please try again: https://github.com/h2oai/h2ogpt/blob/main/docs/README_WINDOWS.md