While using the integration of bitsandbytes, Error shows: name 'torch' is not defined

[46319943]

System Info

• transformers version: 4.41.2
• Platform: Windows-10-10.0.22631-SP0
• Python version: 3.10.14
• Huggingface_hub version: 0.23.2
• Safetensors version: 0.4.3
• Accelerate version: 0.30.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@SunMarc @youse

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I'm using the quantized version of a model. The quantization is conducted using the AWQ. After installing the AutoAWQ and run the following code:

from transformers import AutoTokenizer, AutoModelForCausalLM

revision = '991bfd84966d664085cb2c1cdb2d8b1f8526b3fa'
tokenizer = AutoTokenizer.from_pretrained("01-ai/Yi-34B-Chat-4bits", revision=revision)
model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-34B-Chat-4bits", revision=revision).to('cuda')

The error shows:

You have loaded an AWQ model on CPU and have a CUDA device available, make sure to set your model on a GPU device in order to run your model.
`low_cpu_mem_usage` was None, now set to True since model is quantized.
Traceback (most recent call last):
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\utils\import_utils.py", line 1535, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\integrations\bitsandbytes.py", line 331, in <module>
    def dequantize_bnb_weight(weight: torch.nn.Parameter, state=None):
NameError: name 'torch' is not defined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Document\Yi\quick_start.py", line 5, in <module>
    model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-34B-Chat-4bits", revision=revision).to('cuda')
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\models\auto\auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\modeling_utils.py", line 3640, in from_pretrained
    hf_quantizer.preprocess_model(
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\quantizers\base.py", line 182, in preprocess_model
    return self._process_model_before_weight_loading(model, **kwargs)
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 78, in _process_model_before_weight_loading
    from ..integrations import get_keys_to_not_convert, replace_quantization_scales, replace_with_awq_linear
  File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\utils\import_utils.py", line 1525, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "C:\Users\PiaoYang\anaconda3\envs\Yi\lib\site-packages\transformers\utils\import_utils.py", line 1537, in _get_module
    raise RuntimeError(
RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):
name 'torch' is not defined

which implies that there is something wrong about the torch.

However, after viewing the transformers\integrations\bitsandbytes.py, it can be found that the torch is imported only if the is_bitsandbytes_available() is true.
So, this is not the problem with the torch, but the installation of the bitsandbytes.

After installing the bitsandbytes pip install bitsandbytes, the error is resolved.

Expected behavior

So, the error message is confusing and misleading. It should give the error about the installation of the bitsandbytes, instead of the torch error.

[SunMarc]

Hi @46319943, thanks for report ! This issue is the same as #31243 (comment) and a fix has already been merged on main. Could you please try it again with the main branch of transformers ?

[46319943]

Wow, I encountered this problem just yesterday, and it has already been fixed! There are no issues in the main branch now.

@SunMarc, thank you for the quick response.

However, I'm still confused about the import of torch and bnb. Since many other functions in the file rely on them, shouldn't there be a check to raise an exception if bitsandbytes isn't imported?

[SunMarc]

The issue was that we didn't import torch since is_bitsandbytes_available was False. This triggered an error when python tried to read def dequantize_bnb_weight(weight: torch.nn.Parameter, state=None): since we called torch. LMK if this is clear !

[46319943]

Yes, your explanation is very clear. The existing solution uses a forward reference to avoid importing at the time of definition.

What I'm confused is that there are many other functions that use torch, and if bitsandbytes is not installed, shouldn't they raise an error because torch is not imported initially?

For example, the first function in the file, set_module_quantized_tensor_to_device, uses torch. If bitsandbytes is not installed, torch is not imported at the beginning, which should result in an import error for torch.

[SunMarc]

If torch is called inside the function, it's fine. Moreover, all the function in this file are used only when bitsandbytes is actually installed.

[46319943]

After carefully reviewing Python's import mechanism, I realized what was confusing me before.

Initially, I thought the reason why it's fine to call torch inside the function is because of Python's transitive import mechanism, where a module imported in A.py can be accessed by B.py if B.py imports A.py. However, this is not the case in Python.

The real reason it's fine is that these functions are designed to be called only if bitsandbytes is installed, and therefore torch should be imported at the beginning of the file.

I assumed that importing transformers.integrations.bitsandbytes meant that the user would use the functions inside and therefore needed bitsandbytes installed. However, as clarified in this pull request review, transformers.integrations.bitsandbytes does not require bitsandbytes to run. It is the user's responsibility to install bitsandbytes if they want to use functions that rely on it.