想问下是否可以支持cpu的部署

[joebnb]

如题，是否支持cpu部署

[duzx16]

可以。将demo中的.half().cuda()换成.float()，即可，但是推理速度会比较慢

[joebnb]

好的，谢谢啦，建议文档中更新下。

[wbs306]

我替换成 .float() 之后显示 FileNotFoundError: Could not find module 'nvcuda.dll' (or one of its dependencies). Try using the full path with constructor syntax. 这个错误，电脑上没有独显，这个是完整的报错：

Traceback (most recent call last):
  File "D:\Workspace\ChatGLM-6B\cli_demo.py", line 6, in <module>
    model = AutoModel.from_pretrained("THUDM/chatglm-6b", trust_remote_code=True, mirror='https://mirror.nju.edu.cn/hugging-face-models').float()
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\transformers\models\auto\auto_factory.py", line 455, in from_pretrained
    model_class = get_class_from_dynamic_module(
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\transformers\dynamic_module_utils.py", line 363, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\transformers\dynamic_module_utils.py", line 274, in get_cached_module_file
    get_cached_module_file(
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\transformers\dynamic_module_utils.py", line 237, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\transformers\dynamic_module_utils.py", line 129, in check_imports
    importlib.import_module(imp)
  File "D:\Program Files\Python310\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\cpm_kernels\__init__.py", line 1, in <module>
    from . import library
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\cpm_kernels\library\__init__.py", line 2, in <module>
    from . import cuda
  File "C:\Users\Wbs\AppData\Roaming\Python\Python310\site-packages\cpm_kernels\library\cuda.py", line 7, in <module>
    cuda = Lib.from_lib("cuda", ctypes.WinDLL("nvcuda.dll"))
  File "D:\Program Files\Python310\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'nvcuda.dll' (or one of its dependencies). Try using the full path with constructor syntax.

[brealisty]

可以。将demo中的.half().cuda()换成.float()，即可，但是推理速度会比较慢

@duzx16 cpu模式下运行，内存占用24G左右，是否合理？