Tries to access root lock file

[lakshmanok]

Result of python -m bitsandbytes:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/conda/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/conda/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 13, in <module>
    setup.run_cuda_setup()
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 120, in run_cuda_setup
    binary_name, cudart_path, cc, cuda_version_string = evaluate_cuda_setup()
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 337, in evaluate_cuda_setup
    cudart_path = determine_cuda_runtime_lib_path()
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 295, in determine_cuda_runtime_lib_path
    cuda_runtime_libs.update(find_cuda_lib_in(value))
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 232, in find_cuda_lib_in
    resolve_paths_list(paths_list_candidate)
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 227, in resolve_paths_list
    return remove_non_existent_dirs(extract_candidate_paths(paths_list_candidate))
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 201, in remove_non_existent_dirs
    raise exc
  File "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 197, in remove_non_existent_dirs
    if path.exists():
  File "/opt/conda/lib/python3.10/pathlib.py", line 1290, in exists
    self.stat()
  File "/opt/conda/lib/python3.10/pathlib.py", line 1097, in stat
    return self._accessor.stat(self, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/root/google_vm_config.lock'

The problem persists even after compiling from source. I'm running in a Jupyter notebook (Google Cloud Vertex AI)

[erlakshmi123]

hit the same issue. w/o is to catch the errno==13 and skip

[hanneshapke]

Same here. It happens on versions 0.40.1 and 0.40.2 (I have yet to test older versions).
It occurs in a Google Cloud compute instance (plain instance, not Vertex).

[lakshmanok]

My work around is to run as root (obviously not great) thanks, Lak

…

On Fri, Jul 21, 2023, 8:35 AM Hannes Hapke ***@***.***> wrote: Same here. It happens on versions 0.40.1 and 0.40.2 (I have yet to test older versions). It occurs in a Google Cloud compute instance (plain instance, not Vertex). — Reply to this email directly, view it on GitHub <#620 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANJPZ5KS7DBZGPCQXVLOOLXRKOVPANCNFSM6AAAAAA2SAXQKY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[hanneshapke]

The permission error occurred when I tried to load a script. I narrowed down the imports to

from transformers import TrainingArguments

and then the error was more specific

RuntimeError: Failed to import transformers.training_args because of the following error (look up to see its traceback):
No module named 'scipy'

The error got resolved after installing scipy and switching the base instance image from common-cu113 to pytorch-latest-gpu.

[lakshmanok]

Thanks, Hannes! Looks like this was fixed 4 days ago: 8a20cd8 Curious why it isn't fixed when you install from source.

[hanneshapke]

Thank you, @lakshmanok, for pointing out the change. I tried to install it from source (before learning about the missing scipy), but it didn't resolve the issue for me. I suspect I messed something else up. Looking at the commit, it should fix the issue.

[muellerzr]

Also seeing this issue on my side, let me know if there's anything I can do to help with a clearer report. Also: installing scipy did not work (already installed), I'll attempt using pytorch-latest-gpu as I was indeed using Google, Debian 10 based Deep Learning VM with , M109, Base CUDA 11.0, Deep Learning VM Image with CUDA 11.0 preinstalled. (or common-cu110)

[muellerzr]

Actually @hanneshapke do you know the exact image name (that's findable when starting a new instance) for pytorch-latest-gpu?

[xaptronic]

I was able to get around this by

adding GOOGLE_VM_CONFIG_LOCK_FILE

to the set ignorable inside def to_be_ignored in this file lib/python3.10/site-packages/bitsandbytes/cuda_setup/env_vars.py

happy to pr this if it's a right soln - this would likely only affect GCP setups.

[karthikcs]

GOOGLE_VM_CONFIG_LOCK_FILE

This is great! It fixed the issue. Please push to the main branch..

[ckgresla]

PR is up- #785

[xaptronic]

PR is up- #785

Duplicates #715 ?

[madhavthaker1]

@xaptronic -- Any ideas on how to address this when working in a vertex workbench? I don't have access to lib/python3.10/site-packages/bitsandbytes/cuda_setup/env_vars.py

[samlhuillier]

@madhavthaker1 that's not necessarily where the bitsandbytes config will be. It depends on your system/virtual envs etc. Run pip show bitsandbytes to find that path.

[rishabhmehrotra]

@madhavthaker1 on vertex workbench, it might be in a path like "/opt/conda/lib/python3.10/site-packages/bitsandbytes/cuda_setup/env_vars.py"

[Titus-von-Koeller]

Thanks all for debugging and providing the fix! Closing this, as the fix is now merged. Will be made available with the next release.