Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add env var related to google systems to ignored list #715

Merged
merged 1 commit into from
Jan 1, 2024

Conversation

xaptronic
Copy link
Contributor

#620

When running transformers or other applications that use bitsandbytes on some GCP resources (such as Google VMs) the following traceback is encountered:

Traceback (most recent call last):
File "/home/gcpuser/sky_workdir/vanilla.py", line 3, in <module>
  from peft import AutoPeftModelForCausalLM
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/peft/__init__.py", line 22, in <module>
  from .auto import (
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/peft/auto.py", line 30, in <module>
  from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/peft/mapping.py", line 20, in <module>
  from .peft_model import (
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/peft/peft_model.py", line 26, in <module>
  from accelerate import dispatch_model, infer_auto_device_map
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
  from .accelerator import Accelerator
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
  from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
  from .utils import (
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 132, in <module>
  from .bnb import has_4bit_bnb_layers, load_and_quantize_model
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/accelerate/utils/bnb.py", line 42, in <module>
  import bitsandbytes as bnb
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/__init__.py", line 6, in <module>
  from . import cuda_setup, utils, research
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/research/__init__.py", line 1, in <module>
  from . import nn
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in <module>
  from .modules import LinearFP8Mixed, LinearFP8Global
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/research/nn/modules.py", line 8, in <module>
  from bitsandbytes.optim import GlobalOptimManager
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/optim/__init__.py", line 6, in <module>
  from bitsandbytes.cextension import COMPILED_WITH_CUDA
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cextension.py", line 13, in <module>
  setup.run_cuda_setup()
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 120, in run_cuda_setup
  binary_name, cudart_path, cc, cuda_version_string = evaluate_cuda_setup()
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 337, in evaluate_cuda_setup
  cudart_path = determine_cuda_runtime_lib_path()
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 295, in determine_cuda_runtime_lib_path
  cuda_runtime_libs.update(find_cuda_lib_in(value))
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 232, in find_cuda_lib_in
  resolve_paths_list(paths_list_candidate)
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 227, in resolve_paths_list
  return remove_non_existent_dirs(extract_candidate_paths(paths_list_candidate))
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 201, in remove_non_existent_dirs
  raise exc
File "/opt/conda/envs/vanilla/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py", line 197, in remove_non_existent_dirs
  if path.exists():
File "/opt/conda/envs/vanilla/lib/python3.10/pathlib.py", line 1290, in exists
  self.stat()
File "/opt/conda/envs/vanilla/lib/python3.10/pathlib.py", line 1097, in stat
  return self._accessor.stat(self, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/root/google_vm_config.lock'

The cause appears to be that bitsandbytes is searching (I think) for Cuda drivers by scanning environment variables for strings that look like file paths.

/root/google_vm_config.lock is found in GOOGLE_VM_CONFIG_LOCK_FILE and is probably not needed to be searched.

@xaptronic
Copy link
Contributor Author

Fixes the same issue as: #622

Unsure, which is better, but if that is merged, this can be closed.

@dcaustin33
Copy link

Running into this issue a lot recently - would love to have this merged!

@pculliton
Copy link

This issue is impacting our ability to deploy BnB on Vertex. Would be really happy to see it merged.

@gabegrand
Copy link

Agreed, this is a no-brainer. Very lightweight PR that fixes a common issue on GCP. Would love to see this merged.

Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

@github-actions github-actions bot closed this Dec 30, 2023
@TimDettmers
Copy link
Collaborator

Thank for for this PR! Sorry it took so long.

@TimDettmers TimDettmers reopened this Jan 1, 2024
@TimDettmers TimDettmers merged commit 2fd74f3 into bitsandbytes-foundation:main Jan 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants