[PermissionDeniedError] : bitsandbytes 0.41 pointing to incorrect version of libcudart.so

[greeshmasmenon]

bitsandbytes 0.41 has been installed and I am getting the Permission Denied error coming as seen below. However, the installation is in another place and i do not have sudo permissions to change the current place it is referenced to.

0 [gmenon@iris-180 ~](3191390 2N/4T/2(x2)CN)$ python3 -m bitsandbytesTraceback (most recent call last):
  File "/opt/apps/resif/iris-rhel8/2020b/gpu/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/opt/apps/resif/iris-rhel8/2020b/gpu/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/opt/apps/resif/iris-rhel8/2020b/gpu/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cextension.py", line 13, in <module>
    setup.run_cuda_setup()
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 120, in run_cuda_setup
    binary_name, cudart_path, cc, cuda_version_string = evaluate_cuda_setup()
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 337, in evaluate_cuda_setup
    cudart_path = determine_cuda_runtime_lib_path()
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 295, in determine_cuda_runtime_lib_path
    cuda_runtime_libs.update(find_cuda_lib_in(value))
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 231, in find_cuda_lib_in
    return get_cuda_runtime_lib_paths(
  File "/home/users/gmenon/.local/lib/python3.8/site-packages/bitsandbytes/cuda_setup/main.py", line 217, in get_cuda_runtime_lib_paths
    if (path / libname).is_file():
  File "/opt/apps/resif/iris-rhel8/2020b/gpu/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/pathlib.py", line 1423, inis_file
    return S_ISREG(self.stat().st_mode)
  File "/opt/apps/resif/iris-rhel8/2020b/gpu/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/pathlib.py", line 1197, instat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/usr/share/lmod/lmod/init/ksh_funcs/libcudart.so'

I have used the cuda_installer.sh file to install cuda-12.0 in the /local path and have updated the .bashrc to reference this path

0 [gmenon@iris-180 lib](3191390 2N/4T/2(x2)CN)$ pwd
/local/cuda-12.0/targets/x86_64-linux/lib


0 [gmenon@iris-180 lib](3191390 2N/4T/2(x2)CN)$ ls
cmake                         libcufile_rdma.so.1          libcusparse_static.a     libnppisu_static.a
libcublasLt.so                libcufile_rdma.so.1.5.0      libmetis_static.a        libnppitc_static.a
libcublasLt.so.12             libcufile_rdma_static.a      libnppc.so.12.0.0.30     libnpps_static.a
libcublasLt.so.12.0.1.189     libcufile.so                 libnppc_static.a         libnvblas.so
libcublasLt_static.a          libcufile.so.0               libnppial.so             libnvblas.so.12
libcublas.so                  libcufile.so.1.5.0           libnppial.so.12          libnvblas.so.12.0.1.189
libcublas.so.12               libcufile_static.a           libnppial.so.12.0.0.30   libnvJitLink_static.a
libcublas.so.12.0.1.189       libculibos.a                 libnppial_static.a       libnvjpeg_static.a
libcublas_static.a            libcurand.so                 libnppicc.so             libnvrtc-builtins.so
libcudadevrt.a                libcurand.so.10              libnppicc.so.12          libnvrtc-builtins.so.12.0
libcudart.so                  libcurand.so.10.3.1.50       libnppicc.so.12.0.0.30   libnvrtc-builtins.so.12.0.76
libcudart.so.12               libcurand_static.a           libnppicc_static.a       libnvrtc-builtins_static.a
libcudart.so.12.0.107         libcusolver_lapack_static.a  libnppidei.so.12.0.0.30  libnvrtc.so
libcudart_static.a            libcusolver_metis_static.a   libnppidei_static.a      libnvrtc.so.12libcufft.so                   libcusolverMg.so             libnppif_static.a        libnvrtc.so.12.0.76
libcufft.so.11                libcusolverMg.so.11          libnppig_static.a        libnvrtc_static.a
libcufft.so.11.0.0.21         libcusolverMg.so.11.4.2.57   libnppim.so              libOpenCL.so
libcufft_static.a             libcusolver.so               libnppim.so.12           libOpenCL.so.1
libcufft_static_nocallback.a  libcusolver.so.11            libnppim.so.12.0.0.30    libOpenCL.so.1.0
libcufftw.so                  libcusolver.so.11.4.2.57     libnppim_static.a        libOpenCL.so.1.0.0
libcufftw.so.11               libcusolver_static.a         libnppist.so             stubs
libcufftw.so.11.0.0.21        libcusparse.so               libnppist.so.12
libcufftw_static.a            libcusparse.so.12            libnppist.so.12.0.0.30
libcufile_rdma.so             libcusparse.so.12.0.0.76     libnppist_static.a

.bashrc file

# .bashrc

#Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# User specific aliases and functions
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/local/cuda-12.0/targets/x86_64-linux/lib
export PATH=$PATH:/local/cuda-12.0/bin

Please help in resolving this. Scheduling is done through SLURM. The error shown above is coming inspite of any changes.

[fozziethebeat]

I think this PR should fix this problem. I had a similar issue: #644

[greeshmasmenon]

Thanks for the response. When will it be merged?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.