Investigate being unable to find cublas_v2.h

[jeongseok-meta]

https://github.com/pytorch/pytorch/blob/v2.5.1/cmake/Modules/FindCUDAToolkit.cmake#L809

  -- Unable to find cublas_v2.h in either "/home/conda/feedstock_root/build_artifacts/libtorch_1730670943607/_build_env/targets/sbsa-linux/include" or "/home/conda/feedstock_root/build_artifacts/libtorch_1730670943607/_build_env/math_libs/include"

Affected builds:

• linux64 + mkl + cuda 12.0/12.6
• aarch64 + cuda 12.0/12.6

Related:

• [BUG+FIX] LibTorch silently fails to find cublas_v2.h for NVHPC pytorch/pytorch#132599

Originally posted by @hmaarrfk in #271 (comment)

[hmaarrfk]

this was in the aarch64 + cuda 12.6 build log.

[jeongseok-meta]

FWIW, this issue is not new, as it also occurred in a build log from two weeks ago (randomly picked): https://github.com/conda-forge/pytorch-cpu-feedstock/actions/runs/11317968091/job/31472007572

[hmaarrfk]

It seems like it is available in the libcublas-dev package since at least 12.0
https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-64%2Flibcublas-dev-12.0.1.189-hd3aeb46_3.conda

[jakirkham]

It is worth noting the paths in the error are from the build environment and not the host environment. Do we know why it isn't using the latter?

[mgorny]

Indeed looks like CMake determines CUDA path based on nvcc location, and our overrides don't seem to work. I'm going to try figuring out if we can override this properly without patching, or prepare a patch to fix it.

[mgorny]

Good news, everyone! It seems that I have a fix, it wasn't as horrible as I've anticipated, and additionally it seems to remove the need for build dependencies on the additional CUDA libraries. Not creating a PR yet, as I don't want to torture the CI: mgorny/pytorch-cpu-feedstock@cufile...cuda-prefix