[RFC] [CUDA] distribute Python wheels with CUDA support

[jameslamb]

Summary

Using the lightgbm Python package with CUDA support currently requires either installing it with conda or building it from source.

LightGBM should provide pre-compiled wheels supporting at least the latest major version of CUDA (CUDA 12 as of this writing).

I am opening this as a tracking issue for this feature, and to have a public discussion about this topic.

Benefits of this work

• would make it easier for users who do not want to use conda to use the CUDA version, without needing a build toolchain
• would help encourage more migration away from the (currently unmaintained) OpenCL-based version to the newer, better-maintained CUDA version

Acceptance Criteria

• CUDA wheels for at least Linux (and ideally both x86_64 and aarch64) are:
  • built and tested in CI
  • published to PyPI

Approach

Some design choices need to be made:

• compile support into lightgbm (https://pypi.org/project/lightgbm/), or distribute under a separate name like lightgbm-cu12?
• what range of CUDA architectures to support?
  • affects whether wheels are larger than 100MiB, and therefore require exceptions like File Limit Request: XGBoost - 300 MB pypi/support#2146

See some prior discussions from xgboost on implementation details

• [CI] Upload wheel variants for CUDA 11 and 12 dmlc/xgboost#10807
• [CI] Use rapids-dependency-file-generator to generate pyproject.toml dmlc/xgboost#10803
• Release a CPU only wheel on PyPI dmlc/xgboost#10596

Notes

Issues people have had with building from source:

• [python-package] [CUDA] installing CUDA version with 'pip' hangs, consume all the memory (32 G) and swap space #6824
• [CUDA] GPU build on Ubuntu 20.04 crashing on training #6705 (double-posted to https://stackoverflow.com/questions/79139271/how-to-install-lightgbm-on-ubuntu-20-04-with-cuda-support)
• Lightgbm does not work on GPU? #6205
• Compiling with flags USE_CUDA, USE_GPU and USE_DEBUG fails #5777
• https://stackoverflow.com/questions/77728334/install-lightgbm-gpu-in-a-wsl-conda-env
• https://stackoverflow.com/questions/77284293/making-lightgbm-work-with-gpu-on-google-colab
• https://stackoverflow.com/questions/49606886/installing-gpu-support-for-lightgbm-on-google-colab/77078844

[StrikerRUS]

Related: #6539 (comment)

[shiyu1994]

Thanks for bringing this up. I agree that we should constrain our supported CUDA version range. For now, the built CUDA version could be larger than 500M.