Release a CPU only wheel on PyPI

[rbeilvert]

I am using XGBoost on CPU only and do not require the NCCL library

Until before XGBoost v2.1.0 I managed to installed the lib with pip and --no-binay option to compile the package without NCCL

Since v2.1.0, nvidia-nccl-cu12 is installed from PyPI and its huge 190MB weight makes my app not fit on my Heroku server anymore.

Is there another way to avoid the installation of this dependency than to edit the pyproject.toml file from XGBoost locally and install it locally ?

[trivialfis]

Perhaps pip install . --no-deps?

[trivialfis]

Not sure if toml project allows a different dependencies profile that overrides the default one. So far the optional dependency configuration only allows additional dependencies.

[trivialfis]

Alternatively, you can consider conda, which has greater flexibility but the version update is a bit behind.

[rbeilvert]

Perhaps pip install . --no-deps?

Not ideal as I have other packages in my requirements file for which I want to keep the dependencies, unless I create a specific Heroku buildpack for XGBoost install which sounds a bit overkill.
I'd also have to check XGBoost's pyproject.toml anyway for any shenanigans every time I update its version

Why not moving "nvidia-nccl-cu12 ; platform_system == 'Linux' and platform_machine != 'aarch64'" down into [project.optional-dependencies] ?

Having NCCL as a required dependency really penalizes CPU users

[hcho3]

It might be better to release a CPU-only package to PyPI, e.g. xgboost-cpu, for space-constrained environments.

[trivialfis]

I will leave it to @hcho3 to decide whether it's needed to maintain a fleet of packages (arm linux, x86 linux, windows, macos, arm macos, sdist) for a new pypi project.

It's a bit odd that pep517 has a way to turn on features in a package by having optional dependencies, but there's no way to turn off a feature. Maybe I'm missing something there.

[hcho3]

@trivialfis For xgboost-cpu, let's just maintain x86 linux and Windows (since we build GPU code only for these targets). No need to complicate things.

[trivialfis]

Thank you for looking into this @hcho3 . Please note that the feature has been requested for Python https://discuss.python.org/t/help-packaging-optional-application-features-using-extras/14074/18 before and there are lots of discussions. It's not surprising since packages all want to provide as many features as possible in the default build. If we were to make a new project, we should prepare for deprecating/archiving that project as well if pep517 were to support these types of optional dependencies in the future.

[hcho3]

If we were to make a new project, we should prepare for deprecating/archiving that project as well if pep517 were to support these types of optional dependencies in the future.

I agree. We can archive xgboost-cpu once the Python ecosystem provides a way to opt out of optional dependencies.

[hcho3]

@rbeilvert As a workaround, you can add the following URL to your requirements.txt:

https://files.pythonhosted.org/packages/5e/30/b4b1c071964acefeda3faade5b86e1bf2f428c5713404b212941942e835d/xgboost-2.1.0-py3-none-manylinux2014_x86_64.whl

This will install a variant of XGBoost 2.1.0 that doesn't require NCCL. Our plan is to upload this package under the name xgboost-cpu so that it's easy to locate and install.

[hcho3]

@trivialfis It turns out that I already did the necessary work to repackage XGBoost without GPU code in #10478 and #10483. So all it takes is the renaming of the package to xgboost-cpu.

[trivialfis]

It still needs to remove NCCL from pyproject right?

[hcho3]

It still needs to remove NCCL from pyproject right?

This line removes NCCL from pyproject:

xgboost/tests/buildkite/build-manylinux2014-x86_64.sh

Line 13 in e9fbce9

patch -p0 < tests/buildkite/manylinux2014_warning.patch

I will adapt the patch so that it doesn't show the warning about old glibc.

[hcho3]

The xgboost-cpu package is now available on PyPI: https://pypi.org/project/xgboost-cpu/
You can install it with pip install xgboost-cpu.

@rbeilvert @agossard-gpfw @yassinezaim-cp @kmodry @jmarichez @NicolasLegeay @mathieu-luciani @gtramoy @CyprienDuv