manylinux1 wheel no longer gets downloaded with the latest pip version

[Toad2186]

Environment info

Operating System: Ubuntu 18.04 (Docker)

CPU/GPU model: N/A

C++/Python/R version: Python: 3.7.3, gcc 7.4.0, no R.

LightGBM version or commit hash: 2.3.1 (From pip)

Error message

(Not really an error, see below)

Reproducible examples

Using the latest (20.0.1) version of pip gets you building a from-source version of lightgbm.

(test) root@269ac7781b82:# pip --version
pip 20.0.1 from /tmp/test/lib/python3.7/site-packages/pip (python 3.7)
(test) root@269ac7781b82:# pip install lightgbm
Collecting lightgbm
  Using cached lightgbm-2.3.1.tar.gz (679 kB)

Using a pre-20 version of pip gets you the prebuilt manylinux wheel:

(test) root@269ac7781b82:# pip --version
pip 19.2.2 from /tmp/test/lib/python3.7/site-packages/pip (python 3.7)
(test) root@269ac7781b82:# pip install lightgbm
Collecting lightgbm
  Using cached https://files.pythonhosted.org/packages/0b/9d/ddcb2f43aca194987f1a99e27edf41cf9bc39ea750c3371c2a62698c509a/lightgbm-2.3.1-py2.py3-none-manylinux1_x86_64.whl

I think pip changed the way it parses wheels such that py3-none-manylinux1 no longer works, but I'm not sure if this is a valid wheel configuration/name per the RFC. Note that this works:

(test) root@269ac7781b82:# pip --version
pip 20.0.1 from /tmp/test/lib/python3.7/site-packages/pip (python 3.7)
(test) root@269ac7781b82:# pip install lightgbm-2.3.1-py2.py37-none-manylinux1_x86_64.whl
Processing ./lightgbm-2.3.1-py2.py37-none-manylinux1_x86_64.whl

(Using py37 instead of py3)

Steps to reproduce

1. Install lightgbm on a manylinux1-compatible OS via pip.
2. With pip >= 20, note that it gets built from source
3. With pip < 20, note that the pre-built wheel is downloaded.

I'm not sure if this is intentional from the pip team and lightgbm needs to change, or vice versa, so starting here first.

[StrikerRUS]

@Toad2186 Thank you very much for the insight!

For wheel naming schema, please refer to https://www.python.org/dev/peps/pep-0491/#file-name-convention.

[StrikerRUS]

Also, here are some links to recent issue reports in pip repo:

• Cannot install platform-specific wheels with non-python extension binaries on pip>=20.0.0 pypa/pip#7636
• "No matching distribution" due to wheel tags not matching in pip 20.0 pypa/pip#7626

[Toad2186]

I took a look, and this PR from packaging should fix it (But needs to be merged into pip upstream):
pypa/packaging#258

tl;dr the problem is an exhausted iterator that was being reused.

Thanks for the quick reply @StrikerRUS! Doesn't seem like it's something actionable for LightGBM.

[StrikerRUS]

@Toad2186 Thank you for your investigation!

But needs to be merged into pip upstream

Inspiring a hope post from pip maintainer:

pradyunsg commented 3 hours ago
Expect a bugfix release for this tomorrow, assuming I recover from my current headache by then. :)
pypa/pip#7626 (comment)