pip's lack of support for PEP 508 URLs in requirements.txt / argument breaks pep517

[ghost]

Environment

• pip version: 18.1 (pretty sure newer ones are affected as well)
• Python version: 3.7.2
• OS: linux

Description
pip's lack of support for PEP 508 URLs in requirements.txt / argument breaks pep517: any setuptools package that will have them in setup_requires will result in pep517's envbuild.py attempting to install it with pip: https://github.com/pypa/pep517/blob/970f48beebf29b0831a3a067da4a0a3daac20193/pep517/envbuild.py#L135 - and this doesn't work, because pip doesn't take the PEP 508 URLs.

However, using these URLs in setup_requires is kinda mandatory now, because it is the only way to reference a not-released-to-pypi-yet git project in a way that installs properly without using --process-dependency-links. And it works splendidly - unless you try to use pep517, then the perfectly installable package suddenly will fail to be processed properly due to this issue.

A different, very annoying side effect of this is that my setup.py needs to process requirements.txt to change the url format because I need non-PEP 508 URLs in setup.py due to this limitation, but PEP 508 URLs in setup_requires, and I didn't want to store it twice in two places.

Related PEP 517 ticket: pypa/pyproject-hooks#43

Expected behavior
requirements.txt and other arguments of pip take PEP 508 URL format

How to Reproduce

1. Write a requirements.txt e.g. with wobblui @ https://github.com/wobblui/wobblui/archive/master.zip
2. Attempt install via pip install -r requirements.txt

Output

$ pip3 install -r requirements.txt 
Requirement 'wobblui @ https://github.com/wobblui/wobblui/archive/master.zip' looks like a filename, but the file does not exist
Processing ./wobblui @ https:/github.com/wobblui/wobblui/archive/master.zip
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/jonas/testbla/wobblui @ https:/github.com/wobblui/wobblui/archive/master.zip'

[cjerdonek]

Is this a duplicate of #6097 or just related, or does one depend on the other being fixed?

[cjerdonek]

See also #6202 where PR #6203 has been submitted to resolve this.

[cjerdonek]

@pradyunsg Should this still be labeled a vendored dependency given e.g. @pfmoore's comment here: pypa/pyproject-hooks#43 (comment) ?

[pfmoore]

@cjerdonek I don't believe it should, so I removed that label.

[pfmoore]

From the description, I believe this is partially the same as #6097. But this issue also claims that pip doesn't support PEP 508 URLs in command line arguments. I haven't verified that, but I have no reason to believe it's not the case.

[ghost]

I just tested it to make sure I wasn't just imagining it, and command line arguments don't seem to like it either:

$ pip3 install wobblui@https://github.com/wobblui/wobblui/archive/master.zip
Requirement 'wobblui@https://github.com/wobblui/wobblui/archive/master.zip' looks like a filename, but the file does not exist
Processing ./wobblui@https:/github.com/wobblui/wobblui/archive/master.zip
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/jonas/wobblui@https:/github.com/wobblui/wobblui/archive/master.zip'

[pfmoore]

I don't think it'll help, but IIRC, the syntax requires spaces around the @. On the other hand, pip will happily install https://github.com/wobblui/wobblui/archive/master.zip so there's no immediately obvious reason why you'd need the PEP 508 format in this case.

[pradyunsg]

Should this still be labeled a vendored dependency

Nope. Thanks @pfmoore for handling this! :D

[ghost]

@cjerdonek yeah pretty sure #6097 is the same and this could be considered a duplicate just with more information on other indirectly caused trouble (that being pep517's buildenv module breaking)

edit: ...since I believe pep517 passes via requirements.txt (so the argument issue also mentioned here is probably irrelevant for any practical purpose as already pointed out by @pfmoore ) although I suppose you could leave the ticket open for that

[cjerdonek]

I don't think it'll help, but IIRC, the syntax requires spaces around the @.

@pfmoore FWIW, I just learned myself from one of the other issues that spaces aren't required here. It's only after an url_reqthat a space is required (e.g. before a semicolon). For example (from PEP 508):

Non line-breaking whitespace is mostly optional with no semantic meaning. The sole exception is detecting the end of a URL requirement.

@Jonast Thanks. Then what I'll do is close this issue, and hopefully you can keep an eye on the other two issues #6097 and #6202 and check when those issues are closed that your issue is resolved. It could also help if you could review PR's before they're merged and/or manually check from master that any proposed PR addresses it.

[cjerdonek]

Closing as a duplicate of #6097 (and possibly #6202).

[pfmoore]

I just learned myself from one of the other issues that spaces aren't required here.

Yeah, I just spotted that today as well. I wonder where I got the impression spaces were required? It's a weird constraint to have whitespace be significant, and I doubt I'd come up with it myself. Oh, well, at least I know the truth now :-)

[cjerdonek]

Yeah, I just spotted that today as well. I wonder where I got the impression spaces were required?

I went through the exact same thought process as you as I remembered seeing something before, too. I tracked down the issue, and it was this comment that put this notion in my head: #5903 (comment)

wheel generate package @ URL; markers, which is invalid, the correct syntax is package @ URL ; markers, note the space after the URL before ;

When I read it quickly at the time, I registered it as being in reference to the @ symbol, but it was about the semicolon ;.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.