Support relative paths for file links. #4208

blr246 · 2016-12-28T16:07:10Z

When searching paths in the InstallRequirement class, use
os.path.abspath(self.source_dir) whenever referring directly to an
artifact in the filesystem. Doing so supports relative file paths
properly.

The SETUPTOOLS_SHIM compiles and executes a setup.py for an editable
file link. The form of the command opens directly the path from
file:<path> links while also setting cwd=<path>. This may break
relative paths where the dependent project is at a different depth in a
source tree from the referenced requirement. In that case, the opened
file <path> relative to the relative path <path> will not align,
leading to an OSError.

This change is

When searching paths in the `InstallRequirement` class, use `os.path.abspath(self.source_dir)` whenever referring directly to an artifact in the filesystem. Doing so supports relative file paths properly. The `SETUPTOOLS_SHIM` compiles and executes a `setup.py` for an editable file link. The form of the command opens directly the path from `file:<path>` links while also setting `cwd=<path>`. This may break relative paths where the dependent project is at a different depth in a source tree from the referenced requirement. In that case, the opened file `<path>` relative to the relative path `<path>` will not align, leading to an `OSError`.

dstufft

Hi! Can you please add a test to ensure that this behavior doesn't change in the future? In addition you'll also need to add a news file (see our documentation for an example of how to do that).

blr246 · 2017-04-05T13:55:19Z

Thanks so much @dstufft for the reply. I will gladly add a news file and test case. Please standby for revisions within the next day or two.

blr246 · 2017-04-05T16:01:02Z

@dstufft added news file and test. Confirmed that new test fails on current master commit (491294f).

xavfernandez

Looks good.

xavfernandez · 2017-04-05T16:15:12Z

tests/functional/test_install.py

+
+        # Install from relative path using direct pip invocation.
+        result = script.pip('install', to_install_relative,
+                            expect_error=False, cwd=my_dir)


expect_error already defaults to False

xavfernandez · 2017-04-05T16:18:59Z

tests/functional/test_install.py

+
+    # Create directories in scratch of varying depth to test relative path to
+    # requirements.
+    for relative_depth in range(3):


I'd say, no need to test to the third level, 2 seems enough ( the test suite is already quite slow).

xavfernandez · 2017-04-05T16:19:55Z

tests/functional/test_install.py

+        # Install from relative path using direct pip invocation.
+        result = script.pip('install', to_install_relative,
+                            expect_error=False, cwd=my_dir)
+        assert (fspkg_folder in result.files_created or


The or is suspect, why is it needed ?

xavfernandez · 2017-04-05T16:20:34Z

tests/functional/test_install.py

+                    reqs_fmt.format(rel_dir=to_install_relative),
+                    my_dir) as reqs_file:
+                result = script.pip_install_local('-r', reqs_file.name,
+                                                  expect_error=False,


again, no need for expect_error=False. That's what we expect everywhere.

@xavfernandez

Responding to comments from @xavfernandez.

xavfernandez · 2017-04-05T16:40:12Z

You seem to have a consistent failure on python >=3.5 cf https://travis-ci.org/pypa/pip/builds/218937396

blr246 · 2017-04-05T17:16:20Z

@xavfernandez looks like it's passing with latest patch:
https://travis-ci.org/pypa/pip/builds/218949546

The failures were all in Popen.communicate(). I just looked at the trace from scripttest.TestFileEnvironment and didn't see anything jump out. I installed python3.6 locally and could not reproduce the error.

Let me know if there's any further investigation you think is necessary.

xavfernandez · 2017-04-05T17:24:23Z

Strange, it went away but you had the same issue on pypy with the latest patch.
I relaunched this job.

blr246 · 2017-04-05T21:52:07Z

The issue is that the test is timing out. I'm going to split it in line with the functional test modules. That should prevent timeout.

Tests are timing out and also not following requirements/install test factorization. Split to conform to test organization and to prevent test timeout.

xavfernandez · 2017-04-06T08:24:33Z

tests/functional/test_install.py

+    # For each relative path, install as either editable or not using either
+    # URLs with egg links or not.
+    for req_path in (full_rel_path,
+                     'file:' + full_rel_path + '?egg=FSPkg',


The ?egg=... is not used by any code I think.
The expected syntax would be #egg=...

xavfernandez · 2017-04-06T08:24:41Z

tests/functional/test_install_reqs.py

+    # For each relative path, install as either editable or not using either
+    # URLs with egg links or not.
+    for req_path in (full_rel_path,
+                     'file:' + full_rel_path + '?egg=FSPkg',


Also, reduce the number of test cases because these tests take too long. The issue is that the setup and teardown of the test environment takes a long time, and in order to reuse the same test environment, we must perform install/uninstall cycles to test different URL and path formats. A better approach might be to generate `FSPkg` using a function that writes the `setup.py` and `__init__.py` files to a new directory. The test packages would be easy to generate and save the uninstall time. For now, we reduce the number of URL/path formats tested, since the previous test coverage was even more limited.

blr246 · 2017-04-07T00:29:39Z

CI failures are no longer related to this change. The reduction in the number of test cases appears to have worked.

Thanks for the PR reviews and looking forward to version 10!

xavfernandez · 2017-04-07T08:03:09Z

@blr246 Thanks for your work 👍

tony · 2017-08-12T00:13:35Z

Is this in pip 9.0.1?

Relative paths aren't working for me. Am I doing it wrong?

requirements/base.txt:

-e file:./develtech/features/rst/websupport#egg=sphinxcontrib-websupport

pip install -r requirements/base.txt

ls develtech/features/rst/websupport
CHANGES       LICENSE       MANIFEST.in   README.rst    setup.cfg     setup.py      sphinxcontrib test-reqs.txt tests         tox.ini

Obtaining sphinxcontrib-websupport from file:./develtech/features/rst/websupport#egg=sphinxcontrib-websupport (from -r requirements/base.txt (line 43))
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/me/work/develtech/.venv/lib/python3.6/tokenize.py", line 452, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: './develtech/features/rst/websupport/setup.py'

tony · 2017-08-12T00:18:26Z

Looks like there really isn't relative dir support? #3772 (comment)

Not sure, this PR landed after that comment, but it's not working for me. #328

blr246 · 2017-08-12T11:59:28Z

This PR is still pending with release milestone 10.0. @dstufft any chance it can get into a release on a shorter term? To my knowledge, the feature is working.

villasv · 2018-03-28T17:11:24Z

It is possible to use relative links with pip 9.0, just not as reliably as this PR implemented.
You can just put ./youdir in the requirements.txt, (notice the lack of file: descriptor), but the relative paths are resolved in respect to the CWD of the pip install command.

prokher · 2018-03-29T08:49:01Z

@villasv I suppose the fact that relative paths are resolved in respect to the CWD of the pip install is the issue. For example, I need an ability to specify the relative paths which is resolved in respect to the directory where current requirements.txt file resides.

CailanStm · 2018-03-29T18:24:30Z

@prokher @villasv It seems to work if you put -r [relative_path]/requirements.txt in your requirements.txt. This should find the path relative to the requirements.txt file, rather than relative to your CWD

mahmoud · 2018-10-11T21:14:08Z

@CailanStm While you are correct that -r ../../other_project/requirements.txt results in the requirements.txt path being resolved relative to the requirements.txt file that line appears in, any package relative paths within the other project's requirements.txt are still resolved in relation to the current working directory of the original pip command, so the relative path problem still doesn't seem solved to me (on pip version 18.1).

insysion · 2018-12-18T14:57:42Z

@prokher @villasv It seems to work if you put -r [relative_path]/requirements.txt in your requirements.txt. This should find the path relative to the requirements.txt file, rather than relative to your CWD

@CailanStm Can you elaborate? I tried creating a requirements.txt in my working directory that referenced the requirements file in a relative path as you described and I'm getting ... should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

lock · 2019-05-31T04:21:44Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

blr246 changed the title ~~Support relative paths for editable file links.~~ Support relative paths for file links. Dec 28, 2016

blr246 mentioned this pull request Mar 14, 2017

Ambiguity in relative file urls #328

Closed

dstufft requested changes Apr 1, 2017

View reviewed changes

Brandon L. Reiss added 3 commits April 5, 2017 10:00

Merge remote-tracking branch 'upstream/master' into relative-file-paths

ddcf6b0

Add news/4208.bugfix.

465ca09

Add install relative path test.

f4f67ea

dstufft approved these changes Apr 5, 2017

View reviewed changes

xavfernandez requested changes Apr 5, 2017

View reviewed changes

Fix from PR comments.

97667d5

Responding to comments from @xavfernandez.

xavfernandez approved these changes Apr 5, 2017

View reviewed changes

blr246 force-pushed the relative-file-paths branch from 580987d to 537ae44 Compare April 5, 2017 22:35

Split relative paths tests to avoid timeout

c932706

Tests are timing out and also not following requirements/install test factorization. Split to conform to test organization and to prevent test timeout.

blr246 force-pushed the relative-file-paths branch from 537ae44 to c932706 Compare April 5, 2017 22:38

xavfernandez requested changes Apr 6, 2017

View reviewed changes

xavfernandez added this to the 10.0 milestone Apr 6, 2017

xavfernandez approved these changes Apr 6, 2017

View reviewed changes

xavfernandez merged commit 4ee8685 into pypa:master Apr 7, 2017

blr246 deleted the relative-file-paths branch April 7, 2017 12:04

cognifloyd mentioned this pull request Jun 19, 2017

packs - ability to share code between actions/ and sensors/ StackStorm/st2#3490

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label May 31, 2019

lock bot locked as resolved and limited conversation to collaborators May 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support relative paths for file links. #4208

Support relative paths for file links. #4208

blr246 commented Dec 28, 2016 •

edited by dstufft

Loading

dstufft left a comment

blr246 commented Apr 5, 2017

blr246 commented Apr 5, 2017

xavfernandez left a comment

xavfernandez Apr 5, 2017

blr246 Apr 5, 2017

xavfernandez Apr 5, 2017

blr246 Apr 5, 2017

xavfernandez Apr 5, 2017

blr246 Apr 5, 2017

xavfernandez Apr 5, 2017

blr246 Apr 5, 2017

xavfernandez commented Apr 5, 2017

blr246 commented Apr 5, 2017

xavfernandez commented Apr 5, 2017

blr246 commented Apr 5, 2017

xavfernandez Apr 6, 2017

blr246 Apr 6, 2017

xavfernandez Apr 6, 2017

blr246 Apr 6, 2017

blr246 commented Apr 7, 2017

xavfernandez commented Apr 7, 2017

tony commented Aug 12, 2017

tony commented Aug 12, 2017

blr246 commented Aug 12, 2017

villasv commented Mar 28, 2018 •

edited

Loading

prokher commented Mar 29, 2018

CailanStm commented Mar 29, 2018

mahmoud commented Oct 11, 2018

insysion commented Dec 18, 2018

lock bot commented May 31, 2019

Support relative paths for file links. #4208

Support relative paths for file links. #4208

Conversation

blr246 commented Dec 28, 2016 • edited by dstufft Loading

dstufft left a comment

Choose a reason for hiding this comment

blr246 commented Apr 5, 2017

blr246 commented Apr 5, 2017

xavfernandez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xavfernandez commented Apr 5, 2017

blr246 commented Apr 5, 2017

xavfernandez commented Apr 5, 2017

blr246 commented Apr 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blr246 commented Apr 7, 2017

xavfernandez commented Apr 7, 2017

tony commented Aug 12, 2017

tony commented Aug 12, 2017

blr246 commented Aug 12, 2017

villasv commented Mar 28, 2018 • edited Loading

prokher commented Mar 29, 2018

CailanStm commented Mar 29, 2018

mahmoud commented Oct 11, 2018

insysion commented Dec 18, 2018

lock bot commented May 31, 2019

blr246 commented Dec 28, 2016 •

edited by dstufft

Loading

villasv commented Mar 28, 2018 •

edited

Loading