Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

don't install empty dirs during wheel installs #1743

Merged
merged 6 commits into from
Apr 25, 2014
Merged

don't install empty dirs during wheel installs #1743

merged 6 commits into from
Apr 25, 2014

Conversation

qwcode
Copy link
Contributor

@qwcode qwcode commented Apr 22, 2014

a fix for #1632, that solves the problem by not installing empty directories (including the empty __pycache__ dirs in pypy under dist-info). this should really only be an issue an odd cases, as packaging doesn't allow/create empty dirs afaik

note that I'm wanting to start a trend here of commiting the src for test packages (when it's something we've constructed) to tests/data/src, and in general use src for project src, and leave packages just for packages.

@Ivoz
Copy link
Contributor

Ivoz commented Apr 23, 2014

So you'd like this for a 1.5.5?

@qwcode
Copy link
Contributor Author

qwcode commented Apr 23, 2014

@dstufft @Ivoz @pfmoore looking for a +1 from someone. the main change is a small, subtle change to move_wheel_files to prevent the pypy __pycache__ problem.

cc @alex

@dstufft
Copy link
Member

dstufft commented Apr 23, 2014

I'm not sure I understand the changes or how they fix things, but pip.wheel is really hard to follow.

@qwcode
Copy link
Contributor Author

qwcode commented Apr 23, 2014

@dstufft in #1632, you say: "So I would say ideally we should only install files (creating any directories we need to)". this change does that, whereas before, we created directories regardless of whether we knew a directory had files.

So you'd like this for a 1.5.5?

#1632 was in the 1.5.5 milestone, so I developed against 1.5.X

@alex
Copy link
Member

alex commented Apr 23, 2014

I have no idea what's going on. But thanks so much for working on this <3

# directory creation is lazy and after the file filtering above
# to ensure we don't install empty dirs
if not os.path.exists(destdir):
os.makedirs(destdir)
shutil.move(srcfile, destfile)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming files is a list of all of the files in the Wheel without any empty directories, then we should probably switch this shutil.move to shutil.copy2(), then it will fail if we tell it to copy a directory.

@pfmoore
Copy link
Member

pfmoore commented Apr 23, 2014

The core change looks OK to me. Why did you add a scheme argument? Was that for the tests? Sorry I don't have time to do a full review at the moment.

@qwcode
Copy link
Contributor Author

qwcode commented Apr 23, 2014

Why did you add a scheme argument? Was that for the tests?

yes, for tests, instead of patching. not 100% sure on that, but I try to limit patching when possible.

@Ivoz
Copy link
Contributor

Ivoz commented Apr 24, 2014

I'd also like to write the dist-info stuff out of this function on develop as well so I can refactor it somewhat to add INSTALLER / REQUESTED files. Atm it does more than clobber :/

@qwcode
Copy link
Contributor Author

qwcode commented Apr 24, 2014

one thing to note is that if someone (on pypy) happens to manually use compileall to compile the installed files (after pip has installed them), then they'd still have a problem, since this fix works on the install side, not the uninstall side. I'm inclined to let that specific manual case still remain a problem, and hope that pypy fixes the problem on their end eventually.

If that concerns anyone, then maybe a special case in the uninstall logic that looks specifically for empty __pycache__ subdirs only? because I hesitate to do anything more general than that and feel confident about it.

thoughts?

@alex
Copy link
Member

alex commented Apr 24, 2014

People manually calling compileall doesn't seem like a huge deal to me.
That said, I want to point out that this isn't a PyPy issue, per-se, it's a
Debian's patched PyPy issue.

Alex

On Thu, Apr 24, 2014 at 12:15 PM, Marcus Smith notifications@github.comwrote:

one thing to note is that if someone (on pypy) happens to manually use
compileall to compile the installed files (after pip has installed them),
then they'd still have a problem, since this fix works on the install side,
not the uninstall side. I'm inclined to let that specific manual case still
remain a problem, and hope that pypy fixes the problem on their end
eventually.

If that concerns anyone, then maybe a special case in the uninstall logic
that looks specifically for empty pycache subdirs only? because I
hesitate to do anything more general than that and feel confident about it.

thoughts?


Reply to this email directly or view it on GitHubhttps://github.com//pull/1743#issuecomment-41320294
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

@pfmoore
Copy link
Member

pfmoore commented Apr 24, 2014

this isn't a PyPy issue, per-se, it's a Debian's patched PyPy issue.

In which case let's leave it to Debian to patch their pip if they want the compileall bit fixed. I think it's enough of a corner case that I'm not convinced we need to add a special case anyway and this pushes me over to "let's not".

@dstufft
Copy link
Member

dstufft commented Apr 24, 2014

Eh, it's a general issue with our install code, there's no reason for us to be installing random empty directories, we shouldn't be installing directories at all, we should be installing files. It just happens that the issue that exposed this is a corner case.

@pfmoore
Copy link
Member

pfmoore commented Apr 24, 2014

I'm +1 with the patch as a whole. My comment was about the question @qwcode asked about special casing __pycache__ subdirs in uninstall for people who manually run compileall. I'm going on @alex's comment that it's specific to Debian's patched PyPy - if so, then I'm -1 on special casing, if not I'm neutral.

@dstufft
Copy link
Member

dstufft commented Apr 24, 2014

Oh, yea I don't see any reason to special case __pycache__ here.

qwcode added a commit that referenced this pull request Apr 25, 2014
don't install empty dirs during wheel installs
@qwcode qwcode merged commit 7f355a6 into pypa:1.5.X Apr 25, 2014
@qwcode
Copy link
Contributor Author

qwcode commented Apr 25, 2014

I'll get this over to develop as well, in a bit.

@qwcode qwcode mentioned this pull request Apr 25, 2014
qwcode added a commit that referenced this pull request Apr 25, 2014
@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 5, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 5, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants