Fix pip when using Py2 on Windows with non-ASCII hostname or username. #3970
Conversation
Two related problems are fixed: - Previously non-ASCII characters in hostname blew up `pip install` completely. This is rather severe. - Non-ASCII characters in username crashed printing help text. Not so bad but definitely annoying. Non-ASCII usernames are pretty common in non-English speaking countries. That also makes non-ASCII hostnames pretty common, because Windows creates hostname based on the username by default. The reason for these failures was that `user_cache_dir` returned Unicode on Windows and bytes elsewhere, and rest of the codebase was expecting paths to be bytes. It could be argued that pip should always use Unicode internally, but that would require a lot more changes and fixing also some of the vendored dependencies. We can also argue, like PEP-519 does, that paths should not be considered to be strings at all, and thus the "all Unicode internally" guideline wouldn't apply in this case. Fixes pypa#3463. See that issue for more discussion and details.
|
@pekkaklarck thanks for your patch. |
|
@xavfernandez I don't have a non-ASCII account set up to test this right now, nor do I have a py2.7 install to hand. I'll try to find some time to set up a VM to test it at some point. But I'd prefer it if we could add a test to the test suite to ensure this remains fixed. It could easily enough be done by mocking _get_win_folder. Make sure it's run by the appveyor tests (which only run unit tests not integration tests). |
|
@xavfernandez cool! Already fixed the pep8 issue with docs. Also updated CHANGES.txt but apparently forgot to update my clone first. Will fix the problem ASAP. Do you prefer all changes to be squashes or do you use the newish GitHub functionality to do that when merging? |
|
@pfmoore: If you have a Windows machine available, you only need to temporally set its hostname to something like I agree it would be good to add a unit test. If mocking turns out to be hard, would it be enough to add a test that |
|
Squashed or not, that's fine but @pfmoore is right: an unit test to prevent future regression would definitely close this issue. |
Conflicts: CHANGES.txt
|
Getting too late here and heading to bed. Some questions:
|
|
|
|
Sorry, didn't understand what you mean. Notice that I don't know appveyor at all and don't have much experience from tox either. I would expect tests to work on a fresh clone without extra configuration or possibly configuration to be explicitly documented somewhere. Notice also that so far I've tried running tests on my primary Linux machine. |
Couldn't unfortunately run this locally due to unit test execution failing totally. This test is pretty simple so hopefully it works anyway. Would like to add a separate test to make sure return type is str also on Windows when _get_win_folder returns Unicode. Adding a test with mocking and skipping would be easier if I could run it locally too.
|
@pekkaklarck What @xavfernandez was getting at is that to run the tests on Windows, you need tox installed and then you can do |
|
@pfmoore: I see. I haven't yet even tried running tests on Windows because the whole execution fails on my primary Linux development machine for a pretty uninformative |
|
All the checks are now passing, there's a unit test for the fix and CHANGES.txt has been updated. Is there something else you'd like me to do or is this good to be merged? |
|
|
||
| def test_return_path_as_str(self): | ||
| """Test that path is bytes on Python 2 and Unicode on Python 3.""" | ||
| assert isinstance(user_cache_dir('test'), str) |
There was a problem hiding this comment.
I think I would go with something like:
@pytest.mark.skipif("sys.platform != 'win32'")
@patch('pip.utils.appdirs._get_win_folder', _get_win_folder)
def test_your_test(self, _get_win_folder):
_get_win_folder.return_value = u"läppäri"
assert isinstance(user_cache_dir('test'), str)
# Test against regression #3463
from pip import create_main_parser
create_main_parser().print_help() # This should not crash
There was a problem hiding this comment.
I was thinking something similar but didn't want to add so complex test when I cannot run tests locally.
|
Are all tests run automatically on Windows and can those machines be configured? If yes, changing user and host names to non-ASCII might be the best way to test this. |
|
I'd be a strong -1 on changing machine config in tests (even if it's possible) as that would impact people running tests on their development machine. As I noted, the pip unit tests are run automatically on appveyor. I'd add a unit test like @xavfernandez suggested, and if you can't run it locally, add it to the PR and watch the appveyor run to confirm it worked. (Probably start by adding the test but disabling the fix, to ensure that the test fails as expected on the current code - standard test-driven sort of approach). That should be a perfectly reasonable way for someone without access to Windows to test a Windows-specific change like this. |
|
I tried running unit tests on Windows. I installed Could you @xavfernandez please add the test you proposed yourself? It is a bit complicated and committing it without being able to run tests locally wouldn't be nice. I could just check the CI results like @pfmoore suggested, but that would likely require several iterations and took lot of time waiting for results. Notice that the test I already added verifies the actual fix. It ought to thus be safe to merge this PR also without new tests for |
Conflicts: CHANGES.txt
|
Well, I'm not using windows so I will end up doing what @pfmoore suggested and test it on appveyor. |
|
Alternatively, if I get some time I'll try to look at it. At least I have the option of running the tests locally. I do have very little time right now, though, so no promises, sorry. |
|
It would be awesome if either of you could write the additional test. If it turns out you don't have time before 8.2 release, I can take a look at myself. Without much experience from py.test and Mock, and not being able to run tests locally, it would be complicated but doable anyway. |
|
I've just pushed #4000 which includes a test as suggested by @xavfernandez and the changes from this PR. The test failed on 2.7 when run with master on my PC, and then succeeded when run with this PR. (It succeeded with or without this PR on 3.5). @pekkaklarck could you take a look at my PR and confirm it looks OK to you? I've manually copied your code changes and I'd appreciate it if you could confirm you're happy with it. I don't want to misrepresent your work. |
|
Changes in #4000 look good to me. Only issue I saw was that Thanks @pfmoore for applying the fix and @xavfernandez for help! Already looking forwards for pip 8.2 and being able to tell users encountering this problem that |
Fix pip when using Py2 on Windows with non-ASCII hostname or username (originally #3970 from @pekkaklarck)
|
Closed via #4000 - thanks @pekkaklarck for the fix! |
Two related problems are fixed:
pip installcompletely. This is rather severe.
so bad but definitely annoying.
Non-ASCII usernames are pretty common in non-English speaking
countries. That also makes non-ASCII hostnames pretty common, because
Windows creates hostname based on the username by default.
The reason for these failures was that
user_cache_dirreturnedUnicode on Windows and bytes elsewhere, and rest of the codebase was
expecting paths to be bytes.
It could be argued that pip should always use Unicode internally, but
that would require a lot more changes and fixing also some of the
vendored dependencies. We can also argue, like PEP-519 does, that
paths should not be considered to be strings at all, and thus the "all
Unicode internally" guideline wouldn't apply in this case.
Fixes #3463. See that issue for more discussion and details.