Python 3.2: GCC warning message leads to UnicodeDecodeError, and Tox stops with InvocationError.

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/10
• Originally reported by: phn
• Originally created at: 2011-08-18T07:30:36.987

Hello,

Tox 1.1 fails with UnicodeDecodeError, when GCC throws a warning message. For example:

"src/tpm/delta_UT.c:15:14: warning: ‘rcsid’ defined but not used".

I can run pip-3.2 (versions 1.0.1 and 1.0.2) outside Tox and the same warning message doesn't cause any trouble.

The same error occurs when using Tox installed on Python 2.6 and on Python 3.2.

I am attaching the last few lines from the Tox log.

Thanks,

Prasanth

Last several lines from tox/log/1.log:

gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/tmp/pip-sienco-build/src/tpm -I/home/phn/bin/python32/include/python3.2m -c src/tpm/delta_UT.c -o build/temp.linux-i686-3.2/src/tpm/delta_UT.o
Exception:
Traceback (most recent call last):
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/basecommand.py", line 126, in main
self.run(options, args)
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/commands/install.py", line 228, in run
requirement_set.install(install_options, global_options)
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/req.py", line 1100, in install
requirement.install(install_options, global_options)
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/req.py", line 572, in install
cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/init.py", line 230, in call_subprocess
line = console_to_str(stdout.readline())
File "/home/phn/tmp/tox/py32/lib/python3.2/site-packages/pip-1.0.2-py3.2.egg/pip/backwardcompat.py", line 60, in console_to_str
return s.decode(console_encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 35: ordinal not in range(128)

[pytoxbot]

Original comment by phn

Hello,

Thanks very much!

Prasanth

[pytoxbot]

Original comment by @hpk42

fix issue 10 (thanks Marc Abramowitz)

→ <>

[pytoxbot]

Original comment by @msabramo

OK, I took a stab at fixing the problem upstream in pip.

The changes in the following tree seem to eliminate this problem:

https://github.com/msabramo/pip/tree/fix-issue-326-UnicodeDecodeError

Pull request: pypa/pip#374

[pytoxbot]

Original comment by @msabramo

The answers to my questions above are:

Nope. pip-1.0.2 is the latest version. I might take a stab at a patch to pip as this might be a common problem in any tools that call pip.

Yes, tox can be modified I think. Here's the pull request: https://bitbucket.org/hpk42/tox/pull-request/5/fix-issue-10-unicodedecodeerror-from-pip

[pytoxbot]

Original comment by @msabramo

On second thought and after looking at http://stackoverflow.com/questions/4545661/unicodedecodeerror-when-redirecting-to-file/4551045#4551045, this might not be a pip problem per se.

The difference might be that pip writing to a terminal works fine but pip writing to a file or pipe does not because Python 2.x can guess the encoding of the terminal, but assumes an ASCII encoding for files and pipes. Tox is calling pip in a subprocess which makes it write to a pipe.

Doing:

{{{
$ tox --recreate -e py31
...
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/backwardcompat.py", line 60, in console_to_str
return s.decode(console_encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 60: ordinal not in range(128)
}}}

But:

{{{
$ PYTHONIOENCODING=utf_8 tox --recreate -e py31
...
[TOX] py31: commands succeeded
[TOX] congratulations :)
}}}

works just fine for me.

Now the questions are:

Does this not happen with a newer version of pip? Perhaps, they added code to explicitly encode their output to UTF-8?

If 1 is not True, then presumably tox could be modified to set PYTHONIOENCODING when it calls pip?

[pytoxbot]

Original comment by @msabramo

See pypa/pip#326

[pytoxbot]

Original comment by @msabramo

I've seen this too, but I think the error is in pip, as evidenced by the stack trace:

{{{
$ tox -e py31
...
Running setup.py install for phpserialize

Running setup.py install for PyYAML
checking if libyaml is compilable
Compiling with an SDK that doesn't seem to exist: /Developer/SDKs/MacOSX10.4u.sdk
Please check your Xcode installation
gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/3.1/include/python3.1 -c build/temp.macosx-10.3-fat-3.1/check_libyaml.c -o build/temp.macosx-10.3-fat-3.1/check_libyaml.o
build/temp.macosx-10.3-fat-3.1/check_libyaml.c:2:18: error: yaml.h: No such file or directory
Exception:
Traceback (most recent call last):
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/basecommand.py", line 126, in main
self.run(options, args)
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/commands/install.py", line 228, in run
requirement_set.install(install_options, global_options)
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 1100, in install
requirement.install(install_options, global_options)
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/req.py", line 572, in install
cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/init.py", line 230, in call_subprocess
line = console_to_str(stdout.readline())
File "/Users/marc/dev/git-repos/anyserializer/.tox/py31/lib/python3.1/site-packages/pip-1.0.2-py3.1.egg/pip/backwardcompat.py", line 60, in console_to_str
return s.decode(console_encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 60: ordinal not in range(128)

Storing complete log in /Users/marc/.pip/pip.log

[TOX] ERROR: could not install deps [pytest, phpserialize, PyYAML]
}}}

Note that the exception occurs in pip and tox handles it by displaying a reasonable error message. I guess the error message could be slightly better if it stated that pip install (and the version of pip) is what failed.

Not sure why this happened for you with tox but not with pip directly -- perhaps tox is using an old version of pip? Duplicate of https://bitbucket.org/hpk42/tox/issue/15/update-to-distribute-0621 perhaps?