TypeError: write() argument must be str, not bytes #426
Comments
|
Thanks for the report @fschulze. It crashes in the |
|
@fschulze where you able to collect some more info on this? I think this is a problem in py lib ... |
|
Found this from here: http://stackoverflow.com/questions/13906623/using-pickle-dump-typeerror-must-be-str-not-bytes |
|
Thanks @pjdicke - I think in this case the problem must occur here:
as all arguments are turned into strings, something in verbosty0 must turn the whole thing into bytes again. I did not check myself whats going on there yet, but this is where I would start looking. |
|
I've spent a few hours trying to figure this out, here is where I'm up to: The issue is within the TextIOWrapper ( So, the issue is that something has sent bytes to the stdout stream that shouldn't have. My strong suspicion is that it's related to the Popen/stdout/stderr buffer for the threads that Tox instantiates. Since Tox is designed to call Python 3 and Python 2 equally, it's probably in a subprocess that's the issue, not in the main tox application. There are 2 places that concern me, one is- The second and I think the culprit (because this is almost impossible to properly debug is https://github.com/tox-dev/tox/blob/master/tox/interpreters.py#L74-L77 Here is my tox.ini If I call my process with the environments, py27 it's fine, py36 is fine, py36,p27 is fine. Whenever I include the environment 'lint' it crashes. Yes, I know this tox.ini doesn't make any sense because I haven't defined what the lint environment should run. It calls by default Python 3.6, because that's what my virtualenv is configured with. The error from Tox is on invocation Note, my test suite is supposed to fail, I'm checking that it fails properly. So, in summary if you are getting this issue, try selecting only certain environments then try running your lint/pylint etc. environments under Python 2.7 virtualenv's instead of 3+ |
|
Hi @tonybaloney - thank you for digging into this one. I started a minimal reproducer with your information here: https://github.com/obestwalter/tox-reproducers. At the moment this does not yet trigger the error (see log). Would it be possible for you to add what is needed to reproduce it (ideally in a PR against that repo)? I did not override the test command and use pytest, because I guess these are not specific to reproduce the error. if they are ... well that would be another interesting aspect of that bug. According to your info it should then be possible to run |
|
I'm able to reproduce a crash similar to this with the following combination:
The trace looks as follows: This will affect all platforms. The fix for this one is simple, patch incoming! |
|
Here's a patch, I can make this into a branch later: diff --git a/tox/session.py b/tox/session.py
index 2754787..2c62f13 100644
--- a/tox/session.py
+++ b/tox/session.py
@@ -135,7 +135,7 @@ class Action(object):
fout.write("actionid: %s\nmsg: %s\ncmdargs: %r\n\n" % (self.id, self.msg, args))
fout.flush()
outpath = py.path.local(fout.name)
- fin = outpath.open()
+ fin = outpath.open('rb')
fin.read() # read the header, so it won't be written to stdout
stdout = fout
elif returnout:
@@ -159,6 +159,9 @@ class Action(object):
self.report.logpopen(popen, env=env)
try:
if resultjson and not redirect:
+ # we read binary from the process and must write using a
+ # binary stream
+ buf = getattr(sys.stdout, 'buffer', sys.stdout)
assert popen.stderr is None # prevent deadlock
out = None
last_time = time.time()
@@ -168,12 +171,12 @@ class Action(object):
# might be no output for a long time with slow tests
data = fin.read(1)
if data:
- sys.stdout.write(data)
- if '\n' in data or (time.time() - last_time) > 1:
+ buf.write(data)
+ if b'\n' in data or (time.time() - last_time) > 1:
# we flush on newlines or after 1 second to
# provide quick enough feedback to the user
# when printing a dot per test
- sys.stdout.flush()
+ buf.flush()
last_time = time.time()
elif popen.poll() is not None:
if popen.stdout is not None: |
|
Yeah that was another reason I posted it as a patch because I knew it would conflict :) go for it! |
|
@asottile go ahead with the PR on top of master |
…revent str vs bytes issues.
…revent str vs bytes issues.
|
Just "for the record" or in case somebody else runs into the same isse: I've adjusted a small project to use Python 3 and running
def bar():
import pdb; pdb.set_trace()
return 42
def test_bar():
from foo import bar
assert bar() == 42
from setuptools import setup, find_packages
setup(name='foo',
version='0.1',
packages=find_packages(),
)
[tox]
envlist = py36
[testenv]
usedevelop = true
deps =
pytest
pdbpp
commands =
py.test {posargs}Running Without the Now I wonder if we could or rather should try to detect the presence of |
|
@witsch that's unrelated to this issue (I imagine you'll even be able to reproduce without tox by running I also can't reproduce your findings: |
|
@witsch I suspect the brokenness of your environment stems from whatever |
|
@asottile Thanks for looking into it, and yes, it's probably unrelated to tox, which is why I put the "for the record"... It was pretty much the only hit I could find for this problem and I thought I'd put a note in case somebody runs into the same issue... 🙂 Apart from that, the strange |
|
That "strange" pdb is just the prefix where buildout.python installs, besides that it's the regular pdb from the Python source release. I had the issue Tox on Travis, so in my case it's unrelated to buildout.python. Either it's related to the output capturing of pytest, or pdb++ does something similar and needs to be fixed. |
I get the following traceback when there is an error on Linux and Windows with Python 3.5.
Not sure which tox version is used. I guess the current one, because I use
devpi testand it creates a fresh virtualenv.The only info I have is:
platform win32 -- Python 3.5.2, pytest-3.0.5, py-1.4.31, pluggy-0.4.0
The text was updated successfully, but these errors were encountered: