os.system causes TypeError in Python 3

[jondo]

Here's my test case:

from __future__ import print_function
import os

command = "echo x"
response = os.system(command)
print(response)

When I run this in WinPython-64bit-3.4.3.3, Spyder's "Internal console" opens and displays

Traceback (most recent call last):
  File "C:\Program Files\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\baseshell.py", line 285, in write_output
    self.shell.write(self.get_stdout(), flush=True)
  File "C:\Program Files\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 572, in write
    self.flush(error=error, prompt=prompt)
  File "C:\Program Files\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found

Switching back to the IPython console, I see that the response 0 has been printed correctly, though.

In WinPython-64bit-2.7.9.4, the test case runs without error.

[Nodd]

I can't reproduce it with master branch on Linux.

[jondo]

@stonebig - can you reproduce this on Windows?

[stonebig]

Winpython3.4.3.3-64bit/Spyder2.3.4/French windows7-64bit:

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> import os
>>> 
>>> command = "echo x"
>>> response = os.system(command)
x
>>> print(response)
0
>>> 

Shots in the dark:

• an old version of Winpython/Spyder is used,
• a development version of Spyder is used,
• you share the same spyder "work" directory between a Python2 and a Python3,
• there is something special the young coworker did to his pc,
• you are under XP,
• you are not under Windows,
• the disk is full, or the memory is full,
• an exotic package/application/plugin/whatever, like Yhat\rodeo (mpldatacursor ?) has made something strange...

[Nodd]

What if you create a script and run it in the current IPython console ?

[ccordoba12]

@jondo, maybe your coworker is using an old Spyder version. We fixed a lot of unicode bugs for the Python console in 2.3.2.

[jondo]

I have now found out that I can only reliably reproduce this bug when I use

command = "ping -n 1 127.0.0.1"

instead.

I am using German Windows 7,64 bit, and I now did a fresh install of the current WinPython-64bit-3.4.3.3.exe (with Spyder 2.3.4) into a new directory, without any additional packages. Neither disk nor memory are full.
The code is in a script file, which I run in the default IPython console or plain Python console to reproduce this.

When I alternatively paste the code line by line into the Python console after a fresh start of Spyder.exe, I see:

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> import os
>>> command = "ping -n 1 127.0.0.1"
>>> response = os.system(command)

and after entering this, the internal console opens with:

Spyder Internal Console

This console is used to report application
internal errors and to inspect Spyder
internals with the following commands:
  spy.app, spy.window, dir(spy)

Please don't use it to run your code

>>> D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\IPython\nbformat\current.py:19: UserWarning: IPython.nbformat.current is deprecated.

- use IPython.nbformat for read/write/validate public API
- use IPython.nbformat.vX directly to composing notebooks of a particular version

  """)
Traceback (most recent call last):
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\baseshell.py", line 285, in write_output
    self.shell.write(self.get_stdout(), flush=True)
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 572, in write
    self.flush(error=error, prompt=prompt)
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found
Traceback (most recent call last):
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\pythonshell.py", line 563, in write_error
    self.shell.write_error(self.get_stderr())
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 557, in write_error
    self.flush()
  File "D:\RobertPollak\WinPython-64bit-3.4.3.3\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found

(My colleague, who also sees this issue, is on the same OS and has never had any Python version installed on his machine before. He has also got a fresh install of WinPython-64bit-3.4.3.3.exe and definitely no additional Python packages.)

[stonebig]

ok, now I have an error message too:

Console interne de Spyder

Il s'agit d'une console de débogage
utilisée par Spyder pour signaler des erreurs
internes ou pour inspecter les entrailles de
l'application avec les commandes ci-dessous :
  spy.app, spy.window, dir(spy)

Ne l'utilisez pas pour exécuter votre propre code.

>>> D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\IPython\nbformat\current.py:19: UserWarning: IPython.nbformat.current is deprecated.

- use IPython.nbformat for read/write/validate public API
- use IPython.nbformat.vX directly to composing notebooks of a particular version

  """)
Traceback (most recent call last):
Traceback (most recent call last):
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\pythonshell.py", line 563, in write_error
    self.shell.write_error(self.get_stderr())
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 557, in write_error
    self.flush()
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\baseshell.py", line 285, in write_output
    self.shell.write(self.get_stdout(), flush=True)
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 572, in write
    self.flush(error=error, prompt=prompt)
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found
Traceback (most recent call last):
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 823, in paste
    self.execute_lines(lines)
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\externalshell\pythonshell.py", line 70, in execute_lines
    self.flush()
  File "D:\result_tests\winpython-3.4.3.3.amd64\python-3.4.3.amd64\lib\site-packages\spyderlib\widgets\shell.py", line 581, in flush
    text = "".join(self.__buffer)
TypeError: sequence item 0: expected str instance, bytes found

[stonebig]

so, the ping result give some non-ascii character, in French So I assume In German too:

Envoi d'une requête 'Ping'  127.0.0.1 avec 32 octets de données :
Réponse de 127.0.0.1 : octets=32 temps<1ms TTL=128

If I replace your command:

• per "echo ping -n 1 127.0.0.1" I get no error
• per "echo é ping -n 1 127.0.0.1" I get the error

==> I guess the output of the console, not the return message, is given as "latin" or "german" code page, but improperly handled by Spyder console which forgets to convert it to Unicode

nota: it's a bit curious that on the Ipython Qtconsole of spyder:

• !dir works perfectly even if returning characters like Répertoire de D:\result_tests\winpython-3.4.3.3.amd64\settings\.spyder2-py3 ,
• but you're very same commands create the issue also.

[stonebig]

as a bit of fun, this works perfectly on the Ipython Qtconsole of spyder:

!echo é ping -n 1 127.0.0.1

[jondo]

Yes, I can also reproduce this with

os.system("echo é")

And the German ping output has a non-ascii character:

Ping-Statistik für 127.0.0.1:
    Pakete: Gesendet = 1, Empfangen = 1, Verloren = 0

The reason for my originally wrong test case was that the error "sticks" until Spyder is restarted: When the error has appeared once, it also appears on os.system("echo x").

[Nodd]

I guess that the error sticks because the output buffer is not emptied on error and still contains non-ascii chars.