Spyder console crashes when trying to type in console after running script

[spyder-bot]

From daan.rab...@gmail.com on 2012-07-01T09:53:07Z

Spyder Version: 2.1.9
Python Version: 2.7.2
Qt Version: 4.7.4, PyQt4 (API v1) 4.8.6 on Windows Vista

What steps will reproduce the problem?

1. create python script with print command as last line
   1. run by pressing F5
   2. click on console and press enter What is the expected output? What do you see instead? Interaction should continue working as desired.

Instead, I see:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\pythonshell.py", line 536, in send_to_process
"toggle_inputhook_flag(True)")
File "C:\Python27\lib\site-packages\spyderlib\utils\bsdsocket.py", line 64, in communicate
write_packet(sock, command)
File "C:\Python27\lib\site-packages\spyderlib\utils\bsdsocket.py", line 24, in write_packet
sock.send(struct.pack("l", len(sent_data)) + sent_data)
socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host

Please provide any additional information below

. - see screenshot

• type of error resembles issue Spyder issue interfacing with external console #945 - tried empty script: same issue
• occurence: very often, but not always
• remark: Spyder installed via pythonxy, 32 bit version installed on 64 bit computer

Attachment: error.png

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1106

[spyder-bot]

From ccordoba12 on 2012-07-03T08:44:40Z

I'm also getting a failure on Linux following your procedure but I got a different traceback:

Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/home/carlos/Projects/spyder/v21/spyderlib/widgets/externalshell/introspection.py", line 65, in run
self.send_socket(shell_id, conn)
File "/home/carlos/Projects/spyder/v21/spyderlib/widgets/externalshell/introspection.py", line 49, in send_socket
shell.set_introspection_socket(sock)
File "/home/carlos/Projects/spyder/v21/spyderlib/widgets/externalshell/pythonshell.py", line 246, in set_introspection_socket
'set_remote_view_settings()', settings=[settings])
File "/home/carlos/Projects/spyder/v21/spyderlib/utils/bsdsocket.py", line 67, in communicate
return read_packet(sock)
File "/home/carlos/Projects/spyder/v21/spyderlib/utils/bsdsocket.py", line 36, in read_packet
dlen, = struct.unpack("l", datalen)
error: unpack requires a string argument of length 8

In my case the problem was triggered if I don't select 'Interact with the Python interpreter after execution'. If I have it checked then I see no problem (which is the quite the opposite of your problem!).

The thing is I really don't know how to debug this.

Summary: Spyder console crashes when trying to type in console after running script
Status: Accepted
Labels: Cat-Console MS-v2.1

[spyder-bot]

From ccordoba12 on 2012-07-03T09:52:04Z

I could see the last failure in Win XP too. However I was not able to reproduce your problem in Win 7 with the latest Python(x,y).

You see the issue only when there is a print line at the end or for any other file you're trying to execute? I say it because this seems to me a problem with your antivirus. What happens if you deactivate it?

[spyder-bot]

From space...@gmail.com on 2012-07-13T11:22:03Z

I am getting the same error in Mac OS Lion. I've just installed Spyder using MacPorts and from the first test I made it gave these errors. The big problem is that sometimes my scripts hang up and I have to restart spyder. It's quite annoying. Any idea on how to solve this?

---

Exception in thread Thread-2:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/spyderlib/widgets/externalshell/introspection.py", line 65, in run
self.send_socket(shell_id, conn)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/spyderlib/widgets/externalshell/introspection.py", line 49, in send_socket
shell.set_introspection_socket(sock)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/spyderlib/widgets/externalshell/pythonshell.py", line 246, in set_introspection_socket
'set_remote_view_settings()', settings=[settings])
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/spyderlib/utils/bsdsocket.py", line 67, in communicate
return read_packet(sock)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/spyderlib/utils/bsdsocket.py", line 36, in read_packet
dlen, = struct.unpack("l", datalen)
error: unpack requires a string argument of length 8

[spyder-bot]

From ebori...@gmail.com on 2012-07-19T12:42:43Z

This should be fixed in MacPorts now. Try upgrading.

Patch: https://trac.macports.org/browser/trunk/dports/python/py-spyder/files/spyderlib_utils_bsdsocket.py.diff?rev=95608

[spyder-bot]

From ebori...@gmail.com on 2012-07-20T10:45:30Z

Updated patch. https://trac.macports.org/browser/trunk/dports/python/py-spyder/files/spyderlib_utils_bsdsocket.py.diff?rev=95738

[spyder-bot]

From ccordoba12 on 2012-07-22T15:31:00Z

Hi eborisch, thanks a lot for the patch. I have three questions for you:

1. Could you describe what it does exactly and how it improves over our current implementation?
2. How can we test it?
3. Do you think the patch will help us to fix this error once and for all in all platforms? (I don't see any special reference to Mac in it).

[spyder-bot]

From ebori...@gmail.com on 2012-07-23T06:48:30Z

The issue I observed was that the first recv would return without error and len(datalen) < SZ. There are many reasons this can happen (I didn't try to suss out which was occurring) on a socket -- indeed by design, without MSG_WAITALL, the size returned may be less than SZ without any error [1]. This would then fail unpacking and raise the error seen in comments 1 and 3 above.

I changed both recv calls in the function to use MSG_WAITALL, and removed the (now unneeded) "while len(data) < dlen:" loop construct. Any shorter receptions should now be due to errors and handled as needed[2].

I am not at all familiar with windows sockets, but this should work on *nix (MacOS included.)

From the recv manpage:
[1] "The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested; this behavior is affected by the socket-level options SO_RCVLOWAT and SO_RCVTIMEO described in getsockopt(2)."
[2] "MSG_WAITALL wait for full request or error"

[spyder-bot]

From pierre.raybaut on 2012-07-28T06:06:34Z

Thanks eborisch for taking the time to fix things and share them afterwards.

However, I think that there's been a misunderstanding.

There are two issues here:

1. The initial submitter (daan.rabijns) was doing the following: creating a script with a print statement, run this script, then click on the console and type Enter. This produces a frightening traceback in the internal console. It should not produce any traceback. Actually, when typing Enter in an already dead console (the script has been executed, Python has terminated, the console is not running anymore), absolutely nothing should happen. This is now fixed with revision 4032473a38f7 .
2. The "error: unpack requires a string argument of length 8" issue which is completely independent. So maybe, a specific thread would have been a good idea.
   Anyway, thanks for the fix, eborisch but this is not cross-platform. The MSG_WAITALL flag has no equivalent on Windows platforms, hence the current implementation. Has this bug been reproduced on Linux, or only on MacOSX? If it's affecting both Linux and MacOSX, I suggest a switch like this:

def read_packet(sock, timeout=None):
"""
Read data from socket sock
Returns None if something went wrong
"""
sock.settimeout(timeout)
dlen, data = None, None
try:
if os.name == 'nt':
# Windows implementation
datalen = sock.recv(SZ)
dlen, = struct.unpack("l", datalen)
data = ''
while len(data) < dlen:
data += sock.recv(dlen)
else:
# Linux/MacOSX implementation
datalen = sock.recv(SZ, socket.MSG_WAITALL)
if len(datalen) == SZ:
dlen, = struct.unpack("l", datalen)
data = sock.recv(dlen, socket.MSG_WAITALL)
except socket.timeout:
raise
except socket.error:
data = None
finally:
sock.settimeout(None)
if data is not None:
try:
return pickle.loads(data)
except (EOFError, pickle.UnpicklingError):
return

[spyder-bot]

From pierre.raybaut on 2012-07-28T06:11:57Z

This issue was updated by revision 4032473a38f7 .

Fixes original issue as described by daan.rabijns. In other words, a dead
console should not offer any interaction at all to the user.
@-daan.rabijns: if you want to keep interacting with the console, you'll have to
check the "Interact with the Python interpreter after execution" option in the
"Run configuration" dialog box (F6).

[spyder-bot]

From pierre.raybaut on 2012-07-28T06:12:41Z

Here is the patch associated to my suggestion at the end of comment #8.

Status: Started

Attachment: patch_for_bsdsocket.diff

[spyder-bot]

From mvi...@gmail.com on 2012-09-10T00:47:11Z

I suggest to keep the Windows version of the patch for all platforms. MSG_WAITALL is not quite portable, but calling send() and recv() in a loop always works no matter what (it's the usual way of programming with sockets, recommended by pretty much every book out there).

In Python you can avoid the loop for send() by calling sendall() instead, but sadly there's no recvall() equivalent. :(

[spyder-bot]

From ebori...@gmail.com on 2012-09-10T06:53:38Z

The windows version of the patch doesn't handle the case when 'datalen = sock.recv(SZ)' is not fully sized (len(datalen) < SZ). This is the error that was occurring in comments 1 and 3, above. Please fix that before removing the !windows flavor.

[spyder-bot]

From mvi...@gmail.com on 2012-09-10T12:35:20Z

Also if the socket connection is cut from the other side the receiving end would enter an infinite loop. And it's also possible to read more data than wanted in the last recv() call.

A possible fix:

datalen = recvall(sock, SZ)
dlen, = struct.unpack("l", datalen)
data = recvall(sock, dlen)

And add this auxiliary function:

def recvall(sock, length):
data = ''
while len(data) < length:
new_data = sock.recv(length - len(data))
if not new_data:
raise socket.error()
data += new_data
return data

In recvall() the length of the data already read is substracted to the total length to prevent reading more bytes than wanted, and the check on new_data prevents the infinite loop if the connection is closed from the other end.

There's also the problem with the struct format string but that belongs to issue #1016 so I didn't change it, but it should probably be "<L" or "!L" or something like that.

One more thing, it wouldn't hurt to have a maximum size limit, so in case of an error we don't end up trying to read 4Gb of data from the socket.

Yeah, socket programming sucks ;)

[spyder-bot]

From mvi...@gmail.com on 2012-09-10T13:17:10Z

Sorry, that was issue #1146 I meant back there.

[spyder-bot]

From mvi...@gmail.com on 2012-09-10T13:21:28Z

As clarified in issue #1146 the struct pack format string was not the issue here. It may be worth considering using "<L" instead just to simplify the code when checking for a maximum size limit (otherwise there'd have to be a minimum size limit too, to prevent negative numbers).

[spyder-bot]

From ebori...@gmail.com on 2012-09-10T21:04:47Z

And this is where MSG_WAITALL works so nicely... (on systems where it is supported.) It basically does what you are doing with your helper function, but without needing the helper function.

Is there some portability concern beyond Windows for the MSG_WAITALL version? It's just cleaner if it is available, IMHO. It's likely faster, too.

As we're communicating all on the same machine here, the "l" just needs to match what is used on the sending side in spyder (which I'm guessing is the write_packet() def directly above read_packet()) ... it's using struct.pack("l"), too.

[spyder-bot]

From ebori...@gmail.com on 2012-09-10T21:05:55Z

"As we're communicating all on the same machine here" << Someone more involved with the project please speak up if I am incorrect here!!

[spyder-bot]

From pierre.raybaut on 2012-09-13T11:42:55Z

"Is there some portability concern beyond Windows for the MSG_WAITALL version?" >> Yes, unfortunately, as MSG_WAITALL simply does not exist on Windows platforms.

"As we're communicating all on the same machine here" >> that's correct!

[spyder-bot]

From ebori...@gmail.com on 2012-09-13T17:24:10Z

My question was if for modern UNIX systems there was any portability concern.

[spyder-bot]

From ccordoba12 on 2012-09-19T07:37:51Z

issue #1149 has been merged into this issue.

[spyder-bot]

From pierre.raybaut on 2012-09-23T05:14:53Z

This issue was closed by revision ad0b96cff9a3 .

Status: Fixed

[spyder-bot]

From ccordoba12 on 2013-02-04T14:40:34Z

issue #1162 has been merged into this issue.

[spyder-bot]

From ebori...@gmail.com on 2013-02-06T07:35:46Z

I'm not sure this is fixed for windows: (but perhaps it is working for most people anyway?)

Windows implementation

datalen = sock.recv(SZ) # len(datalen) >= 0 w/o raising exceptions; not guaranteed to be == SZ
dlen, = struct.unpack("l", datalen) # So this might fail.

I think perhaps the best thing is to use a recvall function like suggested in comment #13, which could be defined differently for windows and Unix.

if os.name == "nt":
def recvall(sock, length):
data = ''
while len(data) < length:
new_data = sock.recv(length - len(data))
if not new_data:
raise socket.error()
data += new_data
return data
else:
recvall = lambda sock,length: sock.recv(length, socket.MSG_WAITALL)

def read_packet(sock, timeout=None):
{ ... }
try:
datalen = recvall(sock, SZ)
dlen, = struct.unpack("l", datalen)
data = recvall(sock, dlen)
except { ... }

[spyder-bot]

From pierre.raybaut on 2013-02-10T05:47:46Z

Why not?

Have you tested this on both Windows and Linux?

We could add this change in v2.2 but if we do, it has to be now, for beta3 for example.

[spyder-bot]

From jtaylor....@googlemail.com on 2013-03-15T15:19:46Z

the sending in bsdsocket.py in this does not handle partial sends, this should be fixed too.
I don't know if there is a flag equivalent to MSG_WAITALL
also EINTR loops are missing, for the latter I recommend adding a function like this (similar to gnulibc TEMP_FAILURE_RETRY macro):

def temp_fail_retry(error, fun, _args):
while True:
try:
ret = fun(_args)
return ret
except error as e:
if e.args[0] == errno.EINTR:
continue
raise

[spyder-bot]

From jtaylor....@googlemail.com on 2013-03-15T15:31:34Z

heres a patch:
--- a/spyderlib/utils/bsdsocket.py Thu Mar 14 16:40:42 2013 +0100
+++ b/spyderlib/utils/bsdsocket.py Fri Mar 15 23:31:05 2013 +0100
@@ -15,7 +15,17 @@
import struct
import cPickle as pickle
import threading
+import errno

+def temp_fail_retry(error, fun, *args):

• while 1:

•    try:
  

•        ret = fun(*args)
  

•        return ret 
  

•    except error as e:
  

•        if e.args[0] == errno.EINTR:
  

•            continue
  

•        raise
  

  SZ = struct.calcsize("l")

@@ -26,7 +36,10 @@
sent_data = data
else:
sent_data = pickle.dumps(data, pickle.HIGHEST_PROTOCOL)

• sock.send(struct.pack("l", len(sent_data)) + sent_data)

• sent_data = struct.pack("l", len(sent_data)) + sent_data

• nsend = len(sent_data)

• while nsend > 0:

•    nsend -= temp_fail_retry(socket.error, sock.send, sent_data)
  

  def read_packet(sock, timeout=None):
  @@ -48,10 +61,12 @@
  # Linux/MacOSX implementation
  # Thanks to eborisch:
  # https://code.google.com/p/spyderlib/issues/detail?id=1106 - datalen = sock.recv(SZ, socket.MSG_WAITALL)

•        datalen = temp_fail_retry(socket.error, sock.recv,
  

•                                  SZ, socket.MSG_WAITALL)
       if len(datalen) == SZ:
           dlen, = struct.unpack("l", datalen)
  

•            data = sock.recv(dlen, socket.MSG_WAITALL)
  

•            data = temp_fail_retry(socket.error, sock.recv,
  

•                                   dlen, socket.MSG_WAITALL)
  

  except socket.timeout:
  raise
  except socket.error:

[spyder-bot]

From pierre.raybaut on 2013-03-17T07:27:30Z

This issue was updated by revision 6c978bee1d85 .

Please test this change and mark this issue as Verified once it's done.
I've just tested it successfully on Windows 8.

[spyder-bot]

From ccordoba12 on 2013-03-17T07:37:28Z

Labels: -MS-v2.1 MS-v2.2

[spyder-bot]

From jtaylor....@googlemail.com on 2013-04-08T14:13:50Z

works fine on ubuntu 13.04

[spyder-bot]

From ccordoba12 on 2013-04-08T14:22:53Z

Thanks Julian for checking it.

Status: Verified

[spyder-bot]

From ccordoba12 on 2013-07-04T10:32:50Z

issue #1474 has been merged into this issue.

[spyder-bot]

From ccordoba12 on 2014-08-23T12:55:21Z

issue #113 has been merged into this issue.

[spyder-bot]

From ccordoba12 on 2015-01-02T18:26:25Z

issue #1146 has been merged into this issue.

[spyder-bot]

From ccordoba12 on 2015-01-02T18:33:19Z

issue #1173 has been merged into this issue.