Crash while getting completions of DataFrames on the Editor because of Jedi 0.9

[chrisjcameron]

Description

Crash while editing files

What steps will reproduce the problem?

1. Type in editor, crashes after a few moments.
2. It happened in particular when I am trying to type the groupby() command from pandas on a pandas data frame called p_data.
3. from_grp = p_data.groupb
4. Does not seem to happen when I turn off introspection-related features in the Editor preferences.

What is the expected output? What do you see instead?

Please provide any additional information below
See stack trace at end.

Version and main components

• Spyder Version: 2.3.6
• Python Version: 2.7.10
• Qt Versions: 4.8.6, PyQt4 (API v2) 4.11.3 on Darwin

Optional dependencies

pyflakes >=0.5.0:  0.9.2 (OK)
pep8 >=0.6      :  1.6.2 (OK)
pygments >=1.6  :  2.0.2 (OK)
IPython >=1.0   :  4.0.0 (OK)
zmq >=2.1.11    :  14.7.0 (OK)
qtconsole >=4.0 :  4.0.1 (OK)
pandas >=0.13.1 :  0.16.2 (OK)
sphinx >=0.6.6  :  1.3.1 (OK)
psutil >=0.3    :  2.2.1 (OK)
rope >=0.9.2    :  0.9.4 (OK)
jedi >=0.8.1    :  0.9.0 (OK)
matplotlib >=1.0:  1.4.3 (OK)
sympy >=0.7.3   :  0.7.6 (OK)
pylint >=0.25   :  None (NOK)

[ccordoba12]

Thanks for reporting. Does this error happens with any DataFrame you're working on on the Editor?

[ccordoba12]

Another question: did this happen in our previous version? I mean, with 2.3.5.2?

[chrisjcameron]

I did not notice this issue before and I have done quite a bit of work with pandas data frames before.

[chrisjcameron]

It happens with different data sets as well. Here is an example that uses a data set from the web:

# cd '~/Downloads'
# http://bconnelly.net/assets/data/TradeoffData.csv

import pandas

# Load the data into a DataFrame
data = pandas.read_csv('TradeoffData.csv')

data.head(n=10)

# Turn on code introspection features. 
# Start typing this line. Stop after data.gr.. notice gr is highlighted in yellow. 
#  wait 10 seconds without typing further and Spyder crashes. 
#from_grp = data.groupby("Treatment")

[ccordoba12]

Thanks for the reproducible example. That would make much easier to fix this problem :-)

I think it was caused by me, trying to fix issue #2514. There are other bug reports that seem to suggest the same.

[mcaccia]

I have the same problem. It would be amazing if this could be fixed

[stedeck]

It's the same for me... I've also tested it with older version, but the error still occurs.

[stephenjshaw]

Same for me. One example: it crashes as soon as I type "sum()" to sum up values in a pandas Dataframe column... This is really hampering my coding.

[chrisjcameron]

For people suffering from this bug:
As a temporary workaround, you can disable the introspection features. It is not ideal but it is more workable than crashing constantly.

[ccordoba12]

@blink1073, this seems a problem with Jedi because Rope can't introspect DatafFrames.

Could you take a look at it?

[blink1073]

I can't reproduce the error on master, I tried it with groupby() and sum() and it works fine.

## Version and main components

* Spyder Version: 3.0.0dev 3e4a74d
* Python Version: 2.7.10
* Qt Versions: 4.8.6, PyQt4 (API v2) 4.11.3 on Darwin

## Optional dependencies

pyflakes >=0.5.0:  0.5.0 (OK)
pep8 >=0.6      :  1.6.2 (OK)
pygments >=1.6  :  2.0.2 (OK)
IPython >=3.0   :  3.2.1 (OK)
zmq >=13.0.0    :  14.7.0 (OK)
pandas >=0.13.1 :  0.16.1 (OK)
sphinx >=0.6.6  :  1.3.1 (OK)
psutil >=0.3    :  None (NOK)
rope >=0.9.4    :  0.10.2 (OK)
jedi >=0.8.1    :  0.8.1 (OK)
matplotlib >=1.0:  1.4.3 (OK)
sympy >=0.7.3   :  0.7.6 (OK)
pylint >=0.25   :  None (NOK)

[blink1073]

Also worked with 2.3.5.2.

[blink1073]

Aha, the crash occurs when I update to Jedi 0.9, I'll investigate further.

[blink1073]

This is some strange interaction between the QThread and Jedi, because I printed out the function call and ran it in IPython and it comes out fine:

In [1]: import jedi
In [2]: s = jedi.Script(u"import pandas\ndata = pandas.read_csv('TradeoffData.csv')\nfrom_grp = data.", 3, 16)
In [3]: s.completions()
[<Completion: abs>,
 <Completion: add_prefix>,
 <Completion: add_suffix>,
...

The error does not occur in Python 3. I'm afraid switching to QProcess may be our only way out of this one...

[goanpeca]

One more reason to get to work ;-)

[ccordoba12]

@blink1073, is there something we can about this to avoid the crash in our stable branch?

This is very serious! :-)

[goanpeca]

At least some try except so that spyder does not crash?

[ccordoba12]

Nop, it just occurred to me a simpler solution: forbid Jedi 0.9 as an accepted version, and only use 0.8.1 instead (if present) :-)

[goanpeca]

Or that

[ccordoba12]

Yep, I'll do it for 2.3.7 ;-)

[blink1073]

Yeah, I think that is the only way. There is nothing to "catch" because it is a segfault.

[blink1073]

Once it is a separate process, it should work fine, but if it crashes, we just restart it.

[bilderbuchi]

forbid Jedi 0.9 as an accepted version, and only use 0.8.1 instead (if present) :-)

so, will Jedi >=0.9 be admitted again at a later time? Is there an issue tracking this? could this be related to davidhalter/jedi#520?

[ccordoba12]

@bilderbuchi, yes, it will be admitted again. We think this issue was generated because we are using Qt threads to get completions from both rope and jedi.

For 3.0 we plan to move to use QProcess'es instead.