Having a lof of data in the Variable explorer slows down Spyder considerably

[Diti24]

Description

The variable explorer slows down the spyder IDE, when big data is present in the workspace.
Activating the "Truncate values" option and disabling the min/max option does not improve the situation (as expected from documentation). Is there a way out? Closing the variable explorer/ or switching on another tab removes the problem.

• What steps will reproduce the problem?

1. Create a big big data array
2. Look in to the variable explorer
3. Work with the IDE (should become quite slow)
4. Maybe removing/hiding the Value colum could help, often just the size and type are interesting and one could look at the values just on demand... just a hing

• What is the expected output? What do you see instead?
• Please provide any additional information below

Version and main components

• Spyder Version: 2.3.4
• Python Version: 3.4.3
• Qt Version : 4.8.6, PyQt4 (API v2) 4.11.3 on Linux

Optional dependencies

pyflakes >=0.6.0: 0.8.1 (OK)
pep8 >=0.6 : 1.6.2 (OK)
IPython >=1.0 : 3.1.0 (OK)
zmq >=2.1.11 : 14.6.0 (OK)
pygments >=1.6 : 2.0.2 (OK)
pandas >=0.13.1 : 0.15.2 (OK)
sphinx >=0.6.6 : 1.2.3 (OK)
psutil >=0.3 : 2.2.0 (OK)
rope >=0.9.2 : 0.9.4-1 (OK)
jedi >=0.8.1 : 0.8.1 (OK)
matplotlib >=1.0: 1.4.2 (OK)
sympy >=0.7.3 : 0.7.6 (OK)
pylint >=0.25 : 1.4.3 (OK)

[ccordoba12]

Please post how you're creating your big big data array.

[Diti24]

Hi,
I extracted part of the code in order to be able to reproduce it.

import numpy as np
from pandas import DataFrame

aa = dict()
for i in range(0,222):
    key = "%04i" %(i)

    bufData = np.concatenate((np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1),
                              np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1), np.random.rand(10000, 1)), 1)
    aa[key] = DataFrame.from_records(bufData, columns=['globCnt', 'adcHigh', 'adcLow', 'distance', 'posErr', 'preshForce', 'propForce', 'derivForce', 'integForce', 'totalForce'])

if you have the variable explorer open it becomes difficult to work with the IDE. If you close or switch to another tab, the IDE is responsive again.

[ccordoba12]

Thanks for the example. I'll look into it for 3.0

[ccordoba12]

This problem was partially addressed on issue #2560, but I have another idea to improve the situation.

[tianfudhe]

I'm a fan of Spyder, and suffers from this problem too.
loading data can really slow down the ipython console, as others mentioned above.
Actually itergrated-ipython-console slows down because of the interactive response time lag, i.e. delay the same time what ever cell you input.

Anyway it's amazing Python IDE, like matlab-for-python :)

[seanbraxton]

I've run in to a similar issue using Spyder 3.3.2 with both Python 2.7 32-Bit and Python 3.7 64-Bit. In particular I was trying to determine which part of my code is taking the most time. I implemented timeit.timeit() to see how long it was taking to csv.read() of a large dataset ~3Million Rows, Single Column. Timeit was telling me the average time was microseconds; however, running the code within the Spyder IDE takes ~ 4seconds. I am not sure if timeit() doesn't actually load variables into memory or if it's executing the same routine. It's hard to know without having some output.

[ccordoba12]

@seanbraxton, please open a new issue about your problem along with some simple code to recreate it.