PR: Mark introspection test slow and re-enable it on certain CIs #6138
Conversation
|
Is there a way to disable |
| @@ -58,6 +58,7 @@ def get_spyder_pythonpath(*args): | |||
| editor.introspector.plugin_manager.close() | |||
|
|
|||
|
|
|||
| @pytest.mark.slow | |||
| @pytest.mark.skipif(os.environ.get('CI', None) is not None, | |||
| reason="This test fails too much in the CI :(") | |||
There was a problem hiding this comment.
Please remove this condition to see if it passes in our CIs because else this test wouldn't be run anywhere.
There was a problem hiding this comment.
Ah yes, silly me. I tested it previously with #6044 since it was failing locally too, but after adding huge qtbot.wait()s I got it to pass, though still not perfectly consistently. I figured that had a good chance at fixing it on the CIs, but at the cost of a minimum of 30+ seconds added to every build in the best case; if it failed (which just happened sometimes maybe 1 out of 10 on my machine) and reran 3x, that would multiply to 90, and with 3 flaky runs, (which never seemed to change the outcome and just triple the runtime when it failed, so I dropped them in #6099 ) it would be around 5 extra minutes of build time worst case, before failing anyway. That said, doesn't hurt giving it a shot...
|
Please rebase this PR in top of latest 3.x and skip |
I've raised the treshold of codecov from 0% to 1% before failing. But it is in #6095, which has not been merged yet. Sorry for the inconvenience. |
b471e0e to
1ce2ee6
Compare
Done and done. However, this time the Py3.5/PyQt4 test failed in the same manner; a segfault a few tests after the introspection one (though not on the same test I believe). Maybe some kind of persistent object is sticking around causing trouble? I restarted the previous tests that failed just to see if it was reproducible, and sure enough they failed again. No idea what's really going on, but it seems like it fails on both Py2 and Py3, and PyQt4 and PyQt5, so not sure any build is safe. It seems this one test is just nothing but trouble...perhaps it is almost better to both mark it as slow, and skip it at least on Linux + CI (i.e. Travis), just let AppVeyor run it along with users if they want to (with the right flag to run slow tests, presumably).
No worries. I just didn't want @ccordoba12 to think the actual tests were failing or something. |
Let's try it one more time, but now please skip it in PyQt4. I think Travis should pass this time. |
|
Now things pass in Travis and AppVeyor, and with a nice increase in coverage! Sweet! |
| @pytest.mark.skipif(os.environ.get('CI', None) is not None, | ||
| reason="This test fails too much in the CI :(") | ||
| @pytest.mark.slow | ||
| @pytest.mark.skipif(PY2 or PYQT4, |
There was a problem hiding this comment.
Last time: please remove PY2 from here.
There was a problem hiding this comment.
I think the failures only happen in PyQt4.
There was a problem hiding this comment.
But it segfaulted on Py2 + PyQt5 too... ?
There was a problem hiding this comment.
I can't see it now because I rebased it, but I'm 99% sure both Python tests failed, the QtPy4 one with segfault and the QtPy5 just hung and stopped, and then I restarted the QtPy5 one and it segfaulted too. But if you're sure, I can resend it again...
There was a problem hiding this comment.
Yup, here you go. I cancelled midway but it did indeed segfault.
There was a problem hiding this comment.
Ok, not prob. Thanks for trying 👍
A one line PR: Just mark the
editor_introspectiontest asslow, as due to the timeouts required to make it pass reliably on all systems, it is the slowest in the entire suite, taking over 30s just by itself. Therefore, mark it as such.