WIP: debug pypy3 failures due to "Unroll calls to any #5062 (#5103)"

[blueyed]

This reverts commit 2b9ca34, reversing
changes made to 0a57124.

This is when pypy3 started to fail (https://travis-ci.org/pytest-dev/pytest/builds/537992776).

Ref: #5317

/cc @Tadaboody

[nicoddemus]

How about if instead of reverting this completely, we don't try to unroll calls to any in pypy3 instead? It seems to me this would make it easier to fix the problem than trying to reintroduce the entire feature later.

[blueyed]

How about if instead of reverting this completely

This is just for testing / investigating. I do not want / plan to revert it actually (as-is).

[nicoddemus]

Oh OK!

[blueyed]

The build passed: https://travis-ci.org/pytest-dev/pytest/jobs/538946951
But IIRC it is rather flaky instead, due to something appearing on stderr.

[codecov]

Codecov Report

Merging #5334 into features will decrease coverage by 1.49%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           features    #5334     +/-   ##
===========================================
- Coverage     94.71%   93.22%   -1.5%     
===========================================
  Files           115      115             
  Lines         26333    26249     -84     
  Branches       2601     2591     -10     
===========================================
- Hits          24942    24470    -472     
- Misses         1080     1458    +378     
- Partials        311      321     +10

Impacted Files	Coverage Δ
src/_pytest/assertion/rewrite.py	95.77% <ø> (+0.23%)	⬆️
testing/test_assertrewrite.py	84.06% <ø> (-0.23%)	⬇️
testing/test_reports.py	52.48% <0%> (-47.52%)	⬇️
testing/test_pytester.py	54.04% <0%> (-35.25%)	⬇️
testing/python/collect.py	81.66% <0%> (-17.7%)	⬇️
testing/test_terminal.py	84.6% <0%> (-14.94%)	⬇️
testing/python/integration.py	80.71% <0%> (-10.72%)	⬇️
testing/python/setup_only.py	92.85% <0%> (-7.15%)	⬇️
src/_pytest/pytester.py	86.11% <0%> (-3.72%)	⬇️
src/_pytest/reports.py	92.11% <0%> (-2.96%)	⬇️
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 882f3a4...b474341. Read the comment docs.

[blueyed]

From the warnings summary:

.tox/pypy3-xdist/site-packages/_pytest/fixtures.py:1270
/home/travis/build/pytest-dev/pytest/.tox/pypy3-xdist/site-packages/_pytest/fixtures.py:1270: ResourceWarning: unclosed file <_io.FileIO fd=445 mode='rb' closefd=True>
marker = getfixturemarker(obj)

[nicoddemus]

Thanks for the last commmit.

Argh this makes it harder to debug because I can't get pypy3 to work on Windows. 😞

FTR:

Python 3.6.1 (784b254d6699, Apr 16 2019, 12:10:48)
[PyPy 7.1.1-beta0 with MSC v.1910 32 bit] on win32
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``no random bugs in random, after
all''
>>>> from nt import _getfinalpathname
>>>> _getfinalpathname('c:/foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError
>>>>

(_getfinalpathname is used internally by pathlib)

[blueyed]

Good first shot though I guess.
Might be worth to check if there are warnings also in pypy already, and/or if this is really just maybe flaky due to "output on stderr" (and happened before, but not that much).

[nicoddemus]

Might be worth to check if there are warnings also in pypy already, and/or if this is really just maybe flaky due to "output on stderr" (and happened before, but not that much).

Hmmm interesting, we can try with -s next.

[blueyed]

Hmmm interesting, we can try with -s next.

-s appears to be pretty broken in general already (started debugging / looking into it in #5337 - which is against master)

[nicoddemus]

-s and xdist don't play nice together. We might try pypy3 without xdist and only a single test file, to see if we can reproduce the issue with a smaller number of tests and shorter feedback. 🤔

[blueyed]

I could imagine that rewriting might happen in a place where PyPy does some resource handling itself (https://github.com/pytest-dev/pytest/pull/5103/files#diff-a54d1eaf9d064e9a7d25b1152476bb3dR967).

Bisecting/debugging this quickly is not fun, since there are warnings also on master already.
(Would be easier now with enforced --lsof being in place before already)

[blueyed]

Still appears to be flaky: https://travis-ci.org/pytest-dev/pytest/builds/539061916 ...
(via https://travis-ci.org/pytest-dev/pytest/branches)

[nicoddemus]

sigh now travis won't start

[nicoddemus]

It was because it had a merge conflict then.

[blueyed]

btw, just out of interest: could you use pypy3 in Docker maybe?
Also the nt._getfinalpathname should get reported/fixed probably.

[nicoddemus]

btw, just out of interest: could you use pypy3 in Docker maybe?

I have one or two VMs laying around, I guess there's no other way but try this.

Also the nt._getfinalpathname should get reported/fixed probably.

Definitely, didn't do that yesterday as it was late and I was about to clock out. Will do it later.

[nicoddemus]

Spent the hour or so investigating this. Some observations:

1. I get warnings by just running tox -e pypy3 -- testing/acceptance_test.py.
2. If I add --runpytest=subprocess, I don't see any warnings. 🤔

I will clock out now, might give this another try tomorrow.

But I'm not confident we will find the problem anytime soon. The question is, should this block the 4.6 release? IMHO no; while it is unfortunate that we are getting more of those warnings now, it seems they affect mostly pytest's own test suite, so this is something we can deal later.

[blueyed]

should this block the 4.6 release?

No.
We should either allow failures there, or just remove it temporarily.

[nicoddemus]

I agree with allowing failures. I believe you had a PR with that approach right? Could you resurrect it please? 😁

[blueyed]

Closing for now.