How to correctly run pytest.main() programmatically multiple times?

[numirias]

If I invoke pytest.main(['test_foo.py']) multiple time from the same running script, it will give the same result, even if test_foo.py changes between the runs. (This was also observed in #793.)

But then what is the correct way to rerun pytest programmatically? Could you point me to an example? Starting it in a subprocess feels quite clunky and also makes it more cumbersome to process the results.

(Also, could we possibly have a note about that behavior in the docs? At least to me, it was somewhat unexpected that I can't run pytest.main() in the same script twice.)

[RonnyPfannschmidt]

python modules cache in side the same process and due to this property of python its not advisable to run pytest.main twice from the same process

while there may be exceptions to that rule, in general its more safe to run a new process since python is not safe for code reloading

[nicoddemus]

I agree, we should just add a note to the docs regarding that.

[endolith]

So the recommendation is to completely close python and start a new process every time you want to test your code? I don't remember this being a problem with pytest in the past, has it changed?

[nicoddemus]

@endolith this does not have to with pytest, but how Python itself works: it caches the modules it imports.

For example:

# foo.py
def f(): return 1

>>> import foo
>>> foo.f()
1
>>>

Now if we change foo.py to:

# foo.py
def f(): return 2

and continue in the same session:

>>> import foo
>>> foo.f()
1
>>> foo.f()
1

I don't remember this being a problem with pytest in the past, has it changed?

Sorry, I'm reasonably sure pytest never supported this, because again it doesn't have to do with pytest but how Python works... and I don't think pytest had any special code in place to workaround this.

[nicoddemus]

BTW there's a note at the bottom of the docs for exactly this issue: https://docs.pytest.org/en/latest/usage.html

[endolith]

@nicoddemus Apparently the reason it works for me is a side effect of running inside of Spyder: https://stackoverflow.com/questions/54009371/pytestwarning-module-already-imported-so-cannot-be-rewritten-pytest-remotedata/54011939?noredirect=1#comment94930323_54011939 and the warnings about PytestWarning: Module already imported so cannot are the only thing that's changed?

[nicoddemus]

Oh that explains it then. 👍

[endolith]

So what's the recommended way to do this? pytest.main() prints a bunch of warnings, subprocess.call(['pytest', ... doesn't print the output, etc.

[endolith]

Something like this works, but is kind of ugly:

if __name__ == "__main__":
    from subprocess import Popen, PIPE
    with Popen(['pytest',
                '--tb=short',  # shorter traceback format
                '--hypothesis-show-statistics',
                str(__file__)], stdout=PIPE, bufsize=1,
               universal_newlines=True) as p:
        for line in p.stdout:
            print(line, end='')

[nicoddemus]

Indeed running as a subprocess is the recommended way to do this.

[endolith]

@nicoddemus Maybe a convenience function can be written to do this? Or at least add some hints to the note at the bottom of https://docs.pytest.org/en/latest/usage.html

[nicoddemus]

I'm OK with a note to the docs.

Btw, do you know the pytester fixture? It may be used to invoke python tests in a "block-box" style testing, if that's what you are doing.

[endolith]

@nicoddemus No I don't know about pytester, would that help me? All I'm doing is writing a program and tests simultaneously in the same file, and running the tests each time I change the file to make sure I didn't break anything.

[blueyed]

@endolith
Check out https://github.com/joeyespo/pytest-watch/ then maybe, or just use inotify tools/wrappers (e.g. https://github.com/blueyed/dotfiles/blob/master/usr/bin/inotify-run).
Also check out https://github.com/tarpas/pytest-testmon/ maybe to only re-run changed/affected tests.

[nicoddemus]

For that use case @blueyed's suggestions are on point then. 👍

[endolith]

pytest.main([__file__]) also prints colored text to the IDE, character-by-character, just like running pytest filename on the command line, while the monstrosity I posted above is still buffering on some level, so it only prints in chunks, and only white text. Is there really no way to make pytest.main work as expected, so we can edit the test file, run it, see the results in the IDE, fix mistakes, run it again, etc.?

There's no way to add an option like

if __name__ == '__main__':
    pytest.main([__file__], newprocess=True)

or something?

This is the junk I have at the end of every test_….py file to try to mimic the functionality of pytest.main() while still being compatible with numba and being able to be re-run repeatedly: https://github.com/endolith/elsim/blob/master/tests/test_star.py#L630

It feels wrong, but I don't see what the right way is.