coverage starts too late when a plugin imports the project under test

[jaraco]

Consider the scenario described in python/importlib_resources#304:

• the project under test is importlib_resources.
• a plugin (in this case pytest-enabler) imports the project under test.
• that plugin takes effect before the coverage is started (essential in this case because pytest-enabler must run to determine if/how to enable pytest-cov).
• Every line that was executed at import reports as missed, because it doesn't execute after coverage is started.

This issue is similar to #437, but also different.

This issue also affects coverage of jaraco.context, jaraco.functools and any other project on which pytest-enabler depends.

I wonder if perhaps pytest-cov couldn't somehow avoid scenarios like these, such as by clearing sys.modules or importlib.reloading any modules that are being examined for coverage.

Or perhaps it's not that big of an issue and only really affects running tests on any projects that pytest-enabler imports (because most other plugins run after pytest-cov). So maybe there's something that pytest-enabler should do (such as removing its imports from sys.modules after getting their objects.

[jaraco]

Oh, gosh. And now I notice there is already code in pytest-enabler to handle removing dependencies from the cache. So maybe that's the best approach (just keep that list up-to-date).

[ionelmc]

Kind of chicken and egg problem. Not much I can do to make coverage start early enough without creating more problems (messing with sys.modules is a big no-no, already getting "lets just blame pytest-cov" bug reports like #620 when other libraries or projects are doing it).

Right now you only have 2 options (neither are good if you're really committed to using pytest-enabler to activate pytest-cov):

• use pytest-cov internals to force start it: https://pytest-cov.readthedocs.io/en/latest/plugins.html
• use coverage directly (coverage run -mpytest ...)

[jaraco]

• use pytest-cov internals to force start it: https://pytest-cov.readthedocs.io/en/latest/plugins.html

I've read this page several times and I still don't understand what it's recommending.

The current way of dealing with this problem is using the append feature and manually starting pytest-cov’s engine, eg:

COV_CORE_SOURCE=src COV_CORE_CONFIG=.coveragerc COV_CORE_DATAFILE=.coverage.eager pytest --cov=src --cov-append

Which of these settings affects "using the append feature"? I assume it's the --cov-append option.

Which of these settings affects the "manually starting pytest-cov's engine"? Or is manually starting the engine left as an exercise to the reader?

[ionelmc]

Those env vars will make the subprocess recorder start really early (via the pth file, see https://github.com/pytest-dev/pytest-cov/blob/master/src/pytest-cov.pth and https://github.com/pytest-dev/pytest-cov/blob/master/src/pytest_cov/embed.py). Obviously it's contrived, and you'd only do this if really want to avoid using coverage run -mpytest ....

The append setup is there so cov data can be merged when coverage is restarted by the normal invocation of pytest-cov (when pytest decides it's time for the pytest-cov plugin to run its hooks).

[jaraco]

if really want to avoid using coverage run -mpytest ...

I do really want to avoid having to wrap the pytest invocation. There are some nice advantages to using a plugin like pytest-cov. If I were to wrap the pytest invocation in the tox config with coverage, it would be difficult for a user to disable. As it's currently configured for my projects, one can readily disable coverage with tox -- -p no:cov. I've found that inclusion or exclusion of a given plugin, either based on the presence of that plugin in the dependencies or based on configuration of plugins, is the most flexible and consistent way to make behavior selectively available but on by default across a number of behaviors.

I think I can see now how the pytest-cov plugins guidance works now and may be applicable to my use-case. It's a little annoying that it would essentially need to be applied to every project that uses pytest-enabler (since it would be difficult to anticipate in advance which projects pytest-enabler uses). That's going to be a lot of cruft for the few projects that need it.

messing with sys.modules is a big no-no

I agree - it's hacky and prone to error, but it has the huge advantage of being manageable within a single project (in this case pytest-enabler). Since the approach is working, I'll probably stick with it for now.

---

It sure would be nice, though, if there were some way for pytest to isolate the imports that happen prior to running tests such that in the general case, even for dependencies of pytest and its plugins, the tests themselves actually import the modules as they likely would in the real world. That is, in an ideal world, the pytest setup would be hermetic and wouldn't have any effect on the behavior at test-time.

💡 This makes me wonder if Python itself should offer a contextual cache for imports, such that pytest could "enter" an import context for its setup, then exit it before running tests, effectively isolating the imports that happen during setup and collection from the test run.

As time permits, I may pursue something in that vein. In the meantime, I'm closing this issue as I don't have any recommendations for this project at this time. Thanks for all the support.