Consider switching to separate test executables instead of pytest plugins

[jaraco]

@jaraco would you consider switching from pytest-black to fully blown pre-commit with black, flake8 & co ?

Originally posted by @RonnyPfannschmidt in jaraco/backports.functools_lru_cache#14 (comment)

[jaraco]

I'd surely consider switching, but let me first capture the reasons I enjoy the plugin-based approach:

• Shared runner execution. Only one command to run (pytest) instead of needing to run several.
• Requires no additional virtualenvs, tox envs, etc. while still applying the checks across the supported Python envs.
• No dev-side setup. Check out the code and run the tests with one simple command. There's no prerequisite step that needs to be explained/documented or experience required (beyond tox) for a new user to the project.
• Shared reporting. Failures are reported through pytest (and thus junit, etc) in a unified way alongside unit tests, integration tests, functional tests, etc.
• Shared configuration and control. One enables and disables the tests through the same mechanism (-p no:mypy disables mypy, -p no:black disables black, -k keyword deselects tests not matching keyword, conftest discovery customization is shared). "Fail fast" (-x) applies. Tests are run in parallel if using xdist.
• Easy maintenance. Unlike pre-commit, dependencies don't need to be pinned and each repo doesn't need regular scrubbing to keep up to date.
• Run-by-default configuration avoids the workflow where one develops the solution first, tests are passing, and then an additional set of tests is run, maybe only after pushing upstream, failing other tests (linters, coverage, etc). When pytest passes, the code is golden.
• Only one CI infrastructure is needed (i.e. no need to register/configure/maintain integration with pre-commit.com).

I do recognize there are some disadvantages to the plugin-based approach:

• The clean, explicit invocation behavior is more intuitive for a beginner.
• The non-standard runner means more opportunities for bugs to sneak in, requiring distilling issues as implicating the plugin or not.
• Less separation of concerns (checks are interleaved with other tests).
• For tools like black that can auto-correct the failures, a separate tool invocation is needed to enact the additional behavior.
• Plugins have bugs and a relatively small following means there's only a modest investment in maintaining those plugins.

I derive a lot of value from the plugins. They streamline my workflow and lower the investment I need to make in the project to achieve the same results. I'm really quite happy with them.

Can someone make a compelling case why the plugins should be avoided and propose an approach that minimizes the lost value from the advantages above?

[jaraco]

I'll welcome feedback on this approach, but for now, I see no compelling reason to switch and lots of compelling reasons not to switch.

[simon-liebehenschel]

No dev-side setup. Check out the code and run the tests with one simple command. There's no prerequisite step that needs to be explained/documented or experience required (beyond tox) for a new user to the project.

Makes sense.

Run-by-default configuration avoids the workflow where one develops the solution first, tests are passing, and then an additional set of tests is run, maybe only after pushing upstream, failing other tests (linters, coverage, etc). When pytest passes, the code is golden.

Makes sense too.

If the code is not committed yet, then I have to run pre-commit run --all-files before running tests.

Check out the code and run the tests with one simple command.

Running linters after pulling the remote code is useless because this code is already linted a priori.

Easy maintenance. Unlike pre-commit, dependencies don't need to be pinned and each repo doesn't need regular scrubbing to keep up to date

Pre-commit hooks' versions can be auto-updated at your CI/CD via a bot.

If you are talking that you never and nowhere pin your dependencies, then I have bad news for you: you do not have stable and reproducible builds.

Run-by-default configuration avoids the workflow where one develops the solution first, tests are passing, and then an additional set of tests is run, maybe only after pushing upstream, failing other tests (linters, coverage, etc). When pytest passes, the code is golden.

In my organization every developer must have installed "pre-commit" Git hooks, so pushing a non-linted code to remote is almost impossible.

Plugins have bugs and a relatively small following means there's only a modest investment in maintaining those plugins.

Yes, because they are like a "middle man" between linters and Pytest.

--

PS:
My message is to learn from other people, and to discuss best practices. Not not start a "holy war" when someone blindly says that approach "A" is better than "B".