Tox Integration with “setup.py test” command

[ronindesign]

From http://tox.readthedocs.io/en/latest/example/basic.html#integration-with-setup-py-test-command

Integrating tox with setup.py test is as of October 2016 discouraged as it breaks packaging/testing approaches as used by downstream distributions which expect setup.py test to run tests with the invocation interpreter rather than setting up many virtualenvs and installing packages. If you need to define setup.py test you can better see about integrating your eventual test runner with it, here is an example of setup.py test integration with pytest. As the python eco-system rather moves away from using setup.py as a tool entry point it’s maybe best to not go for any setup.py test integration.

Not sure if this is relevant?

[yannmh]

Hey @ronindesign,

It seems to be that it does not apply to the datadogpy project.

We use Tox for TravisCI only, to facilitate testing on multiple versions of Python. I believe this is not even needed anymore, and I crafted a #127 to get rid of it.

On the other hand, setup.py uses the nose test runner. Running python setup.py is equivalent to

nosetests ./tests

[ronindesign]

Ok, just seems like it's an issue of documentation, as CONTRIBUTING.md
mentions running tests via python setup.py test, so I wanted to make sure running nose tests directly didn't miss anything.

Thanks for the follow up on this!

[bittner]

The Tox documentation change quoted by @ronindesign was caused by tox-dev/tox#330 (comment). It's not really a technical flaw or implementation risk. It's rather a manifestation of the overruling powers of a package maintainer over the opinions of other interested users and collaborators.

The preferable way to run tests should be via tox, even at the cost of some overhead, because this makes the CI server execute the test suite the way you do it as a developer on your local developer machine. You can achieve this in the Travis configuration via the TOXENV environment variable. See Organice/django-organice@1031e8e for a related solution.

[ronindesign]

@bittner That makes a lot of sense. Maybe it would be helpful to add examples or comments on this to the contrib docs.