Provide test runner

[wimleers]

Unit tests are currently written per Python module (since I wrote this module per module, until each module was suitably tested and pretty much bug-free) and don't come with a project-wide test runner yet, that runs all individual unit tests.

This issue is about writing that test runner.

(The above is copied almost verbatim from #81, more specifically from this comment.)

[benoitbryon]

I suggest using nose

[benoitbryon]

To illustrate nose usage, I created a testrunner-with-buildout branch.

It introduces a sample buildout configuration to

• run tests with nose
• analyze code with pylint

Full recipe to configure environment, including getting the source:

git clone https://github.com/benoitbryon/fileconveyor.git -b testrunner-with-buildout
cd fileconveyor
python bootstrap.py -d -c buildout-dev.cfg
bin/buildout -c buildout-dev.cfg

Then run nose and pylint:

bin/nosetests --rednose --with-doctest fileconveyor
bin/pylint --output-format=colorized fileconveyor

Notice that this buildout configuration is not intended to be cross-platform (it includes pyinotify). It may be adapted to fit your needs.

[benoitbryon]

A note about my last comment: my intention is not to force you to use buildout. It is to show that we can use third-party tools to run tests. The focus should be on nose (and maybe pylint).

To include a sample buildout configuration or not, that is a true question. Opinions differ about it. I guess it could be discussed in a separate thread.

[wimleers]

• At c6f5cfd, you accidentally committed two .pyc files — this is extremely minor, obviously, but needs to be fixed before it can be merged.
• I'm not sure I like all this overhead and external projects/dependencies. I was thinking of using a simple Python stdlib-based test runner.
  Is buildout the standard way to do this in Python projects? It seems kind of specific to zope etc and it feels like it's somewhat overengineered. I especially dislike the fact that we need to a copy-pasted file (bootstrap.py) from Zope.
• Plus, it failed:

--( ~ )-- cd Desktop/
--( ~/Desktop )-- git clone https://github.com/benoitbryon/fileconveyor.git -b testrunner-with-buildout
Cloning into fileconveyor...
remote: Counting objects: 2746, done.
remote: Compressing objects: 100% (1459/1459), done.
remote: Total 2746 (delta 1362), reused 2609 (delta 1238)
Receiving objects: 100% (2746/2746), 8.46 MiB | 1.10 MiB/s, done.
Resolving deltas: 100% (1362/1362), done.
--( ~/Desktop )-- cd fileconveyor/
--( ~/Desktop/fileconveyor (testrunner-with-buildout) )-- python bootstrap.py -d -c buildout-dev.cfg 
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.21.tar.gz
Extracting in /var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmpy8s4aR
Now working in /var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmpy8s4aR/distribute-0.6.21
Building a Distribute egg in /var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9
/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9/distribute-0.6.21-py2.6.egg
Creating directory '/Users/wimleers/Desktop/fileconveyor/bin'.
Creating directory '/Users/wimleers/Desktop/fileconveyor/parts'.
Creating directory '/Users/wimleers/Desktop/fileconveyor/eggs'.
Creating directory '/Users/wimleers/Desktop/fileconveyor/develop-eggs'.
Generated script '/Users/wimleers/Desktop/fileconveyor/bin/buildout'.
--( ~/Desktop/fileconveyor (testrunner-with-buildout) )-- 
--( ~/Desktop/fileconveyor (testrunner-with-buildout) )-- e bin/buildout 
--( ~/Desktop/fileconveyor (testrunner-with-buildout) )-- bin/buildout -c buildout-dev.cfg
/Users/wimleers/Desktop/fileconveyor/parts/buildout/site.py:262: UserWarning: Module pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Users/wimleers/Desktop/fileconveyor/eggs/distribute-0.6.21-py2.6.egg is being added to sys.path
  import pkg_resources
/Users/wimleers/Desktop/fileconveyor/parts/buildout/site.py:262: UserWarning: Module site was already imported from /Users/wimleers/Desktop/fileconveyor/parts/buildout/site.py, but /Users/wimleers/Desktop/fileconveyor/eggs/distribute-0.6.21-py2.6.egg is being added to sys.path
  import pkg_resources
Develop: '/Users/wimleers/Desktop/fileconveyor/.'
/Users/wimleers/Desktop/fileconveyor/parts/buildout/site.py:262: UserWarning: Module pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Users/wimleers/Desktop/fileconveyor/eggs/distribute-0.6.21-py2.6.egg is being added to sys.path
  import pkg_resources
/Users/wimleers/Desktop/fileconveyor/parts/buildout/site.py:262: UserWarning: Module site was already imported from /Users/wimleers/Desktop/fileconveyor/parts/buildout/site.pyc, but /Users/wimleers/Desktop/fileconveyor/eggs/distribute-0.6.21-py2.6.egg is being added to sys.path
  import pkg_resources
Traceback (most recent call last):
  File "/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmpIs90Ze", line 11, in <module>
    execfile('/Users/wimleers/Desktop/fileconveyor/./setup.py')
  File "/Users/wimleers/Desktop/fileconveyor/./setup.py", line 36, in <module>
    'cssutils',
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 113, in setup
    _setup_distribution = dist = klass(attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/dist.py", line 223, in __init__
    _Distribution.__init__(self,attrs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 270, in __init__
    self.finalize_options()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/dist.py", line 256, in finalize_options
    ep.load()(self, ep.name, value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.py", line 1912, in load
    raise ImportError("%r has no %r attribute" % (entry,attr))
ImportError: <module 'setuptools.dist' from '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/dist.pyc'> has no 'check_packages' attribute
While:
  Installing.
  Processing develop directory '/Users/wimleers/Desktop/fileconveyor/.'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/private/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 1805, in main
  File "/private/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 446, in install
  File "/private/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 686, in _develop
  File "/private/var/folders/Dz/DzSJmQlWGce1iXGJUQS2J++++TI/-Tmp-/tmp2bl3f9/zc.buildout-1.5.2-py2.6.egg/zc/buildout/easy_install.py", line 1189, in develop
AssertionError
--( ~/Desktop/fileconveyor (testrunner-with-buildout) )-- 

[benoitbryon]

Agreed for the .pyc files.

[benoitbryon]

About nose and dependencies

In the proposal, Nose is not really a dependency: Nose is a tool to discover tests, run them and improve feedback. The tests remain standard! One could run them with another testrunner.

My suggest is:

• every developer has his favorite tools. Since there are several ways to run tests, if you provide a testrunner, some developers may not like it.
• don't include a specific testrunner in the project
• in the docs, show how to use a testrunner, as an example. Nose is easy to use, that's why I proposed it.
• optionally provide a sample "developer configuration", so that developers who don't have a favorite preconfigured development environment can use (or adapt) the sample one and then focus on their contribution. Kind of "developer quickstart".

The other dependencies in the sample (ipdb, ipython, rednose, pylint...) were provided as examples too. The fileconveyor project does not depend on them!

The point was to show how to use some external tools to run tests:

• use external tools to run tests
• don't code a specific testrunner
• optionally add a "Development guidelines" chapter in the documentation, which recommends testing and gives some sample recipes. You will be free to use it or not. You will be happy to find it if you don't know how to start.

[benoitbryon]

Is the buildout error a bug with distribute on OSX? https://bitbucket.org/tarek/distribute/issue/113/distribute-fails-on-fresh-osx-106-system

[wimleers]

I have no idea if it's a bug on OS X, but if it fails, then it's not a good choice IMO. (For the record: yes, I'm using OS X.)

Regarding preferred toolchains and not including a testrunner: I agree it's good to offer the user a choice. But most users don't tend to be Python experts such as yourself. Then it's nice to have a testsuite.py (or something similar) to simply run all tests, without installing all this extra stuff. It would of course be great to document how to use nose/buildout to also run pylint, for example. I'd love to be able to do that! (But apparently, I currently can't, due to a bug in buildout).

So, how about a simple testsuite.py?

[benoitbryon]

Yes, you can write some testsuite.py.

Buildout or not buildout was not the point here. I tried to give you some shortcuts to experiment nose. I'm sorry it fails currently. Moreover, the buildout configuration includes pyinotify, so it is not suitable for OS X (not without some edit). My bad.
If you want to experiment nose, you can also follow the nose documentation, which is quite simple. And here is Pylint documentation too.