PROPOSITION: aiohttp-devtools

[samuelcolvin]

This is really an extension of @ohsabry's comment: #778 (comment).

I very much approve of aiohttp being a microframework without the "batteries included" (aka "unwanted confusion included") of django, however sometime too many choices can be baffling.

Including tools useful for getting started, development and testing will both cause bloat in aiohttp and cause unnecessary dependencies in deployment environments (no one want the likes of watchdog installed on heroku or swelling a container image, but having it for auto-reload during testing is a great help).

Then again, lots of dev tools is a mess too, as @ohsabry put it: "it would be a huge turn off for me if all of the debug and development tools are scattered around a bunch of small external repositories that may never get maintained."

Therefore I propose one package containing (or maybe requiring) all the tools required when starting, developing and testing aiohttp applications. It would include:

• a runserver command, roughly equivalent to aiohttp_runserver
• debug toolbar, roughly aiohttp_debugtoolbar
• testing harness, eg. pytest-aiohttp
• a cookie cutter command to create a new bare bones aiohttp app similar to django's startproject. This could be extensible with different subcommands eg. +pg +redis +jinja to control what gets configured.
• an easy way to call management processes like creating or resetting a db. Currently this is a pain as you can't execute a module which is using relative imports, so you have to create a stub cli module outside you app.

Doing this would:

• massively smooth the learning curve of getting started with aiohttp
• encourage elegant design principals by giving a sane and simple starting point - eg. attaching reused objects like connection pools to app rather than keeping them global.
• encourage TDD by creating a tests.py stub like django does.
• prevent aiohttp becoming a behemoth by providing a sensible place for "extra" stuff.

I'm happy to do a fair bit of development towards this but there's no point in starting without agreement from those in the driving seat like @asvetlov, this would need to be included in the aio-libs organisation and be tested before releases of aiohttp to be worthwhile.

What do you think?

[asvetlov]

I love the idea of splitting aiohttp dev tools into separate package.
Cookie cutter template is very long awaited feature also.

My random thoughts:

1. aiohttp_devtools should be very tightly coupled with aiohttp itself.
2. aiohttp documentation should mention development tools in usage sections like http://aiohttp.readthedocs.io/en/stable/web.html but reference detail links should go to aiohttp_devtools.
3. We need a steady maintainer for the library. @samuelcolvin maybe you are?
   De-facto I'm the only maintainer for aiohttp.
   Other guys help me a lot but they are appear and disappear depending on their free time and interests.
   But the library maintenance assumes responsibility on answering questions in issue tracker, working on PRs, making new releases in sync with aiohttp etc.
   I will help to maintain the project but sorry. I cannot and don't want to be the main supporter. My time is limited by 24 hours per day. Now I have no time for significant aiohttp improvements because I should fight with reported bugs. Adding a new library support burden is too high weight for me.
4. Dev tools should be in sink with aiohttp master and follow aiohttp deprecation period (1.5 years) if possible.
5. All current dev tools already published in aiohttp should be maintained until deprecation period will pass out. I'm fine with moving the code into aiohttp_devtools but aiohttp should import these tools for a while, e.g.:

aiohttp.web.py:
```
from aiohttp_devtools import run_server
```

Tests should be moved into dev tools but aiohttp itself must keep basic tests for making sure that the moved API is still accessible.
6. Code quality.
I believe the bullet is obvious but I want to stress: the code for new library should be written well.
100% test covered etc. (I know aiohttp itself is 98% covered only but I'm working hard on the coverage improvement).
7. Moreover, it must be in sync of my gut feels. I would be Benevolent dictator for life for the project.
I'm de-facto BDFL for aiohttp. I believe it brings no hurt :)
But all significant commits for new lib should be done as PRs and merged after careful review only.
Personally I don't always follow the rule on aiohttp development just because reviewing team is not very active.
But for aiohttp_devtools I would volunteer for reviewing every PR until we'll consume a steady reviewers community.
8. Documentation for the library must be perfect. At least as good as aiohttp doc.

Looks like I've finished.

Conclusion: we are need a responsible maintainer for the project.
@samuelcolvin are you want to take the honor and burden?

[samuelcolvin]

I can't predict the future, so I can't say where I'll be in two years, but for the foreseeable future I'm prepared to be the maintainer.

100% test coverage I agree with, the last production grade app I built arq demonstrates that, it also demonstrates that I take documentation seriously too (there are some typo as arq is still a work in progress). It helps for docs that I'm a native english speaker.

I think the best way forward is thus:

I start building aiohttp-devtools and get it to the point of working but not polished. At that point you review, perhaps there are things you disagree with which I fix, I polish it and finish docs.

At that point if we're both happy I transfer the project to aio-libs and we release to pypi, I continue as maintainer with your over-site. If at this point, we absolutely can't agree about how things should be done we stop the project before wasting each other's time.

Make sense?

[argaen]

The idea sounds really nice and +1 for it. Currently I'm working in different personal projects but I'm using aiohttp at work daily so I can always find time to write some stuff, improve docs, review, etc. From my point of view, the things I would include first in this project:

• cookiecutter that generates a basic but complete project that teaches best practices
• test utils

For me those are the most important things a developer needs to start a project.

@asvetlov I quite don't understand (or agree with) the "Tests should be moved into dev tools but aiohttp itself must keep basic tests for making sure that the moved API is still accessible.". For me tests should be kept inside the aiohttp itself (the ones related to the code ofc). If I didn't understand wrong, aiohttp-devtools would be a package that provides a set of tools to ease development, not a must have package to be able to develop for aiohttp. If a maintainer wants to use aiohttp-devtools for easing it's debugging for it's aiohttp release or because it provides autoreload cool, but I think tests should be kept inside the package.

[samuelcolvin]

As per discussion on #1154 test tools have to be a separate package for slightly annoying circular import reasons.

I've changed my mind and I think pytest-aiohttp shouldn't be built into aiohttp-devtools because you don't want to have to install all of aiohttp-devtools on travis etc. to run tests.

We could make pytest-aiohttp a dependency of devtools to make sure people have it installed and make sure stub tests in a cookie cutter project work out-of-the-box.

[asvetlov]

@samuelcolvin sounds like as great plan.
I suggest to register aiohttp_devtools name on PyPI early -- somebody may squat the beautiful name first.
Feel free to ask me any question before you'll be ready for official pronouncement for job finishing.
Or even better -- after creation of aiohttp_devtools under your account I'll subscribe on your changes.

[asvetlov]

@argaen sorry for bad wording.
I've meant tests for aiohttp.web.run_app may be moved into aiohttp_devtools, not the whole aiohttp test suite.
Regarding to pytest support etc. -- let's make the final decision later.
We need to keep backward compatibility for long time anyway.

[ohsabry]

keeping aiohttp lean and not bloated like django/rails/etc, but having a single, well-maintained framework with all of the dev tools seems, like the best of both worlds for me as an end user.

@samuelcolvin 👍 👍 for volunteering to take this on. I'll try to help out when I can. cookiecutter template would have definitely been a big timesaver, so we could try to help out with that first.

So far this project project has been the biggest inspiration of how our app is structured, and although it is very pg centric, it does a lot of things that we thought are sane defaults. I guess we can discuss when the ball is rolling and aiohttp_devtools is a real thing.

[samuelcolvin]

https://github.com/samuelcolvin/aiohttp-devtools

aiohttp-devtools is started!

Emphasis on started, still a lot more to do, however the basic aiohttp-devtools runserver and aiohttp-devtools start are working(ish) and I've uploaded a stub v0.0.1 to pypi to reserve the name.

Closing this issue, any more discussions can happen on that project.

[samuelcolvin]

https://github.com/samuelcolvin/aiohttp-devtools

Is now ready for use in the wild. Feedback very welcome.

@asvetlov please look over the code, and see what you think. If you're willing I think it would be good to mention the package in aiohttp's docs and perhaps use it for the server tutorial.

[fafhrd91]

@samuelcolvin just create PR for doc

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.