Skip to content

Development Roadmap

Jump to bottom
mitchelljkotler edited this page Feb 4, 2016 · 32 revisions

Development Roadmap

It's like a road map, but really just a place for us to discuss and plan how we code

Some of these are one time things we should do, but most are ongoing things we can do to help us be productive and manage technical debt

In no particular order...

Tests

  • better coverage
    • better coverage will make it easier to refactor, upgrade django, etc
      • better infrastructure for running coverage report (run it on codeship and push it live somewhere?)
      • coveralls is another option for running coverage reports
  • use factories instead of fixtures
  • isolate tests
    • In addition to using factories, refactor all tests to isolate them.
    • This will lead to improvements in the design and structure of our data.
    • For more, see this post by Gary Bernhardt
  • test views better
  • better optimized for run time
  • new versions of django have a built-in check command for settings, we could run this on codeship
  • we should probably actually test the provisioning of VMs some how
  • add tests for 3rd party app generated pages (api, api doc, robots, sitemap, etc)
  • Should we add tests for JS? Use something like selenium?

QA

  • We should be checking Sentry regularly and fixing all issues there (Have some sort of formal process for this?)
    • Fixable issues that don't cause much problems should still be fixed ASAP to not hide real bugs
    • Intermittent errors (temporary network problems, etc) should just be cleared to not hide real problems
    • Confusing bugs should be studied and figured out - sometimes very odd things would be in here periodically, and upon closer inspection ended up being important things to fix
  • Should we add sentry logging for JS errors?

Libraries

  • upgrade django whenever we fall behind by 2 versions (ie if we are on 1.5 upgrade once 1.7 is released)
  • when we upgrade django is probably a good time to upgrade other third party libraries as well
  • We could use Django Package Monitor or https://requires.io/ to keep track of packages which need to be updated
    • I installed Django Package Monitor
  • pipdeptree is useful for seeing which packages depend on others

Settings

Code Review

  • we should be reviewing each others code more
    • I know this is hard with my schedule - maybe use gits code commenting feature more?
    • we could handle merges into master as pull requests, giving us a chance to voice concerns before they're live. this wouldn't make sense for bugfixes, but would for new features.
  • be more conservative with pylint disables
    • go through and change old style id # disables to new style name disables so its easy to see whats being disabled
    • see if any of them can be removed
  • remove dead code - have accumulated some cruft over the years

Optimize site

  • make site run faster
    • I think we're pretty fast now, need to make sure we don't regress on performance
  • good profiling/sql monitoring tools, middleware/django debug toolbar
  • new relic
  • have staff report slow pages
  • ~~ run page speed and yslow and other similar tools ~~

Frontend Pipeline

  • better modularization/organization/structure for JS files, particularly
  • ~~ need to gzip and minify static assets (will help site performance post-initial page load) ~~
  • i thought we already minified? (we compress the files on codeship before deploy)
  • Django Gzip middleware - there seems to be security concerns with gzip
  • we also already gzip in nginx (see nginx config). i think we should turn it off though [https://www.djangoproject.com/weblog/2013/aug/06/breach-and-django/], but gziping in nginx should be better performance than in middleware and also gzips static content not touched by middleware
    • Ok, so if I understand the problem correctly it is that by gzipping the initial response that serves the index.html we expose information contained by the response's cookies (like CSRFToken). But examining the responses for the CSS/JS which are requested and loaded after the initial response, those responses don't contain any cookies. gzipping our HTTP in nginx is probably certainly a security risk. But I don't see why we wouldn't gzip our static frontend assets once as part of the production build pipeline rather than doing it through nginx.

Scripts

  • make a script directory with the following
    • ~~ script to run pylint (already checked in) ~~
    • ~~ script to generate coverage report ~~
    • ~~ deployment script (I have a fabfile, would need to make it a little more robust, pretty simple) ~~
    • etc

Dev Database

  • I've just been creating data as i go, @allanlasser started a script to help populate the database initially
  • we could just use a copy of the production database
    • will help expose performance issues with large amounts of data
    • makes debugging data specific errors easier as we could easily recreate them locally
    • this would involve downloading a very large amount of data
    • Heroku Export PG
  • This is now in the fabfile

Data Security

@mitch related to this, what's the state of database backups? How is that done, how could it be done better?

  • we use heroku pg backup - the one thing we that should be done is actually run through restoring from backup so we know how to do it and know it works

API v2.0

  • an API that doesn't just expose data, but also selectively exposes methods for acting on models in predictable ways
  • should really, really avoid directly maniuplating data on models
  • useful for writing more powerful frontend code more quickly, and also adjacent non-website software (Whitewash, etc.)

muckrock SDK

  • in connection with redesigned API, a SDK so someone can do pip install muckrock and have access to some common data and operate on it from within their application
  • basically take the current api-examples repo and make it useful
  • in the spirit of the Stripe library we use in our own app, where the library constructs and performs all the API calls for you in a way that's language-native

Admin

Documentation

  • We basically have zero documentation of our code base
  • Having more than zero would be ideal

Modulerize

  • Try to piece off functionality and open source if possible
    • NestedInlines
    • business_days

Restructure Project

  • It'd be nice if Vagrantfile could live at project root so vagrant can be called from anywhere in the project.
  • Two Scoops recommends nesting the django project directory within a larger 'repo' directory that contains things like requirements and readmes.
    • I believe we already do this
Clone this wiki locally