Falcon Batteries Included

This opinionated project demonstrates how use Falcon and various Python libraries to build a scalable REST API for a movie recommendation website.

Table of Contents

• Design
• Development Workflow
  • Getting Started
  • Search Notes
• Python Best Practices
• Batteries
  • Asynchronous Task Queue
  • Authentication
  • Continuous Integration, Continouous Delivery (CICD)
  • Documentation
  • Full-Text Search
  • ORM (SQLAlchemy)
  • Profiler
  • Security
  • Serialization / Deserialization
  • Testing

---

Design

Use the best tool for the job at hand.

• Most of the CRUD logic is in controllers, but if we have to do perform multiple tasks for an endpoint, a process is kicked off.
• Took inspiration from Twelve-Factor App

Development Workflow

• Development environment leverages Docker-Compose to replicate production environment
• Makefile provides common operations for development
• pre-commit hooks identify comomn code review issues before submission
• CI pipeline is triggered on push to branch and PR creation

Getting Started

1. make up
2. Create virtual environment on local machine, pip install -r requirements_dev.txt to install dependencies locally
3. Point IDE's PYTHONPATH to the python instance in the virtual environment from above to get autocomplete and other tooling working
4. Install pre-commit on your development machine
5. pre-commit install will run existing hook scripts (from .pre-commit-config.yaml)

Server available at http://0.0.0.0:7000/

Search Notes

• Implemented in console in console only, TODO: add search endpoint

Movie.reindex()
Movie.search("top gun", page=1, per_page=5)

Python Best Practices

• Code Formatter: Black
• Logging: Standard Library
• Static Type Checker: mypy
• Updating Dependencies: PyUp

Batteries

Asynchronous Task Queue

• redis + rq
• rq-scheduler to schedule jobs
• rq-dashboard for monitoring. Available at http://0.0.0.0:9181/

Authentication

• JWT authentication via falcon-auth

Continuous Integration, Continouous Delivery (CICD)

• CI builds with drone

Documentation

• apispec + falcon-apispec to generate OpenAPI (aka Swagger) specification
• Serving Redoc-styled docs at http://localhost:7000/swagger/redoc.html

Full-Text Search

• Using elasticsearch via elasticsearch-py
• Leveraging developer utilities from Kibana

ORM (SQLAlchemy)

• Follow pattern described in SQLAlchemy docs
  • Load database (well, declarative base) into the request object
  • Remove database from request before sending response
• Migrations with Alembic

Profiler

• Attach py-spy to Python process to profile
• Connect to Gunicorn process to profile web application
  • Gunicorn architecture

USAGE:
    py-spy --duration <duration> --pid <pid> --rate <rate>

Security

• Bandit is a static analysis tool to find security issues
• 10 common security gotchas in Python and how to avoid them
• Open Web Application Security Project: 2017 Top 10

Serialization / Deserialization

• Marshmallow to serialize objects into JSON (response) and deserialize JSON into object (request)
• webargs to parse requests arguments (query string)
• toasted-marshmallow has 10x performance, investigate

Testing

• pytest
• Functional tests via tavern
  • Works locally and not in drone (currently excluded from CI check)
  • Either write a plugin to have Tavern hit Falcon test API or use Jenkins