Cloud migration (#1215)

* Add docs to app image.

Fixes #1203

* Add blog to Docker image.

Fixes #1204

* Handle CSS generation with JavaScript.

Fixes #1205

* Add strategy for migrating from Postgres to SQLite.

Fixes #1207

* Add unmodified `kamal init` output plus `.tool-versions.

* Fix flaky test.

* Set wal mode on the db.

* Run Docker builds without a .env file.

Fixes #1216

* Changes for PDF checking.

* Add starting kamal configuration.

* Remove psycopg.

* Update referral email subject.

* Add some notes about server hardening.

* Add Huey and task to clear db sessions.

* Ignore huey types.

* Move expire_trials to be a Huey task.

* Move send_referrals to Huey task.

* Move build_bundles to Huey task.

* Cut down CI builds. This project doesn't really get outside contribution.

* Add secrets references.

* Dockerfile updates to reduce image size.

* Set the new IP address.

* Add a healthcheck endpoint.

* Need to hook the healthcheck url to the right view.

* Increase the healthcheck interval to see if that helps the deploy.

* Turn off SSL redirect. I think it disrupts the health check.

* Handle SECURE_SSL_REDIRECT as expected.

* Add a Docker entrypoint script.

* Switch CMD to point to a script with gunicorn.

* Add migrate step.

* Try increasing the healthcheck timeout for migrations to run.

* Try more logging to get a picture of what is going on.

* The first request might have taken just over 30 seconds.

* Clean up the entrypoint script.

* Handle font cache issue.

* Connect the droplet to a staging subdomain.

* Try setting ALLOWED_HOSTS.

* Keep ALLOWED_HOSTS in the wildcard configuration.

* Add some migration notes and configuration.

* Handle the raw parameter to allow loaddata to work.

* Update migration instructions.

* Update to new subdomain.

* Switch to the non-DATABASE_URL method of loading the db.

.dockerignore

@@ -1,3 +1,4 @@
+.env
 db.sqlite3
 node_modules/
 .coverage/
@@ -6,3 +7,5 @@ node_modules/
 .pytest_cache/
 .ruff_cache/
 .venv/
+venv/
+blog_out/

.env.example

@@ -5,8 +5,8 @@ AWS_SECRET_ACCESS_KEY=asecrettoeverybody
 AWS_STORAGE_BUCKET_NAME=some-bucket
 CSRF_COOKIE_SECURE=off
 DATABASE_URL=sqlite:///db.sqlite3
-# Postgres example
-# DATABASE_URL=postgres://postgres:postgres@localhost:5432/mylocaldb
+# Postgres example - assuming postgres image in Docker Compose is named `db`.
+# DATABASE_URL=postgres://postgres:postgres@db:5432/postgres
 DATABASE_SSL_REQUIRE=off
 DEBUG=on
 DEBUG_TOOLBAR=on

.github/workflows/tests.yaml

@@ -1,7 +1,6 @@
 name: Tests Runner
 on:
   - push
-  - pull_request
 
 jobs:
   test:
@@ -17,7 +16,6 @@ jobs:
       HASHID_FIELD_SALT: 'a secret to everybody'
       IS_SECURE: 'off'
       SECRET_KEY: 'a secret to everybody'
-      SECURE_SSL_REDIRECT: 'off'
       SENDGRID_API_KEY: 'a secret to everybody'
       SENTRY_ENABLED: 'off'
       SENTRY_DSN: 'dsn_example'
@@ -52,7 +50,6 @@ jobs:
         env:
           DATABASE_URL: 'sqlite://:memory:'
           SECRET_KEY: 'zt(6jlr#oquxm2t%ryh#n+-72p^(3knbf&q$5x16#o%1im-s7!'
-          SECURE_SSL_REDIRECT: 'on'
 
       - name: Check documentation build
         run: make docs

.gitignore

@@ -54,7 +54,7 @@ coverage.xml
 # Django stuff:
 *.log
 local_settings.py
-db.sqlite3
+*.sqlite3*
 
 # Flask stuff:
 instance/

.kamal/hooks/docker-setup

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+# Split the KAMAL_HOSTS by commas and loop through each IP
+IFS=',' read -ra HOSTS <<< "$KAMAL_HOSTS"
+for HOST in "${HOSTS[@]}"; do
+  echo "Connecting to $HOST"
+
+  ssh root@"$HOST" "
+    if ! getent group app > /dev/null 2>&1; then
+      groupadd --gid 222 --system app
+      echo 'System group app created with GID 222'
+    else
+      echo 'System group app already exists'
+    fi
+
+    if ! id --user app > /dev/null 2>&1; then
+      useradd --uid 222 --gid app --system app
+      echo 'System user app created with UID 222'
+    else
+      echo 'System user app already exists'
+    fi
+
+    mkdir --parents /var/db
+
+    chown --recursive app:app /var/db
+    echo '/var/db ownership changed to app:app'
+  "
+done

.kamal/hooks/docker-setup.sample

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+
+# A sample docker-setup hook
+#
+# Sets up a Docker network on defined hosts which can then be used by the application’s containers
+
+hosts = ENV["KAMAL_HOSTS"].split(",")
+
+hosts.each do |ip|
+  destination = "root@#{ip}"
+  puts "Creating a Docker network \"kamal\" on #{destination}"
+  `ssh #{destination} docker network create kamal`
+end

.kamal/hooks/post-deploy.sample

@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# A sample post-deploy hook
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+# KAMAL_RUNTIME
+
+echo "$KAMAL_PERFORMER deployed $KAMAL_VERSION to $KAMAL_DESTINATION in $KAMAL_RUNTIME seconds"

.kamal/hooks/post-proxy-reboot.sample

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Rebooted kamal-proxy on $KAMAL_HOSTS"

.kamal/hooks/pre-build.sample

@@ -0,0 +1,51 @@
+#!/bin/sh
+
+# A sample pre-build hook
+#
+# Checks:
+# 1. We have a clean checkout
+# 2. A remote is configured
+# 3. The branch has been pushed to the remote
+# 4. The version we are deploying matches the remote
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+
+if [ -n "$(git status --porcelain)" ]; then
+  echo "Git checkout is not clean, aborting..." >&2
+  git status --porcelain >&2
+  exit 1
+fi
+
+first_remote=$(git remote)
+
+if [ -z "$first_remote" ]; then
+  echo "No git remote set, aborting..." >&2
+  exit 1
+fi
+
+current_branch=$(git branch --show-current)
+
+if [ -z "$current_branch" ]; then
+  echo "Not on a git branch, aborting..." >&2
+  exit 1
+fi
+
+remote_head=$(git ls-remote $first_remote --tags $current_branch | cut -f1)
+
+if [ -z "$remote_head" ]; then
+  echo "Branch not pushed to remote, aborting..." >&2
+  exit 1
+fi
+
+if [ "$KAMAL_VERSION" != "$remote_head" ]; then
+  echo "Version ($KAMAL_VERSION) does not match remote HEAD ($remote_head), aborting..." >&2
+  exit 1
+fi
+
+exit 0

.kamal/hooks/pre-connect.sample

@@ -0,0 +1,47 @@
+#!/usr/bin/env ruby
+
+# A sample pre-connect check
+#
+# Warms DNS before connecting to hosts in parallel
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+# KAMAL_RUNTIME
+
+hosts = ENV["KAMAL_HOSTS"].split(",")
+results = nil
+max = 3
+
+elapsed = Benchmark.realtime do
+  results = hosts.map do |host|
+    Thread.new do
+      tries = 1
+
+      begin
+        Socket.getaddrinfo(host, 0, Socket::AF_UNSPEC, Socket::SOCK_STREAM, nil, Socket::AI_CANONNAME)
+      rescue SocketError
+        if tries < max
+          puts "Retrying DNS warmup: #{host}"
+          tries += 1
+          sleep rand
+          retry
+        else
+          puts "DNS warmup failed: #{host}"
+          host
+        end
+      end
+
+      tries
+    end
+  end.map(&:value)
+end
+
+retries = results.sum - hosts.size
+nopes = results.count { |r| r == max }
+
+puts "Prewarmed %d DNS lookups in %.2f sec: %d retries, %d failures" % [ hosts.size, elapsed, retries, nopes ]

.kamal/hooks/pre-deploy.sample

@@ -0,0 +1,109 @@
+#!/usr/bin/env ruby
+
+# A sample pre-deploy hook
+#
+# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
+#
+# Fails unless the combined status is "success"
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_COMMAND
+# KAMAL_SUBCOMMAND
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+
+# Only check the build status for production deployments
+if ENV["KAMAL_COMMAND"] == "rollback" || ENV["KAMAL_DESTINATION"] != "production"
+  exit 0
+end
+
+require "bundler/inline"
+
+# true = install gems so this is fast on repeat invocations
+gemfile(true, quiet: true) do
+  source "https://rubygems.org"
+
+  gem "octokit"
+  gem "faraday-retry"
+end
+
+MAX_ATTEMPTS = 72
+ATTEMPTS_GAP = 10
+
+def exit_with_error(message)
+  $stderr.puts message
+  exit 1
+end
+
+class GithubStatusChecks
+  attr_reader :remote_url, :git_sha, :github_client, :combined_status
+
+  def initialize
+    @remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
+    @git_sha = `git rev-parse HEAD`.strip
+    @github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
+    refresh!
+  end
+
+  def refresh!
+    @combined_status = github_client.combined_status(remote_url, git_sha)
+  end
+
+  def state
+    combined_status[:state]
+  end
+
+  def first_status_url
+    first_status = combined_status[:statuses].find { |status| status[:state] == state }
+    first_status && first_status[:target_url]
+  end
+
+  def complete_count
+    combined_status[:statuses].count { |status| status[:state] != "pending"}
+  end
+
+  def total_count
+    combined_status[:statuses].count
+  end
+
+  def current_status
+    if total_count > 0
+      "Completed #{complete_count}/#{total_count} checks, see #{first_status_url} ..."
+    else
+      "Build not started..."
+    end
+  end
+end
+
+
+$stdout.sync = true
+
+puts "Checking build status..."
+attempts = 0
+checks = GithubStatusChecks.new
+
+begin
+  loop do
+    case checks.state
+    when "success"
+      puts "Checks passed, see #{checks.first_status_url}"
+      exit 0
+    when "failure"
+      exit_with_error "Checks failed, see #{checks.first_status_url}"
+    when "pending"
+      attempts += 1
+    end
+
+    exit_with_error "Checks are still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds" if attempts == MAX_ATTEMPTS
+
+    puts checks.current_status
+    sleep(ATTEMPTS_GAP)
+    checks.refresh!
+  end
+rescue Octokit::NotFound
+  exit_with_error "Build status could not be found"
+end

.kamal/hooks/pre-proxy-reboot.sample

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Rebooting kamal-proxy on $KAMAL_HOSTS..."

.kamal/secrets

@@ -0,0 +1,21 @@
+# Secrets defined here are available for reference under registry/password, env/secret, builder/secrets,
+# and accessories/*/env/secret in config/deploy.yml. All secrets should be pulled from either
+# password manager, ENV, or a file. DO NOT ENTER RAW CREDENTIALS HERE! This file needs to be safe for git.
+
+# Docker build
+KAMAL_REGISTRY_PASSWORD=$(op read op://school-desk/server/KAMAL_REGISTRY_PASSWORD)
+
+# App secrets
+AWS_ACCESS_KEY_ID=$(op read op://school-desk/server/AWS_ACCESS_KEY_ID)
+AWS_SECRET_ACCESS_KEY=$(op read op://school-desk/server/AWS_SECRET_ACCESS_KEY)
+DJSTRIPE_WEBHOOK_SECRET=$(op read op://school-desk/server/DJSTRIPE_WEBHOOK_SECRET)
+HASHID_FIELD_SALT=$(op read op://school-desk/server/HASHID_FIELD_SALT)
+ROLLBAR_ACCESS_TOKEN=$(op read op://school-desk/server/ROLLBAR_ACCESS_TOKEN)
+SECRET_KEY=$(op read op://school-desk/server/SECRET_KEY)
+SENDGRID_API_KEY=$(op read op://school-desk/server/SENDGRID_API_KEY)
+SENTRY_DSN=$(op read op://school-desk/server/SENTRY_DSN)
+SLACK_WEBHOOK=$(op read op://school-desk/server/SLACK_WEBHOOK)
+STRIPE_LIVE_PUBLISHABLE_KEY=$(op read op://school-desk/server/STRIPE_LIVE_PUBLISHABLE_KEY)
+STRIPE_LIVE_SECRET_KEY=$(op read op://school-desk/server/STRIPE_LIVE_SECRET_KEY)
+STRIPE_TEST_PUBLISHABLE_KEY=$(op read op://school-desk/server/STRIPE_TEST_PUBLISHABLE_KEY)
+STRIPE_TEST_SECRET_KEY=$(op read op://school-desk/server/STRIPE_TEST_SECRET_KEY)

.tool-versions

@@ -0,0 +1 @@
+ruby 3.3.4