Initial implementation of Celery support for Figures devsite

.coveragerc

@@ -3,3 +3,4 @@ omit =
     setup.py
     tests/*
     mocks/*
+    devsite/devsite/celery.py

Makefile

@@ -56,6 +56,21 @@ ginkgo.pytest:  ## Run Pytest for the Ginkgo environment
 ginkgo.tox:  ## Run tox just for the Ginkgo environment
 	tox -e py27-ginkgo
 
+### Devsite Docker targets
+
+devsite.docker.prep: ## state needed to run devsite docker
+	pip install -e .
+
+devsite.docker.up: devsite.docker.prep  ## Start devsite docker
+	cd devsite; docker-compose up
+
+devsite.docker.rabbitmq.config:  ## configure RabbitMQ container
+	cd devsite; docker cp rabbitmq-init.sh devsite_rabbitmq_figures_1:/rabbitmq-init.sh ; \
+		docker exec devsite_rabbitmq_figures_1 /rabbitmq-init.sh
+
+devsite.docker.celery.start:  ## Start the celery server
+	cd devsite; celery -A devsite worker --logleve=info
+
 ### Automatically constructed Virtualenv based targets
 
 ve/bin/figures-ws: devsite/requirements/${EDX_PLATFORM_RELEASE}_appsembler.txt

devsite/README.md

@@ -0,0 +1,147 @@
+# Figures Devsite
+
+
+## Overview
+
+This doc is a work in progress. Key details should exist, but needs fleshing out
+for readability and completeness
+
+
+## Status
+
+Figures devsite Celery support is in its initial phase. Devsite can now run
+Figures tasks asynchronously
+
+## Pipeline Sandbox
+
+Figures devsite has an option to run Celery tasks in devsite.
+
+
+### Requirements / Prerequisites
+
+Docker needs to be installed in your development environment
+
+### Setup and Configuration
+
+Note: This is an initial implementation. Steps may break.
+
+The setup and configuration instructions can be mostly run with makefile targets
+in `<figures-project-root>/Makefile`. The steps run from the makefile are also
+described following the makefile target steps
+
+#### Background information
+
+* We run the celery worker from the `figures/devsite` directory.
+* The celery worker needs to discover Figures. In order to do so, we install Figures in the virtualenv we use for our project work. Because we are using Celery for Figures development, we install with the pip editable `-e` option. 
+* In addition to any terminals you currently open for your Figure development, we need two additional terminals
+  * RabbitMQ container
+  * Celery worker
+
+
+#### Steps with Make
+
+1. Go to your Figures project root,  For example `~/projects/figures`. Each time you need to open a new terminal window, you should activate your project's [Python virtual environment](https://www.pythonforbeginners.com/basics/how-to-use-python-virtualenv/). Ther term "Python virtual environment" is typically abbreviated to "venv", which we will use in the following instructions
+
+2. Start your local [Docker](https://www.docker.com) server. Follow instructions specific to your development system's operating system
+
+3. Prepare your development docker environment. Open a terminal, activate your venv and run the following
+
+`make devsite.docker.prep`
+
+This step runs `pip install -e .` to install Figures in the virtualenv, which is needed by the Celery worker
+
+4. Start up Figures Docker containers. Open a terminal, activate your venv and run the following
+
+`make devsite.docker.up`
+
+This step changes directorey to `<figures-project-root>devsite` and runs `docker-compose up`
+
+5. Configure the RabbitMQ container. Open a terminal, activate your venv and run the following
+
+`make devsite.docker.rabbitmq.config`  
+
+This step changes directory to `<figures-project-root>devsite` and updates the RabbitMQ container, creates the figures user, adds a vhost for Figures, sets tags and permissions with the following calls
+
+```
+docker cp rabbitmq-init.sh devsite_rabbitmq_figures_1:/rabbitmq-init.sh
+docker exec devsite_rabbitmq_figures_1 /rabbitmq-init.sh
+```
+
+Docker command reference
+
+* [docker cp](https://docs.docker.com/engine/reference/commandline/cp/) CLI documentation.
+* [docker exec](https://docs.docker.com/engine/reference/commandline/exec/) CLI documentation.
+
+
+6. Run migration in Figures Django environment
+```
+./manage.py migrate djcelery
+Operations to perform:
+  Apply all migrations: djcelery
+Running migrations:
+  Applying djcelery.0001_initial... OK
+```
+
+
+7. Start your Celery worker. You can reuse the terminal from step 5 to start your celery worker. From an open terminal with the Figures venv activated, run the following
+
+`make devsite.docker.celery.start`
+
+8. Now you can test your Figures development environment Celery setup by running a diagnostic [Django management command](https://docs.djangoproject.com/en/3.0/howto/custom-management-commands/) in devsite. Open a terminal, activate your venv and run the following
+
+```
+cd <figures-project-root>/devsite
+./manage.py check_devsite
+```
+
+If all goes well, then you should see the following:
+
+
+from the terminal which you ran `check_devsite`
+
+
+```
+./manage.py check_devsite
+Figures devsite system check.
+Checking Celery...
+Task called. task_id=f9d3a72c-2a06-4ed4-9203-ada9df2be649
+result=figures-devsite-celery-check:run_devsite_check management command
+Done checking Celery
+Done.
+```
+
+And from the open Celery worker terminal window
+
+```
+[2020-06-26 21:30:48,777: INFO/MainProcess] Received task: devsite.celery.celery_check[f9d3a72c-2a06-4ed4-9203-ada9df2be649]
+[2020-06-26 21:30:48,789: WARNING/Worker-1] Called devsite.celery.celery.check with message "run_devsite_check management command"
+[2020-06-26 21:30:48,807: INFO/MainProcess] Task devsite.celery.celery_check[f9d3a72c-2a06-4ed4-9203-ada9df2be649] succeeded in 0.0195774619933s: u'figures-devsite-celery-check:run_devsite_check management command'
+```
+
+### Working with Figures Docker devsite
+
+_This section is a starting point to work with the Docker containers in the Figures development environment_
+
+You can shell into the RabbitMQ Docker container with the following:
+
+```
+docker exec -ti devsite_rabbitmq_figures_1 /bin/bash
+```
+
+# Testing and Exploring
+
+In `figures/devsite`, run `./manage.py check_devsite`
+
+
+# References
+
+## Django Celery
+
+We need to use `django-celery` for Figures running and mocking Open edX Hawthorn. This is because Hawthorn runs Celery 3.1.x. With Celery 4.4.x, we won't need `django-celery` anymore
+
+* https://github.com/celery/django-celery
+
+## Celery Results
+
+* https://github.com/celery/django-celery-results
+* https://stackoverflow.com/questions/26934522/celery-result-get-not-working

devsite/devsite/__init__.py

@@ -0,0 +1,10 @@
+"""Site init file
+"""
+
+from __future__ import absolute_import, unicode_literals
+
+# This will make sure the app is always imported when
+# Django starts so that shared_task will use this app.
+from .celery import app as celery_app
+
+__all__ = ('celery_app',)

devsite/devsite/celery.py

@@ -0,0 +1,51 @@
+"""Site Celery setup
+"""
+
+from __future__ import absolute_import, unicode_literals
+import os
+from celery import Celery
+from django.conf import settings
+
+
+CELERY_CHECK_MSG_PREFIX = 'figures-devsite-celery-check'
+
+
+# set the default Django settings module for the 'celery' program.
+os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'devsite.settings')
+
+app = Celery('devsite')
+
+# For Celery 4.0+
+#
+# Using a string here means the worker doesn't have to serialize
+# the configuration object to child processes.
+# - namespace='CELERY' means all celery-related configuration keys
+#   should have a `CELERY_` prefix.
+# See: https://docs.celeryproject.org/en/4.0/whatsnew-4.0.html
+# `app.config_from_object('django.conf:settings', namespace='CELERY')`
+
+app.config_from_object('django.conf:settings')
+
+
+# Load task modules from all registered Django app configs.
+app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
+
+
+app.conf.update(
+    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
+)
+
+
+@app.task(bind=True)
+def debug_task(self):
+    print('Request: {0!r}'.format(self.request))
+
+
+@app.task(bind=True)
+def celery_check(self, msg):
+    """Basic system check to check Celery results in devsite
+
+    Returns a value so that we can test Celery results backend configuration
+    """
+    print('Called devsite.celery.celery.check with message "{}"'.format(msg))
+    return '{prefix}:{msg}'.format(prefix=CELERY_CHECK_MSG_PREFIX, msg=msg)

devsite/devsite/management/commands/check_devsite.py

@@ -0,0 +1,45 @@
+"""This command serves to run system checks for Figures devsite
+
+Initially, this is a convenience command to check that Celery on devsite works
+properly
+
+It calls the task `devsite.celery.run_celery_check`
+
+"""
+
+from django.core.management.base import BaseCommand
+
+from devsite.celery import celery_check
+
+
+class Command(BaseCommand):
+
+    def run_devsite_celery_task(self):
+        """Perform basic Celery checking
+
+        In production, we typically don't want to call `.get()`, but trying it
+        here just to see if the results backend is configured and working
+
+        See the `get` method here:
+            https://docs.celeryproject.org/en/stable/reference/celery.result.html
+        """
+        print('Checking Celery...')
+        msg = 'run_devsite_check management command'
+        result = celery_check.delay(msg)
+        print('Task called. task_id={}'.format(result.task_id))
+
+        try:
+            print('result={}'.format(result.get()))
+        except NotImplementedError as e:
+            print('Error: {}'.format(e))
+
+        print('Done checking Celery')
+
+    def add_arguments(self, parser):
+        """Stub"""
+        pass
+
+    def handle(self, *args, **options):
+        print('Figures devsite system check.')
+        self.run_devsite_celery_task()
+        print('Done.')

devsite/devsite/settings.py

@@ -36,6 +36,9 @@
 # Set the default Site (django.contrib.sites.models.Site)
 SITE_ID = 1
 
+# TODO: Update this to allow environment variable override
+ENABLE_DEVSITE_CELERY = True
+
 # Adds the mock edx-platform modules to the Python module search path
 sys.path.append(os.path.normpath(os.path.join(PROJECT_ROOT_DIR, MOCKS_DIR)))
 
@@ -68,6 +71,9 @@
     'student',
 ]
 
+if ENABLE_DEVSITE_CELERY:
+    INSTALLED_APPS.append('djcelery')
+
 # certificates app
 
 if OPENEDX_RELEASE == 'GINKGO':
@@ -162,6 +168,24 @@
 }
 
 
+if ENABLE_DEVSITE_CELERY:
+    # TODO: update to allow environemnt variable overrides
+    # the password seting is only for local development environments
+    FIGURES_CELERY_USER = 'figures_user'
+    FIGURES_CELERY_PASSWORD = 'figures_pwd'
+    FIGURES_CELERY_VHOST = 'figures_vhost'
+
+    BROKER_URL = 'amqp://{user}:{password}@localhost:5672/{vhost}'.format(
+        user=FIGURES_CELERY_USER,
+        password=FIGURES_CELERY_PASSWORD,
+        vhost=FIGURES_CELERY_VHOST,
+    )
+
+    CELERY_RESULT_BACKEND = 'djcelery.backends.cache:CacheBackend'
+
+    import djcelery
+    djcelery.setup_loader()
+
 #
 # Declare empty dicts for settings required by Figures
 #

devsite/docker-compose.yml

@@ -0,0 +1,10 @@
+version: '3.5'
+services:
+
+  rabbitmq_figures:
+    # build: ./rabbitmq
+    image: rabbitmq:3
+    env_file: rabbitmq.env
+    ports:
+      - 5672:5672
+      - 15672:15672

devsite/rabbitmq-init.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+#
+# Run this script from within the RabbitMQ docker container
+
+rabbitmqctl add_user figures_user figures_pwd
+rabbitmqctl add_vhost figures_vhost
+rabbitmqctl set_user_tags figures_user figures_tag
+rabbitmqctl set_permissions -p figures_vhost figures_user ".*" ".*" ".*"

devsite/rabbitmq.env

@@ -0,0 +1,4 @@
+# RabbitMQ environment settings for Figures devsite
+
+RABBITMQ_DEFAULT_USER=rabbitadmin
+RABBITMQ_DEFAULT_PASS=gr0kBand

devsite/requirements/hawthorn_multisite.txt

@@ -10,6 +10,7 @@
 ###
 
 celery==3.1.25
+django-celery==3.3.1
 
 # Faker is used to seed mock data in devsite
 Faker==2.0.3