Updated Celery support in Figures development environment

* Added `django-celery` to devsite and Hawthorn multisite requirements
file
* Added the Docker commands as Figures Makefile targets
* Updated devsite/README.md instructions
* Added Celery results support using `django-celery` and the Django
devsite database as the backend

Makefile

@@ -56,6 +56,21 @@ ginkgo.pytest:  ## Run Pytest for the Ginkgo environment
 ginkgo.tox:  ## Run tox just for the Ginkgo environment
 	tox -e py27-ginkgo
 
+### Devsite Docker targets
+
+devsite.docker.prep: ## state needed to run devsite docker
+	pip install -e .
+
+devsite.docker.up: devsite.docker.prep  ## Start devsite docker
+	cd devsite; docker-compose up
+
+devsite.docker.rabbitmq.config:  ## configure RabbitMQ container
+	cd devsite; docker cp rabbitmq-init.sh devsite_rabbitmq_figures_1:/rabbitmq-init.sh ; \
+		docker exec devsite_rabbitmq_figures_1 /rabbitmq-init.sh
+
+devsite.docker.celery.start:  ## Start the celery server
+	cd devsite; celery -A devsite worker --logleve=info
+
 ### Automatically constructed Virtualenv based targets
 
 ve/bin/figures-ws: devsite/requirements/${EDX_PLATFORM_RELEASE}_appsembler.txt

devsite/README.md

@@ -23,80 +23,124 @@ Docker needs to be installed in your development environment
 
 ### Setup and Configuration
 
-Note: This is a WIP
+Note: This is an initial implementation. Steps may break.
 
-We run the celery worker from the `figures/devsite` directory. It needs to discover Figures.
+The setup and configuration instructions can be mostly run with makefile targets
+in `<figures-project-root>/Makefile`. The steps run from the makefile are also
+described following the makefile target steps
 
-Install Figures in the devsite venv:
+#### Background information
 
-From the Figures project root directory, run:
+* We run the celery worker from the `figures/devsite` directory.
+* The celery worker needs to discover Figures. In order to do so, we install Figures in the virtualenv we use for our project work. Because we are using Celery for Figures development, we install with the pip editable `-e` option. 
+* In addition to any terminals you currently open for your Figure development, we need two additional terminals
+  * RabbitMQ container
+  * Celery worker
 
-```
-pip install -e figures
-```
 
+#### Steps with Make
+
+1. Go to your Figures project root,  For example `~/projects/figures`. Each time you need to open a new terminal window, you should activate your project's [Python virtual environment](https://www.pythonforbeginners.com/basics/how-to-use-python-virtualenv/). Ther term "Python virtual environment" is typically abbreviated to "venv", which we will use in the following instructions
+
+2. Start your local [Docker](https://www.docker.com) server. Follow instructions specific to your development system's operating system
+
+3. Prepare your development docker environment. Open a terminal, activate your venv and run the following
+
+`make devsite.docker.prep`
+
+This step runs `pip install -e .` to install Figures in the virtualenv, which is needed by the Celery worker
+
+4. Start up Figures Docker containers. Open a terminal, activate your venv and run the following
 
-* Start Docker on your development machine
+`make devsite.docker.up`
 
-Go to the `figures/devsite` directory and from the shell, run `docker-compose up`
+This step changes directorey to `<figures-project-root>devsite` and runs `docker-compose up`
 
-This will launch the docker container and show console output
+5. Configure the RabbitMQ container. Open a terminal, activate your venv and run the following
 
-In another shell, go to the `figures/devsite` directory and run
+`make devsite.docker.rabbitmq.config`  
+
+This step changes directory to `<figures-project-root>devsite` and updates the RabbitMQ container, creates the figures user, adds a vhost for Figures, sets tags and permissions with the following calls
 
 ```
 docker cp rabbitmq-init.sh devsite_rabbitmq_figures_1:/rabbitmq-init.sh
+docker exec devsite_rabbitmq_figures_1 /rabbitmq-init.sh
 ```
 
-See the [docker cp](https://docs.docker.com/engine/reference/commandline/cp/) CLI documentation.
+Docker command reference
 
+* [docker cp](https://docs.docker.com/engine/reference/commandline/cp/) CLI documentation.
+* [docker exec](https://docs.docker.com/engine/reference/commandline/exec/) CLI documentation.
 
-Run the following script to configure the Figures environment in RabbitMQ
 
+6. Run migration in Figures Django environment
 ```
-docker exec devsite_rabbitmq_figures_1 /rabbitmq-init.sh
+./manage.py migrate djcelery
+Operations to perform:
+  Apply all migrations: djcelery
+Running migrations:
+  Applying djcelery.0001_initial... OK
 ```
 
-See the [docker exec](https://docs.docker.com/engine/reference/commandline/exec/) CLI documentation.
 
-Note: you can shell into the docker container with the following:
+7. Start your Celery worker. You can reuse the terminal from step 5 to start your celery worker. From an open terminal with the Figures venv activated, run the following
+
+`make devsite.docker.celery.start`
+
+8. Now you can test your Figures development environment Celery setup by running a diagnostic [Django management command](https://docs.djangoproject.com/en/3.0/howto/custom-management-commands/) in devsite. Open a terminal, activate your venv and run the following
 
 ```
-docker exec -ti devsite_rabbitmq_figures_1 /bin/bash
+cd <figures-project-root>/devsite
+./manage.py check_devsite
 ```
 
-Update Figures devsite:
+If all goes well, then you should see the following:
+
+
+from the terminal which you ran `check_devsite`
 
 
 ```
-./manage.py createcachetable
-./manage.py migrate django_celery_results
-Operations to perform:
-  Apply all migrations: django_celery_results
-Running migrations:
-  Applying django_celery_results.0001_initial... OK
-  Applying django_celery_results.0002_add_task_name_args_kwargs... OK
-  Applying django_celery_results.0003_auto_20181106_1101... OK
-  Applying django_celery_results.0004_auto_20190516_0412... OK
-  Applying django_celery_results.0005_taskresult_worker... OK
-  Applying django_celery_results.0006_taskresult_date_created... OK
-  Applying django_celery_results.0007_remove_taskresult_hidden... OK
+./manage.py check_devsite
+Figures devsite system check.
+Checking Celery...
+Task called. task_id=f9d3a72c-2a06-4ed4-9203-ada9df2be649
+result=figures-devsite-celery-check:run_devsite_check management command
+Done checking Celery
+Done.
+```
+
+And from the open Celery worker terminal window
+
+```
+[2020-06-26 21:30:48,777: INFO/MainProcess] Received task: devsite.celery.celery_check[f9d3a72c-2a06-4ed4-9203-ada9df2be649]
+[2020-06-26 21:30:48,789: WARNING/Worker-1] Called devsite.celery.celery.check with message "run_devsite_check management command"
+[2020-06-26 21:30:48,807: INFO/MainProcess] Task devsite.celery.celery_check[f9d3a72c-2a06-4ed4-9203-ada9df2be649] succeeded in 0.0195774619933s: u'figures-devsite-celery-check:run_devsite_check management command'
 ```
 
+### Working with Figures Docker devsite
 
-From the `figures/devsite` directory:
+_This section is a starting point to work with the Docker containers in the Figures development environment_
+
+You can shell into the RabbitMQ Docker container with the following:
 
 ```
-celery -A devsite worker --logleve=info
+docker exec -ti devsite_rabbitmq_figures_1 /bin/bash
 ```
 
 # Testing and Exploring
 
-In `figures/devsite`, run `./manage.py devsite_check`
+In `figures/devsite`, run `./manage.py check_devsite`
 
 
 # References
 
+## Django Celery
+
+We need to use `django-celery` for Figures running and mocking Open edX Hawthorn. This is because Hawthorn runs Celery 3.1.x. With Celery 4.4.x, we won't need `django-celery` anymore
+
+* https://github.com/celery/django-celery
+
 ## Celery Results
 
 * https://github.com/celery/django-celery-results

devsite/devsite/celery.py

@@ -4,6 +4,7 @@
 from __future__ import absolute_import, unicode_literals
 import os
 from celery import Celery
+from django.conf import settings
 
 
 CELERY_CHECK_MSG_PREFIX = 'figures-devsite-celery-check'
@@ -14,14 +15,25 @@
 
 app = Celery('devsite')
 
+# For Celery 4.0+
+#
 # Using a string here means the worker doesn't have to serialize
 # the configuration object to child processes.
 # - namespace='CELERY' means all celery-related configuration keys
 #   should have a `CELERY_` prefix.
-app.config_from_object('django.conf:settings', namespace='CELERY')
+# See: https://docs.celeryproject.org/en/4.0/whatsnew-4.0.html
+# `app.config_from_object('django.conf:settings', namespace='CELERY')`
+
+app.config_from_object('django.conf:settings')
+
 
 # Load task modules from all registered Django app configs.
-app.autodiscover_tasks()
+app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
+
+
+app.conf.update(
+    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
+)
 
 
 @app.task(bind=True)

devsite/devsite/settings.py

@@ -36,7 +36,8 @@
 # Set the default Site (django.contrib.sites.models.Site)
 SITE_ID = 1
 
-USE_CELERY_RESULTS = True
+# TODO: Update this to allow environment variable override
+ENABLE_DEVSITE_CELERY = True
 
 # Adds the mock edx-platform modules to the Python module search path
 sys.path.append(os.path.normpath(os.path.join(PROJECT_ROOT_DIR, MOCKS_DIR)))
@@ -70,8 +71,8 @@
     'student',
 ]
 
-if USE_CELERY_RESULTS:
-    INSTALLED_APPS.append('django_celery_results')
+if ENABLE_DEVSITE_CELERY:
+    INSTALLED_APPS.append('djcelery')
 
 # certificates app
 
@@ -167,30 +168,23 @@
 }
 
 
-# Temporary and only for dev env, we want to get the secrets fro ENV
-# CELERY_BROKER_URL='amqp://celery_user:celery_pwd@localhost:5672/myvhost'
+if ENABLE_DEVSITE_CELERY:
+    # TODO: update to allow environemnt variable overrides
+    # the password seting is only for local development environments
+    FIGURES_CELERY_USER='figures_user'
+    FIGURES_CELERY_PASSWORD='figures_pwd'
+    FIGURES_CELERY_VHOST='figures_vhost'
 
-FIGURES_CELERY_USER='figures_user'
-FIGURES_CELERY_PASSWORD='figures_pwd'
-FIGURES_CELERY_VHOST='figures_vhost'
-
-CELERY_BROKER_URL='amqp://{user}:{password}@localhost:5672/{vhost}'.format(
-    user=FIGURES_CELERY_USER,
-    password=FIGURES_CELERY_PASSWORD,
-    vhost=FIGURES_CELERY_VHOST,
-)
-
-if USE_CELERY_RESULTS:
-    CELERY_CACHE_BACKEND = 'default'
-
-    CELERY_CACHES = {
-        'default': {
-            'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
-            'LOCATION': 'devsite_cache'
+    BROKER_URL='amqp://{user}:{password}@localhost:5672/{vhost}'.format(
+        user=FIGURES_CELERY_USER,
+        password=FIGURES_CELERY_PASSWORD,
+        vhost=FIGURES_CELERY_VHOST,
+    )
 
-        }
-    }
+    CELERY_RESULT_BACKEND='djcelery.backends.cache:CacheBackend'
 
+    import djcelery
+    djcelery.setup_loader()
 
 #
 # Declare empty dicts for settings required by Figures

devsite/requirements/hawthorn_multisite.txt

@@ -10,7 +10,7 @@
 ###
 
 celery==3.1.25
-django-celery-results==1.2.1
+django-celery==3.3.1
 
 # Faker is used to seed mock data in devsite
 Faker==2.0.3