Tasks not added to _tasks[] queue for some reason?

[givity]

I setup a rather-simple function to be queued with your lovely decorator and after executing the function, the task is in queue but receives an error message coming from your very own tasks.py. Note that I'm running manage.py process_tasks in a separate terminal on my Windows 7 dev box; I keep coming up with the following error:

Traceback (most recent call last):
File "C:\Users\givity\Envs\givity\lib\site-packages\background_task\tasks.py", line 162, in run_task
tasks.run_task(task.task_name, args, kwargs)
File "C:\Users\givity\Envs\givity\lib\site-packages\background_task\tasks.py", line 46, in run_task
task = self._tasks[task_name]
KeyError: u'projects.lib.files.files_chop'
WARNING:root:Rescheduling task Task(projects.lib.files.files_chop) for 0:00:21 later at 2011-12-30 04:28:00.596000

c:\users\givity\envs\givity\lib\site-packages\background_task\tasks.py(46)run_task()

I added a pdb.set_trace() right above line 44 (the site of the crash) to see if I could grasp what was going on:

-> task = self._taskstask_name self._tasks
{}

I noted that self._tasks was consistently empty... this explains the error message, but not the underlying problem?

The code for my actual tasks is as follows:

@background(schedule=60)
def files_chop(path):
    """ chop away div elements with class givity-chop
        """
    # change the current working dir to the new project folder
    os.chdir(path)
    logger.info('changed folder to %s' % path)

    # remove all class="givity-chop" divs, etc.
    files = find_files(path, search_pattern='*.html')
    logger.info('files found: %s' % files)
    ...

Of course, I'm calling the function files_chop(path) elsewhere:

files_chop(project_path_src)

Thanks for the great app; it's a lot leaner than Celery and looks like it has a lot of promise. You are also #2 in the Google results for "django background task" (props on choosing a good package name).

Givity

[lilspikey]

You need to make sure that your task function exists somewhere that the management command will load it. To aid in this the process_tasks command will look for a tasks.py file in each app and import it if it's found. The other alternative is to put the function in the models.py file of the app.

I'd best document that tasks.py thing...

Glad you like the app, was aiming for very simple. I imagine Celery is better for a larger/more complex setup, but often a basic database backed workqueue is all that's needed.

[lilspikey]

Actually just noticed I had documented the tasks.py thing. See the NB. under "Running tasks" in the README.

[givity]

Thanks a bunch! It's working now with the background function moved into tasks.py.

Is there anyway to use the @background decorator and then determine (within the @background function) which task.id is created?

All the best from Texas,

Givity

[lilspikey]

There's no way (currently) to the the id of the task that is being run. Do you have a particular need for the id?

As part of the "simplicity" approach I've tried not to expose details of how the tasks are scheduled too much - there is some scope in the code for having different backends for storing the tasks to run.

[givity]

I guess I'm trying to write a callback to access a database object within a task; since passing an actual object as a paramater is impossible (params are stored unicode encoded in the db), I've tried many approaches with many fails:

From views.py:

task_x( base_path=settings.BASE_DIR,
                                destination_path=destination_path, 
                                callback_params={'project_id': project.id} )

From tasks.py:

Part of the actual task_x function (snippet):

if project_id and callback_params:
        # we have a callback string, execute it with eval!
        callback_params['status'] = status # this is ok...
        Project.set_status_meta(**callback_params) # this doesn't work...

From models.py:

The set_status_meta is a simple @staticmethod:

@staticmethod
    def set_status_meta(project_id, status=None):
        if project_id and status:
            project = Project.objects.get(pk=project_id)
            project.status_meta = status
            project.save()
            logger.info('saved new status!')

I've tried many different approaches, but for whatever reason, interfacing with the database from within a task seems tricky. Do you have any tips on executing a sort of "callback" to store details of what goes on in a task back in a certain database record (via inputed record.pk)?

All the best from Texas,
Givity

[lilspikey]

What errors are you seeing then you do this?

AFAIK interacting with the database shouldn't be any different in a task than outside of it.

Don't forget if an exception is raised when running the task then it will be stored in the "last_error" field of the task in the database.

You might want to add more logging to make sure you are actually getting the right values in the right places.

Generally speaking though passing in "simple" values like id's etc is the right way to do it.