Thanks for the response, niphlod! Let me explain: The task can be marked FAILED or EXPIRED if: • The code in the task throws an exception • A run of the task exceeds the timeout • The system clock goes past stop_time
And it will just not plain exist if: • You have just set up the code • You install the code on a new database In order to implement a perpetually processing task queue, we need to ensure that there is always an active "process queue" task ready in the scheduler. So what's the best way to do this without creating race conditions? This method creates a race condition: 1. Check if task exists 2. Insert if not ...if multiple processes are checking and inserting at the same time. The only solution I've found is to wrap that code within a postgresql database advisory lock, but this only works on postgres. (And the update_or_insert() function you mentioned does the same thing as this internally, so it still suffers from a race condition.) On Wednesday, June 13, 2012 7:16:56 AM UTC-7, Niphlod wrote: > > Maybe I didn't get exactly what you need , but ...... you have 3 tasks, > that needs to be unique. > Also, you want to be sure that if a task crashes doesn't remain "hanged". > > This should never happen with the scheduler .... the worst situation is > that if a worker crashes (here "crashes" is it disconnects from the > database) leaves the task status as running, but as soon as another > scheduler checks if that one sends heartbeats, he removes the dead worker > and requeue that task. > If your task goes into timeout and it's a repeating task the best practice > should be to raise the timeout. > > Assured this, you need to initialize the database if someone truncates the > scheduler_task table, inserting the 3 records in one transaction. > > If you need to be sure, why all the hassle when you can "prepare" the > task_name column as a unique value and then do > db.update_or_insert(task_name==myuniquetaskname, **task_record) ? > > PS: code in models get executed every request. What if you have no users > accessing the site and in the need to call initialize_task_queue ? Isn't it > better to insert the values and then start the workers ? > > BTW: a task that needs to be running "forever" but can't be "launched" in > two instances seems to suffer some design issues but hey, everyone needs to > be able to do what he wants ;-) >
