Hi Yarin, Thank you for testing it! A QUEUED task is not picked up by a worker, it is first ASSIGNED to a worker that can pick up only the ones ASSIGNED to him. The "assignment" phase is important because: - the group_name parameter is honored (task queued with the group_name 'foo' gets assigned only to workers that process 'foo' tasks (the group_names column in scheduler_workers)) - DISABLED, KILL and TERMINATE workers are "removed" from the assignment alltogether - in multiple workers situations the QUEUED tasks are split amongst workers evenly, and workers "know in advance" what tasks they are allowed to execute (the assignment allows the scheduler to set up n "independant" queues for the n ACTIVE workers)
On Sunday, August 5, 2012 4:54:22 PM UTC+2, Yarin wrote: > > @Niphlod- First of all, thanks for taking this on. An effective scheduler > is critically important to us, and I'll be glad to help out in any way. > > I've downloaded the test app and am making corrections to the > documentation (per your request) for clarity, grammar, etc. > > On thing I'm stuck on is when the ASSIGNED status comes into play. > According to the docs: > >> "Tasks with no stop_time set or picked up BEFORE stop_time are ASSIGNED >> to a worker. When a workers picks up them, they become RUNNING." > > - This doesn't make sense to me. If a QUEUED task is picked up by a > worker, its status changes to RUNNING. So at what point is it ASSIGNED? > > > On Thursday, July 12, 2012 4:36:38 PM UTC-4, Niphlod wrote: >> >> Hello everybody, in the last month several changes were commited to the >> scheduler, in order to improve it. >> Table schemas were changed, to add some features that were missed by some >> users. >> On the verge of releasing web2py v.2.0.0, and seeing that the scheduler >> potential is often missed by regular web2py users, I created a test app >> with two main objectives: documenting the new scheduler and test the >> features. >> >> App is available on github ( >> https://github.com/niphlod/w2p_scheduler_tests). All you need is >> download the trunk version of web2py, download the app and play with it. >> >> Current features: >> - one-time-only tasks >> - recurring tasks >> - possibility to schedule functions at a given time >> - possibility to schedule recurring tasks with a stop_time >> - can operate distributed among machines, given a database reachable for >> all workers >> - group_names to "divide" tasks among different workers >> - group_names can also influence the "percentage" of assigned tasks to >> similar workers >> - simple integration using modules for "embedded" tasks (i.e. you can use >> functions defined in modules directly in your app or have them processed in >> background) >> - configurable heartbeat to reduce latency: with sane defaults and not >> toooo many tasks queued normally a queued task doesn't exceed 5 seconds >> execution times >> - option to start it, process all available tasks and then die >> automatically >> - integrated tracebacks >> - monitorable as state is saved on the db >> - integrated app environment if started as web2py.py -K >> - stop processes immediately (set them to "KILL") >> - stop processes gracefully (set them to "TERMINATE") >> - disable processes (set them to "DISABLED") >> - functions that doesn't return results do not generate a scheduler_run >> entry >> - added a discard_results parameter that doesn't store results "no matter >> what" >> - added a uuid record to tasks to simplify checkings of "unique" tasks >> - task_name is not required anymore >> - you can skip passing the function to the scheduler istantiation: >> functions can be dinamically retrieved in the app's environment >> >> So, your mission is: >> - test the scheduler with the app and familiarize with it >> Secondary mission is: >> - report any bug you find here or on github ( >> https://github.com/niphlod/w2p_scheduler_tests/issues) >> - propose new examples to be embedded in the app, or correct the current >> docs (English is not my mother tongue) >> >> Once approved, docs will be probably embedded in the book ( >> http://web2py.com/book) >> >> Feel free to propose features you'd like to see in the scheduler, I have >> some time to spend implementing it. >> >> >> >> --
