Let me go further:
Field('repeats_failed', 'integer', default=1, comment="0=unlimited"),
Should really be:
Field('retry_failed', 'integer', default=0, comment="-1=unlimited"),
According to the docs, this param is supposed to "set how many times the
function can raise an exception ... and be queued again instead of stopping
in FAILED status with the parameter." If that's the case, then 0 should
mean that we don't want the function to be queued again if it fails. 1
should mean give it one more try. This is a lot clearer than having the
number refer to the number of failures allowed.
On Sunday, August 5, 2012 11:55:19 AM UTC-4, Yarin wrote:
>
> Ok this is clearer to me- I'll see if I can clarify it in the docs..
>
> On to the next issue, this one regarding implementation:
>
> I think the following parameters need to be renamed:
>
> 1. 'repeats' should be 'repeat'
> 2. 'repeats_failed' should be 'retry_failed'
>
> Let me explain:
>
> 1. 'repeat' is a command, whereas 'repeats' sounds like a result.
> Because the task record stores both arguments and results, this becomes
> confusing.
> 2. 'repeats_failed' is even worse, because it sounds like a result
> (like 'times_failed', which *is* a result), and because it is a
> misnomer. We are not instructing it to repeat failures, but to
> *retry*them. Moreover, it needs to be clear that this value is completely
> distinct
> from the 'repeat' value- i.e. retries will be applied to every execution
> attempt, regardless of whether those attempts will be repeated or not.
>
>
>
>
>
>
>
> On Sunday, August 5, 2012 11:13:38 AM UTC-4, Niphlod wrote:
>>
>> Hi Yarin, Thank you for testing it!
>> A QUEUED task is not picked up by a worker, it is first ASSIGNED to a
>> worker that can pick up only the ones ASSIGNED to him. The "assignment"
>> phase is important because:
>> - the group_name parameter is honored (task queued with the group_name
>> 'foo' gets assigned only to workers that process 'foo' tasks (the
>> group_names column in scheduler_workers))
>> - DISABLED, KILL and TERMINATE workers are "removed" from the assignment
>> alltogether
>> - in multiple workers situations the QUEUED tasks are split amongst
>> workers evenly, and workers "know in advance" what tasks they are allowed
>> to execute (the assignment allows the scheduler to set up n "independant"
>> queues for the n ACTIVE workers)
>>
>>
>> On Sunday, August 5, 2012 4:54:22 PM UTC+2, Yarin wrote:
>>>
>>> @Niphlod- First of all, thanks for taking this on. An effective
>>> scheduler is critically important to us, and I'll be glad to help out in
>>> any way.
>>>
>>> I've downloaded the test app and am making corrections to the
>>> documentation (per your request) for clarity, grammar, etc.
>>>
>>> On thing I'm stuck on is when the ASSIGNED status comes into play.
>>> According to the docs:
>>>
>>>> "Tasks with no stop_time set or picked up BEFORE stop_time are ASSIGNED
>>>> to a worker. When a workers picks up them, they become RUNNING."
>>>
>>> - This doesn't make sense to me. If a QUEUED task is picked up by a
>>> worker, its status changes to RUNNING. So at what point is it ASSIGNED?
>>>
>>>
>>> On Thursday, July 12, 2012 4:36:38 PM UTC-4, Niphlod wrote:
>>>>
>>>> Hello everybody, in the last month several changes were commited to the
>>>> scheduler, in order to improve it.
>>>> Table schemas were changed, to add some features that were missed by
>>>> some users.
>>>> On the verge of releasing web2py v.2.0.0, and seeing that the scheduler
>>>> potential is often missed by regular web2py users, I created a test app
>>>> with two main objectives: documenting the new scheduler and test the
>>>> features.
>>>>
>>>> App is available on github (
>>>> https://github.com/niphlod/w2p_scheduler_tests). All you need is
>>>> download the trunk version of web2py, download the app and play with it.
>>>>
>>>> Current features:
>>>> - one-time-only tasks
>>>> - recurring tasks
>>>> - possibility to schedule functions at a given time
>>>> - possibility to schedule recurring tasks with a stop_time
>>>> - can operate distributed among machines, given a database reachable
>>>> for all workers
>>>> - group_names to "divide" tasks among different workers
>>>> - group_names can also influence the "percentage" of assigned tasks to
>>>> similar workers
>>>> - simple integration using modules for "embedded" tasks (i.e. you can
>>>> use functions defined in modules directly in your app or have them
>>>> processed in background)
>>>> - configurable heartbeat to reduce latency: with sane defaults and not
>>>> toooo many tasks queued normally a queued task doesn't exceed 5 seconds
>>>> execution times
>>>> - option to start it, process all available tasks and then die
>>>> automatically
>>>> - integrated tracebacks
>>>> - monitorable as state is saved on the db
>>>> - integrated app environment if started as web2py.py -K
>>>> - stop processes immediately (set them to "KILL")
>>>> - stop processes gracefully (set them to "TERMINATE")
>>>> - disable processes (set them to "DISABLED")
>>>> - functions that doesn't return results do not generate a scheduler_run
>>>> entry
>>>> - added a discard_results parameter that doesn't store results "no
>>>> matter what"
>>>> - added a uuid record to tasks to simplify checkings of "unique" tasks
>>>> - task_name is not required anymore
>>>> - you can skip passing the function to the scheduler istantiation:
>>>> functions can be dinamically retrieved in the app's environment
>>>>
>>>> So, your mission is:
>>>> - test the scheduler with the app and familiarize with it
>>>> Secondary mission is:
>>>> - report any bug you find here or on github (
>>>> https://github.com/niphlod/w2p_scheduler_tests/issues)
>>>> - propose new examples to be embedded in the app, or correct the
>>>> current docs (English is not my mother tongue)
>>>>
>>>> Once approved, docs will be probably embedded in the book (
>>>> http://web2py.com/book)
>>>>
>>>> Feel free to propose features you'd like to see in the scheduler, I
>>>> have some time to spend implementing it.
>>>>
>>>>
>>>>
>>>>
> On Sunday, August 5, 2012 11:13:38 AM UTC-4, Niphlod wrote:
>>
>> Hi Yarin, Thank you for testing it!
>> A QUEUED task is not picked up by a worker, it is first ASSIGNED to a
>> worker that can pick up only the ones ASSIGNED to him. The "assignment"
>> phase is important because:
>> - the group_name parameter is honored (task queued with the group_name
>> 'foo' gets assigned only to workers that process 'foo' tasks (the
>> group_names column in scheduler_workers))
>> - DISABLED, KILL and TERMINATE workers are "removed" from the assignment
>> alltogether
>> - in multiple workers situations the QUEUED tasks are split amongst
>> workers evenly, and workers "know in advance" what tasks they are allowed
>> to execute (the assignment allows the scheduler to set up n "independant"
>> queues for the n ACTIVE workers)
>>
>>
>> On Sunday, August 5, 2012 4:54:22 PM UTC+2, Yarin wrote:
>>>
>>> @Niphlod- First of all, thanks for taking this on. An effective
>>> scheduler is critically important to us, and I'll be glad to help out in
>>> any way.
>>>
>>> I've downloaded the test app and am making corrections to the
>>> documentation (per your request) for clarity, grammar, etc.
>>>
>>> On thing I'm stuck on is when the ASSIGNED status comes into play.
>>> According to the docs:
>>>
>>>> "Tasks with no stop_time set or picked up BEFORE stop_time are ASSIGNED
>>>> to a worker. When a workers picks up them, they become RUNNING."
>>>
>>> - This doesn't make sense to me. If a QUEUED task is picked up by a
>>> worker, its status changes to RUNNING. So at what point is it ASSIGNED?
>>>
>>>
>>> On Thursday, July 12, 2012 4:36:38 PM UTC-4, Niphlod wrote:
>>>>
>>>> Hello everybody, in the last month several changes were commited to the
>>>> scheduler, in order to improve it.
>>>> Table schemas were changed, to add some features that were missed by
>>>> some users.
>>>> On the verge of releasing web2py v.2.0.0, and seeing that the scheduler
>>>> potential is often missed by regular web2py users, I created a test app
>>>> with two main objectives: documenting the new scheduler and test the
>>>> features.
>>>>
>>>> App is available on github (
>>>> https://github.com/niphlod/w2p_scheduler_tests). All you need is
>>>> download the trunk version of web2py, download the app and play with it.
>>>>
>>>> Current features:
>>>> - one-time-only tasks
>>>> - recurring tasks
>>>> - possibility to schedule functions at a given time
>>>> - possibility to schedule recurring tasks with a stop_time
>>>> - can operate distributed among machines, given a database reachable
>>>> for all workers
>>>> - group_names to "divide" tasks among different workers
>>>> - group_names can also influence the "percentage" of assigned tasks to
>>>> similar workers
>>>> - simple integration using modules for "embedded" tasks (i.e. you can
>>>> use functions defined in modules directly in your app or have them
>>>> processed in background)
>>>> - configurable heartbeat to reduce latency: with sane defaults and not
>>>> toooo many tasks queued normally a queued task doesn't exceed 5 seconds
>>>> execution times
>>>> - option to start it, process all available tasks and then die
>>>> automatically
>>>> - integrated tracebacks
>>>> - monitorable as state is saved on the db
>>>> - integrated app environment if started as web2py.py -K
>>>> - stop processes immediately (set them to "KILL")
>>>> - stop processes gracefully (set them to "TERMINATE")
>>>> - disable processes (set them to "DISABLED")
>>>> - functions that doesn't return results do not generate a scheduler_run
>>>> entry
>>>> - added a discard_results parameter that doesn't store results "no
>>>> matter what"
>>>> - added a uuid record to tasks to simplify checkings of "unique" tasks
>>>> - task_name is not required anymore
>>>> - you can skip passing the function to the scheduler istantiation:
>>>> functions can be dinamically retrieved in the app's environment
>>>>
>>>> So, your mission is:
>>>> - test the scheduler with the app and familiarize with it
>>>> Secondary mission is:
>>>> - report any bug you find here or on github (
>>>> https://github.com/niphlod/w2p_scheduler_tests/issues)
>>>> - propose new examples to be embedded in the app, or correct the
>>>> current docs (English is not my mother tongue)
>>>>
>>>> Once approved, docs will be probably embedded in the book (
>>>> http://web2py.com/book)
>>>>
>>>> Feel free to propose features you'd like to see in the scheduler, I
>>>> have some time to spend implementing it.
>>>>
>>>>
>>>>
>>>>
--
