just sent the patch to Massimo. If you're in a hurry, as soon as it is
committed just replace your gluon/scheduler.py with the one from trunk
Thanks for pointing out this misbehaviour of the scheduler.
On Saturday, October 20, 2012 8:25:12 PM UTC+2, Niphlod wrote:
>
> You're right, there's a "bug": for zillions of queued tasks at the same
> "priority" (i.e. got queued first) the "bunch" we assign on every loop
> doesn't take into account that there might be 10 or 20 tasks to assign and
> execute on a faster pace "in the following bunch(es)". Nice catch!
> reviewing the logic right now.
>
> BTW: mysched.queue_task is available only in trunk, it was just a teaser
> :P
>
> On Saturday, October 20, 2012 7:27:55 PM UTC+2, Adi wrote:
>>
>> i can confirm that size of the queued records has something to do with
>> delay to process different queues... once i deleted all outstanding records
>> from main group, fast_track group started working as expected... sorry for
>> a long thread, but i think it's a very neat idea to load scheduler with
>> lots of records which will be processed in a back-end process, while
>> fast-track works in it's own (faster) pace...
>>
>> On Sat, Oct 20, 2012 at 12:03 PM, Adnan Smajlovic
>> <adnan.s...@gmail.com>wrote:
>>
>>> couple things happened...
>>>
>>> The main group worker got created even though I didn't call it... Not
>>> sure why, but i guess because there are lot of leftover tasks queued (500k)
>>> and some were assigned when I stopped the process.
>>>
>>> Even though "fast_tack" worker started, nothing is getting picked,
>>> assigned nor completed... I checked the time, conditions, and it should be
>>> picked...
>>>
>>> Only main are getting assigned, but they are not getting completed, even
>>> though they were completing when I had only main group specified, without
>>> fast_track, and slow_track
>>>
>>> exec python26 /opt/web-apps/web2py/web2py.py -K
>>> crm:fast_track,crm:slow_track &> /var/log/web2py-scheduler.log
>>>
>>>
>>>
>>> On Sat, Oct 20, 2012 at 11:13 AM, Adnan Smajlovic
>>> <adnan.s...@gmail.com>wrote:
>>>
>>>> all clear :) in process of implementing.
>>>>
>>>> Is new api defined in scheduler.py, since i don't see it in there (2.1.1
>>>> (2012-10-17 17:00:46) dev), but I'm modifying the existing code to
>>>> employ fast_track, since order confirmations are getting behind. This will
>>>> be really good :) Thanks again, and again...
>>>>
>>>> On Sat, Oct 20, 2012 at 10:37 AM, Niphlod <nip...@gmail.com> wrote:
>>>>
>>>>> no prio available (it's hard to manage.... a task queued 3 hours ago
>>>>> with prio 7 comes before of after one with prio 8 queued 2 hours ago ?).
>>>>>
>>>>> "hackish way": tasks are picked up ordered by next_run_time. So, queue
>>>>> your tasks with next_runtime = request.now - datetime.timedelta(hours=1)
>>>>> kinda works.
>>>>>
>>>>> Right way: separate queues, "important tasks" and "less important
>>>>> tasks". You can create different queues assigning different group_name
>>>>> to
>>>>> tasks and start - at least 2 - separate scheduler processes. By default
>>>>> tasks are in the group 'main', and the scheduler worker processes those
>>>>> only
>>>>>
>>>>>
>>>>> Then, start one scheduler per queue with
>>>>> web2py.py -K appname:fast_track,appname:
>>>>>
>>>>> def task1(a, b=2):
>>>>> #need high prio
>>>>>
>>>>> def task2(a, b=2):
>>>>> #needs low prio
>>>>>
>>>>> from gluon.scheduler import Scheduler
>>>>> mysched = Scheduler(db)
>>>>>
>>>>> #new api
>>>>> mysched.queue_task(task1, ['a'], {'b': 1}, group_name='fast_track')
>>>>> mysched.queue_task(task2, ['a'], {'b' : 1}, group_name='slow_track')
>>>>>
>>>>> #old api
>>>>> from gluon.serializers import json
>>>>> db.scheduler_task.validate_and_insert(function_name='task1', args=json
>>>>> (['a']), vars=json({'b':1}), group_name='fast_track')
>>>>> db.scheduler_task.validate_and_insert(function_name='task2', args=json
>>>>> (['a']), vars=json({'b':1}), group_name='slow_track')
>>>>>
>>>>> slow_track
>>>>>
>>>>> If you just need some important tasks without assignign "slow_track"
>>>>> to the zillions you have already, just forget about the
>>>>> group_name='slow_track' and start schedulers with this command line
>>>>> web2py.py -K appname,appname:fast_track
>>>>> Then assign to fast_track only the ones you want to exec first and,
>>>>> assuming that fast_track has less tasks in queue, they will be executed
>>>>> before the zillion ones in the main group.
>>>>>
>>>>> Clear ?
>>>>>
>>>>>
>>
--
