You're right, there's a "bug": for zillions of queued tasks at the same
"priority" (i.e. got queued first) the "bunch" we assign on every loop
doesn't take into account that there might be 10 or 20 tasks to assign and
execute on a faster pace "in the following bunch(es)". Nice catch!
reviewing the logic right now.
BTW: mysched.queue_task is available only in trunk, it was just a teaser
:P
On Saturday, October 20, 2012 7:27:55 PM UTC+2, Adi wrote:
>
> i can confirm that size of the queued records has something to do with
> delay to process different queues... once i deleted all outstanding records
> from main group, fast_track group started working as expected... sorry for
> a long thread, but i think it's a very neat idea to load scheduler with
> lots of records which will be processed in a back-end process, while
> fast-track works in it's own (faster) pace...
>
> On Sat, Oct 20, 2012 at 12:03 PM, Adnan Smajlovic
> <adnan.s...@gmail.com<javascript:>
> > wrote:
>
>> couple things happened...
>>
>> The main group worker got created even though I didn't call it... Not
>> sure why, but i guess because there are lot of leftover tasks queued (500k)
>> and some were assigned when I stopped the process.
>>
>> Even though "fast_tack" worker started, nothing is getting picked,
>> assigned nor completed... I checked the time, conditions, and it should be
>> picked...
>>
>> Only main are getting assigned, but they are not getting completed, even
>> though they were completing when I had only main group specified, without
>> fast_track, and slow_track
>>
>> exec python26 /opt/web-apps/web2py/web2py.py -K
>> crm:fast_track,crm:slow_track &> /var/log/web2py-scheduler.log
>>
>>
>>
>> On Sat, Oct 20, 2012 at 11:13 AM, Adnan Smajlovic
>> <adnan.s...@gmail.com<javascript:>
>> > wrote:
>>
>>> all clear :) in process of implementing.
>>>
>>> Is new api defined in scheduler.py, since i don't see it in there (2.1.1
>>> (2012-10-17 17:00:46) dev), but I'm modifying the existing code to
>>> employ fast_track, since order confirmations are getting behind. This will
>>> be really good :) Thanks again, and again...
>>>
>>> On Sat, Oct 20, 2012 at 10:37 AM, Niphlod <nip...@gmail.com<javascript:>
>>> > wrote:
>>>
>>>> no prio available (it's hard to manage.... a task queued 3 hours ago
>>>> with prio 7 comes before of after one with prio 8 queued 2 hours ago ?).
>>>>
>>>> "hackish way": tasks are picked up ordered by next_run_time. So, queue
>>>> your tasks with next_runtime = request.now - datetime.timedelta(hours=1)
>>>> kinda works.
>>>>
>>>> Right way: separate queues, "important tasks" and "less important
>>>> tasks". You can create different queues assigning different group_name to
>>>> tasks and start - at least 2 - separate scheduler processes. By default
>>>> tasks are in the group 'main', and the scheduler worker processes those
>>>> only
>>>>
>>>>
>>>> Then, start one scheduler per queue with
>>>> web2py.py -K appname:fast_track,appname:
>>>>
>>>> def task1(a, b=2):
>>>> #need high prio
>>>>
>>>> def task2(a, b=2):
>>>> #needs low prio
>>>>
>>>> from gluon.scheduler import Scheduler
>>>> mysched = Scheduler(db)
>>>>
>>>> #new api
>>>> mysched.queue_task(task1, ['a'], {'b': 1}, group_name='fast_track')
>>>> mysched.queue_task(task2, ['a'], {'b' : 1}, group_name='slow_track')
>>>>
>>>> #old api
>>>> from gluon.serializers import json
>>>> db.scheduler_task.validate_and_insert(function_name='task1', args=json
>>>> (['a']), vars=json({'b':1}), group_name='fast_track')
>>>> db.scheduler_task.validate_and_insert(function_name='task2', args=json
>>>> (['a']), vars=json({'b':1}), group_name='slow_track')
>>>>
>>>> slow_track
>>>>
>>>> If you just need some important tasks without assignign "slow_track" to
>>>> the zillions you have already, just forget about the
>>>> group_name='slow_track' and start schedulers with this command line
>>>> web2py.py -K appname,appname:fast_track
>>>> Then assign to fast_track only the ones you want to exec first and,
>>>> assuming that fast_track has less tasks in queue, they will be executed
>>>> before the zillion ones in the main group.
>>>>
>>>> Clear ?
>>>>
>>>>
>
--
