Wow, what an answer! Niphlod, thanks a lot for such a detailed info with
examples - now it is crystal clear for me. Very great help, really
appreciate it!!!
You answer make me clarify the future architecture for my app. Before I
thought to use amazon internal tools for task distribution now I think I
can use w2p scheduler at least for the first stage or maybe permanently.
I have several additional question if you allow me. Hope it helps to other
members of the w2p club.
The plan is to start amazon servers (with web2py preinstalled)
programmatically when I need it with the purpose to run w2p scheduler on
it.
Could you give me your point of your on the following questions that I
need to address in order to build such a service:
1)Can I set up and cancel workers under web2py programmatically which
equivalent
to' python web2py.py -K myapp:fast,myapp:fast,myapp:fast'?
2) What is the best way to monitor load of the server to make a decision to
start new worker or new server depends on the resources left?
Looking forward to your answer.
Thank you in advance.
On Saturday, March 1, 2014 3:30:32 PM UTC+3, Niphlod wrote:
>
>
>
> On Friday, February 28, 2014 10:04:59 PM UTC+1, Andrey K wrote:
>>
>> Thank Niphlod for your answer. it is already great if it is possible even
>> theoretically!
>>
>
> it does even in the real world :-P
>
>
>> Regarding implementation. Correct me please if I am wrong in following
>> understanding - am I right that I need to:
>> 1)copy the whole web2py and myapp with necessary libs to another server .
>> 2)set up db connection in the 2nd box's web2py myapp to remote DB (first
>> server) like:
>> db= DAL('postgres://postgres:a...@196.xx.xx.xx/test', lazy_tables=False,
>> migrate=True, fake_migrate=False)
>>
>
> I'd say that migrate must be False here.... the "original" one is doing
> already the migrations. Also, after the scheduler tables have been created,
> use
> from gluon.scheduler import Scheduler
> mysched = Scheduler(db, migrate=False)
>
>
>> 3) run web2py scheduler on the second box as:
>> web2py.py -K appname
>>
>
> exactly
>
>
>>
>> Is that all???It is sounds like magic!!!!
>> How the second server job scheduler get understanding of calling task?
>>
>
> exactly as the one in the "original server". All needed informations are
> right on the scheduler_* tables, nothing else is required.
>
>
>> If it works I wonder how to control what server (not worker) get next
>> task - like what should I put in the model to force job be assigned to the
>> the box I want?
>>
>
> The workers will coordinate themselves and pick up a new task whenever
> it's ready to be processed. The whole point of the scheduler is having a
> task processed ASAP, no matter who processes it: the first available worker
> will do the job.
> Usually given that a task is queued and is ready to be processed, you
> don't even want to know who processes it, as long as it is processed:
> that's how scheduler works with default settings.
>
> That being said, if you want to send a specific task to a worker that
> exists only in a specific server, you can do so with "group_name"(s).
> The concept is having different group of tasks to be processed from
> schedulers being able to do so (i.e. one has all the libs, other one has
> only a few, or first one is more "powerful", other one is less "powerful",
> etc)
>
> You can then specify that a worker has to process only one group of tasks
> and the other one only another group of tasks.
>
> As specified in the book, the syntax to launch schedulers with one or more
> group_name is
>
> python web2py.py -K myapp:group1:group2,myotherapp:group1
>
>
>
> in that case, you'd launch
>
> python web2py.py -K myapp:original
>
> on the original server and
>
> python web2py.py -K myapp:additional
>
> on the additional one.
>
> You can then queue tasks to be processed from the "original" box as
> mysched.queue_task(......, group_name='original')
> and tasks to be processed from the "additional" one as
> mysched.queue_task(......, group_name='additional')
>
>
> PS: all combinations of scheduler and group_names are allowed....let's say
> the "additional" box will need to "help" the "original" one too , then
> you'll have
> - 1 worker from the original box processing "original"
> - 1 worker from the additional box processing "additional" AND "original"
> then if you launch
>
> python web2py.py -K myapp:original
> on the first box and
> python web2py.py -K myapp:original:additional
> on the additional
>
> the workers will coordinate to let BOTH workers process tasks with
> "original" group_name and only the additional box to process tasks with the
> "additional" group.
>
> Another example (more complicated): you have a set of tasks that needs
> "precedence" (i.e. near real-time execution) and a set of tasks that can be
> executed with more "relax", and 3 boxes....you can do some serious math :-P
>
> box1) python web2py.py -K myapp:fast,myapp:fast,myapp:fast
> box2) python web2py.py -K myapp:relax
> box3) python web2py.py -K myapp:fast,myapp:fast,myapp:fast:relax
>
> now you have
> fast) 3 workers from box1, 2.5 workers on box1 --> 5.5 workers
> relax) 1 worker from box2, 0.5 workers on box3 --> 1.5 workers
>
> Now, for speech's sake, if all of your tasks take the same time to be
> executed, and 100 "fast" and 100 "relax" tasks have been queued, in the
> long run you'd observe that:
> - ~ 54 "fast" tasks executed by "box1" (100 / 5.5 * 3)
> - ~ 46 "fast" tasks executed by "box3" (100 / 5.5 * 2.5)
> - ~ 66 "relax" tasks executed by "box2" (100 / 1.5 * 1)
> - ~ 34 "relax" tasks executed by "box3" (100 / 1.5 * 0.5)
>
> Also, you'd observe that to have the last "relax" task to be executed at
> the same time of the last "fast" task, you **can** queue up to 366 "fast"
> tasks to match 100 "relax" tasks (because 5.5 workers are "assigned" to
> "fast" and "only" 1.5 are "assigned" to "relax" tasks)...
>
>
> Hope the whole concept is clear: feel free to ask for more details if
> needed.
>
>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
