Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Terry Jones
Hi Jason > I don't see anything in task.Cooperator to limit the # of > simultaneously-running tasks. Am I missing something? I'm not sure, but I think so. It does limit the number of simultaneously-running tasks. > I guess, technically, could write my own scheduler that limits the # of > simul

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
On Wed, Oct 26, 2011 at 11:08 AM, Jason Rennie wrote: > First off, have you read http://jcalderone.livejournal.com/24285.html ? >> If not, do. Take note of the line >> >> work = (callable(elem, *args, **named) for elem in iterable) >> >> work is a generator. Make sure you understand every line

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Phil Mayers
On 26/10/11 16:08, Jason Rennie wrote: > On Wed, Oct 26, 2011 at 10:22 AM, Terry Jones > wrote: > > Sounds like you should be using a Python generator and that you're not. > > > The issue I'm seeing is that the ~million DeferredSemaphore.run() calls > are inefficient; it's

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
On Wed, Oct 26, 2011 at 10:22 AM, Terry Jones wrote: > Sounds like you should be using a Python generator and that you're not. > The issue I'm seeing is that the ~million DeferredSemaphore.run() calls are inefficient; it's independent of list/generator. First off, have you read http://jcalderon

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Itamar Turner-Trauring
On Wed, 2011-10-26 at 10:02 -0400, Jason Rennie wrote: > The background: > > > I've been using DeferredSemaphore and DeferredList to manage the > running of tasks with a resource constraint (only so many tasks can > run at the same time). This worked great until I tried to use it to > manage mil

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread exarkun
On 02:02 pm, jren...@gmail.com wrote: >The background: > >I've been using DeferredSemaphore and DeferredList to manage the >running of >tasks with a resource constraint (only so many tasks can run at the >same >time). This worked great until I tried to use it to manage millions of >tasks. Simpl

Re: [Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Terry Jones
Hi Jason > - Is there a Twisted pattern for managing tasks efficiently that I be > - missing? Sounds like you should be using a Python generator and that you're not. First off, have you read http://jcalderone.livejournal.com/24285.html ? If not, do. Take note of the line work = (callable

[Twisted-Python] running 1,000,000 tasks, 40 at-a-time

2011-10-26 Thread Jason Rennie
The background: I've been using DeferredSemaphore and DeferredList to manage the running of tasks with a resource constraint (only so many tasks can run at the same time). This worked great until I tried to use it to manage millions of tasks. Simply setting them up to run (DeferredSemaphore.run(