def parallel(iterable, count, callable, *args, **named):
#Copy/paste from http://jcalderone.livejournal.com/24285.html
#
http://oubiwann.blogspot.com/2008/06/async-batching-with-twisted-walkthrough.html
coop = task.Cooperator()
work = (callable(elem, *args, **named) for elem in iterable)
return defer.DeferredList([coop.coiterate(work) for i in xrange(count)])
def _localSend(messages):
return deferToProcessPool(reactor, _p, makeEmails, messages, logger)
_p = Pool(5)
def _localDeliver(messages):
#... Some part left out
d = parallel(messages,2, _localSend)
d.addErrback(_log_error)
return d
_localDeliver call parallel: it will launch at most 2 deferred executing
localSend(messages). These are actually deferToProcess
This is a very very powerful compbination and exarkun really nailed it in
his blog :)
On Mon, Oct 31, 2011 at 2:22 PM, Daryl Herzmann <akrh...@iastate.edu> wrote:
> On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper <nto...@gmail.com> wrote:
> > For a similar use case, we're using a combination of cooperator with
> > defertoprocess.
> >
> > I can explain more if anyone's interested.
>
> I would certainly be interested :) I'm having a difficult time
> understanding how the pieces fit together. Thank you!
>
> daryl
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
