For a similar use case, we're using a combination of cooperator with defertoprocess.
I can explain more if anyone's interested. Le 29 oct. 2011 00:41, "Donal McMullan" <donal.mcmul...@gmail.com> a écrit : > I think twisted.internet.defer.DeferredSemaphore is designed for this. John > Paul Calderone answers a similar question in detail: > > http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker > > Donal McMullan > > > On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote: > > > Hi, > > > > I have an application that ingests data and does a reactor.spawnProcess() > > for each chunk of data (a product) to pass it as STDIN to a compiled > binary > > and then I harvest the STDOUT. It has been working well, but I have an > issue > > when my data rates get too high and the machine gets overloaded with > spawned > > processes (I think) and starts running out of file descriptors (yes I can > > raise it :) causing all sorts of pain. I'm wondering about a mechanism > to > > throttle the number of spawned processes going at one time? It'd be nice > to > > only have 10 of these spawned processes going at any one time. Thanks > for > > your ideas :) > > > > daryl > > > > code snipet: > > > > class SHEFIT(protocol.ProcessProtocol): > > def __init__(self, tp): > > self.tp = tp > > self.data = "" > > > > def connectionMade(self): > > self.transport.write( self.tp.raw ) > > self.transport.closeStdin() > > > > def outReceived(self, data): > > self.data = self.data + data > > > > def errReceived(self, data): > > print "errReceived! with %d bytes!" % len(data) > > print data > > > > def outConnectionLost(self): > > really_process(self.tp, self.data) > > > > def got_product(): > > shef = SHEFIT( tp ) > > reactor.spawnProcess(shef, "shefit", ["shefit"], {}) > > > > def really_process(tp,data): > > print 'Do some work' > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
