[Twisted-Python] Ideas on limiting/throttling spawnProcess

2011-10-28 Thread Daryl Herzmann
Hi, I have an application that ingests data and does a reactor.spawnProcess() for each chunk of data (a product) to pass it as STDIN to a compiled binary and then I harvest the STDOUT. It has been working well, but I have an issue when my data rates get too high and the machine gets overloaded wit

Re: [Twisted-Python] Ideas on limiting/throttling spawnProcess

2011-10-28 Thread Donal McMullan
I think twisted.internet.defer.DeferredSemaphore is designed for this. John Paul Calderone answers a similar question in detail: http://stackoverflow.com/questions/2861858/queue-remote-calls-to-a-python-twisted-perspective-broker Donal McMullan On 29/10/2011, at 11:21 AM, Daryl Herzmann wrote: