Re: [Twisted-Python] Ideas on limiting/throttling spawnProcess

2011-10-31 Thread Daryl Herzmann
On Sat, Oct 29, 2011 at 2:10 AM, Nicolas Toper wrote: > For a similar use case, we're using a combination of cooperator with > defertoprocess. > > I can explain more if anyone's interested. I would certainly be interested :) I'm having a difficult time understanding how the pieces fit together.

[Twisted-Python] Ideas on limiting/throttling spawnProcess

2011-10-28 Thread Daryl Herzmann
Hi, I have an application that ingests data and does a reactor.spawnProcess() for each chunk of data (a product) to pass it as STDIN to a compiled binary and then I harvest the STDOUT. It has been working well, but I have an issue when my data rates get too high and the machine gets overloaded wit