On 28 Oct 2011, at 15:21, Daryl Herzmann wrote: > Hi, > > I have an application that ingests data and does a reactor.spawnProcess() > for each chunk of data (a product) to pass it as STDIN to a compiled binary > and then I harvest the STDOUT. It has been working well, but I have an issue > when my data rates get too high and the machine gets overloaded with spawned > processes (I think) and starts running out of file descriptors (yes I can > raise it :) causing all sorts of pain. I'm wondering about a mechanism to > throttle the number of spawned processes going at one time? It'd be nice to > only have 10 of these spawned processes going at any one time. Thanks for > your ideas :)
Another alternative is to use turtl which was written for a similar purpose. http://pypi.python.org/pypi/turtl https://bitbucket.org/adroll/turtl/ https://launchpad.net/turtl http://twistedmatrix.com/pipermail/twisted-python/2011-August/024415.html -- Valentino Volonghi http://www.adroll.com _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
