Thanks for the reply. I am using twisted conch to connect to the remote machines and start these tests, I am thinking that instead of using one connection for all testing I will open a new connection (and thus a new python) for each test. Like I said increasing the thread pool size worked well for fast machines but no so well on older ones. I am hoping that this behavior has more to do with the GIL and that 5 processes on a slow machine will operate better than 5 python threads. If not, then I will just have to write some sort of system to make sure I dont open too many threads on certain remote machines.
Thanks again On Fri, Mar 18, 2011 at 8:40 PM, <[email protected]> wrote: > On 10 Mar, 11:08 pm, [email protected] wrote: > >I am using PB to run remote methods in a testing system at my company. > >The > >code works very well but breaks down when I start running multiple > >tests at > >once. I have tracked this down to overflowing the thread pool on the > >remote > >machines. I am wondering if anyone might have better suggestions for > >running long methods from a remote method. > > > >I coded up a sample of what I am seeing here: > >http://pastebin.com/rBPp20Ms > > > >Basically I have 1 server that calls remote_execute on many clients on > >a > >remote server. This remote_execute method starts a new method using > >threads.deferToThread and returns the defer to make the server's > >callRemote > >defer wait until the remote long method end. > >What I do in those methods is run test code that waits, blocks, sleeps, > >and > >all sorts of nasty things that make the thread take a while. In the > >example > >code I simply sleep for 20 seconds. > > > >The problem I see with this code specifically is that I run out of > >threads > >on the pool and even though I wanted all execute methods to run at the > >same > >time, I see 10 run, then 10 more, then 10 more.. etc. The testing > >depends > >on all these methods being run at the same time as they run mechanisms > >that > >depend on each other and need everyone running. When I overflow the > >thread > >pool some methods do not run until other methods stop, which makes the > >whole > >test fail. > > This is how the thread pool works. It has a maximum size, which limits > the number of threads it will create to process work given to it. > Beyond that number of concurrent tasks, things will begin to get queued > up and wait for a free thread to execute them. > > Each task you give to the thread pool exclusively uses one of its > threads for the entire duration of the task, regardless of what the task > consists of. > >I am not holding the GIL or blocking the reactor, which was the first > >thing > >I checked. > > > >Setting reactor.suggestThreadPoolSize(50) does help, but I do not think > >its > >the best solution, and does not work very well on our slow and older > >machines. > > Using more threads is the only solution to the problem of not using > enough threads. Alternatively, look for wards to process tasks without > using threads. > > Jean-Paul > > _______________________________________________ > Twisted-Python mailing list > [email protected] > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >
_______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
