On 06:48 pm, [email protected] wrote: >Ah no, I use conch to connect to the remote machines. My conch code >starts >a remote python instance and runs some bootstrapping python code which >connects to the PB Root. Once connected I send the test code to the >remote >process and start testing with PB. When the PB remote method >(remote_execute) is called from the users machine, that starts a new >thread >with deferToThread to run the test code. Since tests have multiple >parts >that need to run simultaneously, some parts are delivered to the same >machine, causing one machine to have to execute 5 or more parts >creating >many twisted threads. >The test code does its thing which stresses a network library. It >creates >servers and clients and makes sure everyone gets connected and gets all >messages involved with the test. It waits for all these events to >happen >simultaneously so the twisted threads can be alive for very long >periods of >time, and the test depends on the fact that all such threads are >running and >not waiting in the thread pool overflow queue. >I am planning on switching the multiple thread, one python design to >multiple python, one thread, which I hope will allow all elements of >the >test to run simultaneously easier than just simply increasing the >thread >pool size.
Does creating servers and clients involve running child processes? If so, you should be able to use reactor.spawnProcess instead of deferToThread and a blocking child process API. But if it involves running arbitrary blocking Python code, then I don't see how you can avoid threads. Switching to processes may help or it may hurt - it depends on the details of your workload (sometimes Python threads scale much worse than linearly making them worse than having an equal number of separate processes, but sometimes they're fine making them cheaper than processes; it all depends on what your code does). Jean-Paul _______________________________________________ Twisted-Python mailing list [email protected] http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
