In article <[EMAIL PROTECTED]>, =?ISO-8859-2?Q?Przemys=B3aw_R=F3=BFycki?= <[EMAIL PROTECTED]> wrote: > >I have written some code, which creates many threads for each connection >('main connection'). The purpose of this code is to balance the load >between several connections ('pipes'). The number of spawned threads >depends on how many pipes I create (= 2*n+2, where n is the number of >pipes). > >For good results I'll presumably share main connection's load between 10 >pipes - therefore 22 threads will be spawned. Now if about 50 >connections are forwarded the number of threads rises to thousand of >threads (or several thousands if even more connections are established).
I'm a bit confused by your math. Fifty connections should be 102 threads, which is quite reasonable. >My questions are: >- What is the cost (in memory / CPU usage) of creating such amounts of >threads? >- Is there any 'upper boundary' that limits the number of threads? (is >it python / OS related) >- Is that the sign of 'clumsy programming' - i.e. Is creating so many >threads a bad habit? (I must say that it simplified the solution of my >problem very much). > >Limiting the number of threads is possible, but would affect the >independence of data flows. (ok I admit - creating tricky algorithm >could perhaps gurantee concurrency without spawning so many threads - >but it's the simplest solution to this problem :) ). My experience with lots of threads dates back to Python 1.5.2, but I rarely saw much improvement with more than a hundred threads, even for heavily I/O-bound applications on a multi-CPU system. However, if your focus is algorithmic complexity, you should be able to handle a couple of thousand threads easily enough. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "The joy of coding Python should be in seeing short, concise, readable classes that express a lot of action in a small amount of clear code -- not in reams of trivial code that bores the reader to death." --GvR -- http://mail.python.org/mailman/listinfo/python-list