Laszlo Nagy <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: >> Has anybody tried to run parallel python applications? >> It appears that if your application is computation-bound using 'thread' >> or 'threading' modules will not get you any speedup. That is because >> python interpreter uses GIL(Global Interpreter Lock) for internal >> bookkeeping. The later allows only one python byte-code instruction to >> be executed at a time even if you have a multiprocessor computer. >> To overcome this limitation, I've created ppsmp module: >> http://www.parallelpython.com >> It provides an easy way to run parallel python applications on smp >> computers. >> I would appreciate any comments/suggestions regarding it. >> > I always thought that if you use multiple processes (e.g. os.fork) then > Python can take advantage of multiple processors. I think the GIL locks > one processor only. The problem is that one interpreted can be run on > one processor only. Am I not right? Is your ppm module runs the same > interpreter on multiple processors? That would be very interesting, and > something new. > The GIL locks all processors, but just for one process. So, yes, if you spawn off multiple processes then Python will take advantage of this. For example we run Zope on a couple of dual processor dual core systems, so we use squid and pound to ensure that the requests are spread across 4 instances of Zope on each machine. That way we do get a fairly even cpu usage.
For some applications it is much harder to split the tasks across separate processes rather than just separate threads, but there is a benefit once you've done it since you can then distribute the processing across cpus on separate machines. The 'parallel python' site seems very sparse on the details of how it is implemented but it looks like all it is doing is spawning some subprocesses and using some simple ipc to pass details of the calls and results. I can't tell from reading it what it is supposed to add over any of the other systems which do the same. Combined with the closed source 'no redistribution' license I can't really see anyone using it. -- http://mail.python.org/mailman/listinfo/python-list
