On Fri, Jan 16, 2009 at 12:52 AM, James Mills <prolo...@shortcircuit.net.au> wrote: > I've noticed over the past few weeks lots of questions > asked about multi-processing (including myself). > > For those of you new to multi-processing, perhaps this > thread may help you. Some things I want to start off > with to point out are: > > "multiprocessing will not always help you get things done faster." > > "be aware of I/O bound applications vs. CPU bound" > > "multiple CPUs (cores) can compute multiple concurrent expressions - > not read 2 files concurrently" > > "in some cases, you may be after distributed processing rather than > multi or parallel processing" > > cheers > James
James is quite correct, and maybe I need to amend the multiprocessing documentation to reflect this fact. While distributed programming and parallel programming may cross paths in a lot of problems/applications, you have to know when to use one versus the other. Multiprocessing only provides some basic primitives to help you get started with distributed programming, it is not it's primary focus, nor is it a complete solution for distributed applications. That being said, there is no reason why you could not use it in conjunction with something like Kamaelia, pyro, $ipc mechanism/etc. Ultimately, it's a tool in your toolbox, and you have to judge and experiment to see which tool is best applied to your problem. In my own work/code, I use both processes *and* threads - one works better than the other depending on the problem. For example, a web testing tool. This is something that needs to generate hundreds of thousands of HTTP requests - not a problem you want to use multiprocessing for given that A> It's primarily I/O bound and B> You can generate that many threads on a single machine. However, if I wanted to say, generate hundreds of threads across multiple machines, I would (and do) use multiprocessing + paramiko to construct a grid of machines and coordinate work. That all being said: multiprocessing isn't set in stone - there's room for improvement in the docs, tests and code, and all patches are welcome. -jesse -- http://mail.python.org/mailman/listinfo/python-list
