On 7 Sep, 00:06, cnb <[EMAIL PROTECTED]> wrote: > If I buy a multicore computer and I have really intensive program. How > would that be distributed across the cores?
It typically depends on how the work done by the program is performed. > Will algorithms always have to be programmed and told specifically to > run on several cores so if not told it will only utilize one core? Some algorithms lend themselves to parallelisation; others do not. Sometimes tools and runtimes can help by executing some instructions in parallel. > So is the free lunch really over or is this just an overhyped > phenomena? The free lunch ended a few minutes ago. ;-) > Is threading with Python hard? Can you start several processes with > Python or just threads? You can start both processes and threads with Python, although the effect of starting many threads - the actual concurrency - will depend on which implementation of Python you're using and where the bulk of the work is performed. If you're spending a lot of CPU time in processing data, and if that processing is taking place in Python code, then for the most effective threading you should consider an implementation like Jython or IronPython which supports free-threading. If most of the work happens in extension code (where you'd probably have little choice about using CPython, anyway), then it might be the case that the extension releases the global interpreter lock in CPython and you might then be able to benefit from having many threads doing work simultaneously, although I imagine that the extension would itself need to be thread- safe, too. If you're spending a lot of time moving data around, performing communication, and so on, then multiple threads may still be effective in CPython, since some of them might be getting a system call to read or write data, thus freeing the CPU for the others. These kinds of situations lend themselves to other approaches such as asynchronous processing of data, however. It doesn't sound like this is like your program, if by "intensive" you mean high levels of CPU activity. As you note, the alternative to threads is processes, and many people advocate multi-process, "shared nothing" solutions. Here's a list which covers multicore and SMP-related solutions as well as high-end clustering solutions: http://wiki.python.org/moin/ParallelProcessing Although the processing module is part of Python 2.6/3.0 as the multiprocessing module, you might want to at least look at the pp, pprocess and papyros solutions. My aim with pprocess was to target multicore UNIX-like systems with an unintrusive API; pp and papyros, on the other hand, seek to cover larger scale systems as well, and I think that the way papyros has been done has some merit, mostly because if you wanted to combine convenience with distributed processing, you'd want to choose distributed object technologies as the foundation (CORBA would have been good for this, too, at least for multi-language support, but its APIs can also seem quite intimidating). Paul -- http://mail.python.org/mailman/listinfo/python-list
