Re: multiprocessing vs thread performance

En Wed, 07 Jan 2009 23:05:53 -0200, James Mills escribió: Does anybody know any tutorial for python 2.6 multiprocessing? Or bunch of good example for it? I am trying to break a loop to run it over multiple core in a system. And I need to return an integer value as the result of the proce

Re: multiprocessing vs thread performance

On Thu, Jan 8, 2009 at 10:55 AM, Arash Arfaee wrote: > Hi All , HI :) > Does anybody know any tutorial for python 2.6 multiprocessing? Or bunch of > good example for it? I am trying to break a loop to run it over multiple > core in a system. And I need to return an integer value as the result of

Re: multiprocessing vs thread performance

Hi All , Does anybody know any tutorial for python 2.6 multiprocessing? Or bunch of good example for it? I am trying to break a loop to run it over multiple core in a system. And I need to return an integer value as the result of the process an accumulate all of them. the examples that I found the

Re: multiprocessing vs thread performance

En Sat, 03 Jan 2009 11:31:12 -0200, Nick Craig-Wood escribió: mk wrote: The results I got are very different from the benchmark quoted in PEP 371. On twin Xeon machine the threaded version executed in 5.54 secs, while multiprocessing version took over 222 secs to complete! Am I doing s

Re: multiprocessing vs thread performance

There doesn't seem to be any good examples on POSH or it's not clear to me. For when using with a for loop like mk is doing who started this thread. How would somethings like this be possible to do with POSH? The example show how to share variables between processes/threads but nothing about How th

Re: multiprocessing vs thread performance

mk wrote: > After reading http://www.python.org/dev/peps/pep-0371/ I was under > impression that performance of multiprocessing package is similar to > that of thread / threading. However, to familiarize myself with both > packages I wrote my own test of spawning and returning 100,000 empty

Re: multiprocessing vs thread performance

On Dec 31, 6:19 pm, Paul Rubin wrote: > Aaron Brady writes: > > I had an idea.  You could use 'multiprocessing' for synchronization, > > and just use an mmap for the actual communication.  (I think it's a > > good idea.) > > Are you reinventing POSH?  (http://poshmod

Re: multiprocessing vs thread performance

On Dec 31, 2008, at 7:19 PM, Paul Rubin wrote: Aaron Brady writes: I had an idea. You could use 'multiprocessing' for synchronization, and just use an mmap for the actual communication. (I think it's a good idea.) Are you reinventing POSH? (http://poshmodule.sourceforge.net) Or sysv_ip

Re: multiprocessing vs thread performance

Aaron Brady writes: > I had an idea. You could use 'multiprocessing' for synchronization, > and just use an mmap for the actual communication. (I think it's a > good idea.) Are you reinventing POSH? (http://poshmodule.sourceforge.net) -- http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing vs thread performance

On Dec 29, 9:29 am, mk wrote: > Christian Heimes wrote: > > mk wrote: > >> Am I doing smth wrong in code below? Or do I have to use > >> multiprocessing.Pool to get any decent results? > > > You have missed an important point. A well designed application does > > neither create so many threads nor

Re: multiprocessing vs thread performance

Christian Heimes wrote: > >You have missed an important point. A well designed application does >neither create so many threads nor processes. The creation of a thread >or forking of a process is an expensive operation. That actually depends on the operating system. As one example, thread creati

Re: multiprocessing vs thread performance

On Wed, Dec 31, 2008 at 8:42 AM, James Mills wrote: (snip) > As I continue to develop circuits and improve it's > core design as well as building it's ever growing set > of Components, I try to keep it as general as > possible - my main aim though is distributed > processing and architectures. (Se

Re: multiprocessing vs thread performance

On Wed, Dec 31, 2008 at 12:29 AM, Aaron Brady wrote: > James, Hi. I'm glad you asked; I never know how "out there" my > comments are (but surmise that feedback is always a good thing). What > I was thinking was, I didn't know Virtual Synchrony, and I've never > used Erlang, but I'm interested in

Re: multiprocessing vs thread performance

On Dec 29, 9:08 pm, "James Mills" wrote: > On Tue, Dec 30, 2008 at 12:52 PM, Aaron Brady wrote: > > On Dec 29, 7:40 pm, "James Mills" > > wrote: > >> On Tue, Dec 30, 2008 at 11:34 AM, Aaron Brady wrote: > >> > The OP may be interested in Erlang, which Wikipedia (end-all, be-all) > >> > claims i

Re: multiprocessing vs thread performance

On Tue, Dec 30, 2008 at 12:52 PM, Aaron Brady wrote: > On Dec 29, 7:40 pm, "James Mills" > wrote: >> On Tue, Dec 30, 2008 at 11:34 AM, Aaron Brady wrote: >> > The OP may be interested in Erlang, which Wikipedia (end-all, be-all) >> > claims is a 'distribution oriented language'. > snip >> I'm pr

Re: multiprocessing vs thread performance

On Dec 29, 7:40 pm, "James Mills" wrote: > On Tue, Dec 30, 2008 at 11:34 AM, Aaron Brady wrote: > > The OP may be interested in Erlang, which Wikipedia (end-all, be-all) > > claims is a 'distribution oriented language'. snip > I'm presently looking at Virtual Synchrony and > other distributed pro

Re: multiprocessing vs thread performance

On Tue, Dec 30, 2008 at 11:34 AM, Aaron Brady wrote: > The OP may be interested in Erlang, which Wikipedia (end-all, be-all) > claims is a 'distribution oriented language'. I would suggest to the OP that he take a look at circuits (1) an event framework with a focus on component architectures and

Re: multiprocessing vs thread performance

On Dec 29, 6:05 pm, "James Mills" wrote: > On Tue, Dec 30, 2008 at 12:52 AM, mk wrote: > > Hello everyone, > > > After readinghttp://www.python.org/dev/peps/pep-0371/I was under > > impression that performance of multiprocessing package is similar to that of > > thread / threading. However, to fa

Re: multiprocessing vs thread performance

On Tue, Dec 30, 2008 at 12:52 AM, mk wrote: > Hello everyone, > > After reading http://www.python.org/dev/peps/pep-0371/ I was under > impression that performance of multiprocessing package is similar to that of > thread / threading. However, to familiarize myself with both packages I > wrote my o

Re: multiprocessing vs thread performance

Roy Smith writes: > In article , > Christian Heimes wrote: > >> You have missed an important point. A well designed application does >> neither create so many threads nor processes. The creation of a thread >> or forking of a process is an expensive operation. You should use a pool >> of thread

Re: multiprocessing vs thread performance

Jarkko Torppa wrote: On the PEP371 it says "All benchmarks were run using the following: Python 2.5.2 compiled on Gentoo Linux (kernel 2.6.18.6)" Right... I overlooked that. My tests I quoted above were done on SLES 10, kernel 2.6.5. With python2.5 and pyProcessing-0.52 iTaulu:src torppa$

Re: multiprocessing vs thread performance

On 2008-12-29, mk wrote: > janislaw wrote: > >> Ah, so there are 100 processes at time. 200secs still don't sound >> strange. > > I ran the PEP 371 code on my system (Linux) on Python 2.6.1: > > Linux SLES (9.156.44.174) [15:18] root ~/tmp/src # ./run_benchmarks.py > empty_func.py > > Importing e

Re: multiprocessing vs thread performance

On Dec 29, 8:52 am, mk wrote: > Hello everyone, > > After readinghttp://www.python.org/dev/peps/pep-0371/I was under > impression that performance of multiprocessing package is similar to > that of thread / threading. However, to familiarize myself with both > packages I wrote my own test of spawn

Re: multiprocessing vs thread performance

In article , Christian Heimes wrote: > You have missed an important point. A well designed application does > neither create so many threads nor processes. The creation of a thread > or forking of a process is an expensive operation. You should use a pool > of threads or processes. It's worth n

Re: multiprocessing vs thread performance

Christian Heimes wrote: mk wrote: Am I doing smth wrong in code below? Or do I have to use multiprocessing.Pool to get any decent results? You have missed an important point. A well designed application does neither create so many threads nor processes. Except I was not developing "well des

Re: multiprocessing vs thread performance

mk wrote: > Am I doing smth wrong in code below? Or do I have to use > multiprocessing.Pool to get any decent results? You have missed an important point. A well designed application does neither create so many threads nor processes. The creation of a thread or forking of a process is an expensive

Re: multiprocessing vs thread performance

janislaw wrote: Ah, so there are 100 processes at time. 200secs still don't sound strange. I ran the PEP 371 code on my system (Linux) on Python 2.6.1: Linux SLES (9.156.44.174) [15:18] root ~/tmp/src # ./run_benchmarks.py empty_func.py Importing empty_func Starting tests ... non_threaded

Re: multiprocessing vs thread performance

On 29 Gru, 15:52, mk wrote: > Hello everyone, > > After readinghttp://www.python.org/dev/peps/pep-0371/I was under > impression that performance of multiprocessing package is similar to > that of thread / threading. However, to familiarize myself with both > packages I wrote my own test of spawnin

multiprocessing vs thread performance

Hello everyone, After reading http://www.python.org/dev/peps/pep-0371/ I was under impression that performance of multiprocessing package is similar to that of thread / threading. However, to familiarize myself with both packages I wrote my own test of spawning and returning 100,000 empty thr