Thanks for this info -- I didn't realize quite how pmap worked. I often launch parallel threads with pmap and have sometimes been puzzled by dips in processor utilization that I can't trace to memory resource contention, etc.
I have similar issues sometimes when I launch parallel threads via sends to agents. Will this behave similarly to pmap? If so, is there a straightforward way to get the same kind of benefit as medusa-pmap in an agent context? -Lee On Sep 22, 2011, at 11:34 PM, Andy Fingerhut wrote: > pmap will limit the maximum number of simultaneous threads. So will the > medusa library's medusa-pmap. > > The difference is that if one job early in the list takes, e.g., 100 times > longer than the next 20, and you have 4 cpus available, pmap will start the > first (4+2)=6 in parallel threads, let jobs 2 through 6 complete, and then > wait for the first one to finish before starting number 7. Thus most of the > time will be spent running one thread. This has the advantage of limiting > the memory required to store intermediate results, but the disadvantage of > low usage of multiple cpu cores. > > Medusa has a medusa-pmap that will also limit the parallelism, but it lets > you pick the level of parallelism (it isn't fixed at (# cpus + 2)), and it > will continue starting new threads, even if that means using more memory when > one job takes much longer than the following jobs. > > If you like pmap's behavior except for the number of threads, you can always > copy its source code into your own program and change that number pretty > easily. > > Andy -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
