In my original message describing pmap's behavior, there was a little
"gotcha" near the end:
"Note: Sometimes working at odds with pmap's "Don't work too far
ahead" approach is if the input sequence to pmap is chunked. When a
chunk is reached, all elements in that chunk have threads start in
parallel, so the number of parallel threads can easily exceed
(availableProcessors+2) during those times."
(range n) produces heavily chunked sequences. Chunks are an
optimization on time and memory when representing large sequences, but
they do cause pmap as written today to suddenly fire off all futures
in the chunk as parallel threads when it reaches the chunk.
A modified version of pmap called (boringly) "modified-pmap" that does
not behave this way when reaching a chunk in its input sequence is
part of one of my submissions to the shootout web site, because I did
not want the extra parallelism:
http://shootout.alioth.debian.org/u32/program.php?test=knucleotide&lang=clojure&id=2
It is nearly identical to the original pmap. Comparing to the source
of the original is the easiest way to see the changes, if you are
curious.
Andy
On Jan 25, 2011, at 6:45 AM, Michael Gardner wrote:
I have run across something else I don't understand about pmap. Why
does the following:
(pmap (fn [_] (clojure.java.shell/sh "sleep" "10")) (range 32))
result in all 32 "sleep" processes being run at once? I thought pmap
used n+2 threads, where n is the number of processors/cores
available (I have two cores).
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient
with your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
