On Jan 25, 2011, at 12:13 PM, Andy Fingerhut wrote: > In my original message describing pmap's behavior, there was a little > "gotcha" near the end: > > "Note: Sometimes working at odds with pmap's "Don't work too far ahead" > approach is if the input sequence to pmap is chunked. When a chunk is > reached, all elements in that chunk have threads start in parallel, so the > number of parallel threads can easily exceed (availableProcessors+2) during > those times." > > > (range n) produces heavily chunked sequences. Chunks are an optimization on > time and memory when representing large sequences, but they do cause pmap as > written today to suddenly fire off all futures in the chunk as parallel > threads when it reaches the chunk.
Interesting! Sorry I missed this gotcha from your original email; I hadn't run across this problem at the time and thus the info didn't stick in my head. Thanks for the link to modified-pmap, too. This seems like undesirable behavior from pmap to me (and/or unwanted chunking-induced behavior[1]). At the least I would like to see pmap adopt an optional extra argument that works like your modified-pmap's num-threads argument, and possibly also the arrival of Clojure's promised de-chunking interface. [1] http://disclojure.org/documents/clojure-1-1-0-rc1-release-notes/ (section 2.3: "Please share any use cases where chunked processing has resulted in behavioral differences that matter to you on the Clojure google group.") -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
