On Jan 23, 2011, at 10:56 PM, Ken Wesson wrote: > I've managed to make this more efficient, by making each agent send > process a larger portion of the job than one single item: > > (defn eager-pmap [f & colls] > (let [cores (.. Runtime getRuntime availableProcessors) > agents (cycle (for [_ (range cores)] (agent nil))) > ccolls (map (partial partition 16 16 []) colls) > promises (apply map (fn [& _] (promise)) ccolls)] > (doall > (apply map (fn [a p & chunks] > (send a > (fn [_] > (deliver p > (apply map f chunks))))) > agents promises ccolls)) > (mapcat deref promises))) > > This is much faster than either of the other eager-pmaps I posted to > this thread, and yet it's only using 60% CPU on my dual-core box. That > means there's still another x1.6 or so speedup possible(!) but I'm not > sure how. > > Raising the size of the partitions from 16 all the way to 1024 makes > no noticeable difference.
Thanks for the work, Ken. Clojure's multi-threaded performance can be mysterious indeed, which is a shame when one of the major benefits of functional code is that it's easily parallelizable. Anyway, for now I will probably just stick with pmap + shuffle, since it's dead-simple and should be enough my purposes. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
