On Tuesday, July 2, 2013 1:37:20 PM UTC-5, Jim foo.bar wrote: > a purely stylistic comment... > > you may want to consider 'with-resources' to get rid of all the > try/finally clutter. I'd write your let statement like this: > > (let [step (fn [x] > (with-resources [sem (Semaphore. limit true)] #(.release > ^Semaphore %) > (.acquire sem) > (future (f x))))] > > Interesting, I hadn't considered with-resources. I don't think your code does the same thing, since the release happens in the calling thread, not inside the future. Right?
> a pretty basic map-reduce style function might the do the trick as well > depending on your case. You still use pmap but in a much more disciplined > fashion due to partitioning. > > (defn mapr > "A pretty basic map-reduce style mapping function. Will partition the data > according to p-size and assign a future to each partition (per pmap)." > ([f coll p-size shuffle?] > (->> (cond-> coll shuffle? shuffle) > (partition-all p-size) > (pmap #(mapv f %) ) > (apply concat)) ) ;;concat the inner vectors that represent the > partitions > ([f coll p-size] > (mapr f coll p-size false)) > ([f coll] > (mapr f coll (+ 2 cpu-no)))) > > hope that helps :) > This is a great technique when using pmap on fine-grained, CPU bound tasks with lots of items. The shuffle? is a nice touch too! In my case, processing each item can take a while, is not CPU intensive and their aren't too many items. That's why I want more than CPUs+2 threads (as in core/pmap). > > Jim > > > On 02/07/13 18:30, Brian Kirkbride wrote: > > Hi all, > > I'd like to get some feedback on an approach I took to limit the > concurrency of parallel processing in some of my Clojure codebase. > > I've often found that I want to split work over some number of threads > greater than CPUs + 2, but less than infinity. In the past I had used > promises that were fulfilled by a FixedThreadPool. But that's a lot of > setup and teardown that distracts from what you're actually trying to > achieve. > > My new approach was to leverage a Semaphore: > > (defn bounded-pmap [limit f coll] > (let [sem (Semaphore. limit true) > step (fn [x] > (.acquire sem) > (future > (try (f x) > (finally (.release sem)))))] > (->> (map step coll) > (doall) > (map deref)))) > > It's not lazy, but in my case the collection is already fully realized. > I'm interested in constructive criticism or other approaches. Thanks! > > Best, > Brian > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
