a purely stylistic comment...
you may want to consider 'with-resources' to get rid of all the
try/finally clutter. I'd write your let statement like this:
(let [step (fn [x]
(with-resources [sem (Semaphore. limit true)]
#(.release ^Semaphore %)
(.acquire sem)
(future (f x))))]
a pretty basic map-reduce style function might the do the trick as well
depending on your case. You still use pmap but in a much more
disciplined fashion due to partitioning.
(defn mapr
"A pretty basic map-reduce style mapping function. Will partition the
data according to p-size and assign a future to each partition (per pmap)."
([f coll p-size shuffle?]
(->> (cond-> coll shuffle? shuffle)
(partition-all p-size)
(pmap #(mapv f %) )
(apply concat)) ) ;;concat the inner vectors that represent the partitions
([f coll p-size]
(mapr f coll p-size false))
([f coll]
(mapr f coll (+ 2 cpu-no))))
hope that helps :)
Jim
On 02/07/13 18:30, Brian Kirkbride wrote:
Hi all,
I'd like to get some feedback on an approach I took to limit the
concurrency of parallel processing in some of my Clojure codebase.
I've often found that I want to split work over some number of threads
greater than CPUs + 2, but less than infinity. In the past I had used
promises that were fulfilled by a FixedThreadPool. But that's a lot of
setup and teardown that distracts from what you're actually trying to
achieve.
My new approach was to leverage a Semaphore:
(defn bounded-pmap [limit f coll]
(let [sem (Semaphore. limit true)
step (fn [x]
(.acquire sem)
(future
(try (f x)
(finally (.release sem)))))]
(->> (map step coll)
(doall)
(map deref))))
It's not lazy, but in my case the collection is already fully
realized. I'm interested in constructive criticism or other
approaches. Thanks!
Best,
Brian
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient
with your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
