pmap will limit the maximum number of simultaneous threads. So will the medusa library's medusa-pmap.
The difference is that if one job early in the list takes, e.g., 100 times longer than the next 20, and you have 4 cpus available, pmap will start the first (4+2)=6 in parallel threads, let jobs 2 through 6 complete, and then wait for the first one to finish before starting number 7. Thus most of the time will be spent running one thread. This has the advantage of limiting the memory required to store intermediate results, but the disadvantage of low usage of multiple cpu cores. Medusa has a medusa-pmap that will also limit the parallelism, but it lets you pick the level of parallelism (it isn't fixed at (# cpus + 2)), and it will continue starting new threads, even if that means using more memory when one job takes much longer than the following jobs. If you like pmap's behavior except for the number of threads, you can always copy its source code into your own program and change that number pretty easily. Andy On Thu, Sep 22, 2011 at 6:55 PM, Kevin Livingston < kevinlivingston.pub...@gmail.com> wrote: > if the list is quite large, is there anything preventing the option > below from creating way too many threads? > > there is presumably an inflection point where this will cost you. > pmap is trying to keep that in check, right? > > Kevin > > On Sep 21, 5:28 pm, Nathan Sorenson <n...@sfu.ca> wrote: > > Futures begin executing their contents immediately, you don't have to > > deref them to trigger the side effects. (perhaps you were thinking of > > delay?) > > > > I'm assuming you are using futures because email-request is an io- > > blocking operation? The thing to note is that the body of a future > > automatically runs in its own thread, so you could go ahead and just > > do a regular map: (doall (map #(future (email-request %)) approved)); > > all the emails would then be sent in parallel threads. > > > > On Sep 21, 4:06 pm, ronen <nark...@gmail.com> wrote: > > > > > > > > > > > > > > > > > I was looking for a pure Clojure solution to run multiple non- > > > dependant tasks on multiple threads, iv considered using Agent, > > > Promises or Futures, yet the simplest cleanest succinct solution iv > > > found is: > > > > > (defn email-approved [approved] > > > (doall (pmap deref (for [req approved] (future (email-request > > > req)))))) > > > > > The only thing that I don't like about it is the use of pmap, main > > > since Im using the parallel mapping action just for side effects > > > (triggering the consumptions of futures by derefing them). > > > > > Iv seen solution that involve Java service executor and the Work > > > project (https://github.com/getwoven/work), yet id rather achieve this > > > in pure Clojure. > > > > > Thanks > > > Ronen > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
