> Sounds like a job for a future. If he knows the workload, and he knows how many threads will be spun up, and he knows that the number will be reasonably small, then (future) is a good bet. But if there is any risk of large numbers of threads being spun up, then he should avoid calling (future) and he should instead use a library that takes on the work of managing a thread pool for him. He can call (future) if he knows there are only going to be 8 threads, or a 100, but if there might be 10,000 threads spun up, then he should probably not call (future) directly.
On Wednesday, September 17, 2014 3:31:34 AM UTC-4, Gary Verhaegen wrote: > > Sounds like a job for a future. Something like: > > (->> job-list > (partition-in-sublists 4) > (map #(future (do-job-on-sublist %))) > (mapv deref)) > > This is untested and written on a phone, so might not even be > syntactically correct, but the future calls will create new threds to > execute the do-job functions on the sublists, and the deref call on a > future is blocking. If the result from the futures is not important, I > guess you could save some memory by using map and doall instead of mapv, > though I doubt it would make any difference if you have only 4 sublists. > > Alternatively, have you looked at pmap ? > > On Wednesday, 17 September 2014, Beau Fabry <imf...@gmail.com > <javascript:>> wrote: > >> We don't have streams of data here, the long running tasks have >> side-effects. I would prefer to avoid adding another whole framework just >> to run a few long running jobs in p//. >> >> I have a list of jobs to do, I'm partitioning that list up into 4 sub >> lists to be worked through by 4 p// workers, I then want to block and wait >> until all 4 workers have finished their tasks. >> >> On Wednesday, September 17, 2014 3:27:07 AM UTC+10, larry google groups >> wrote: >>> >>> >>> This does not look correct to me. Perhaps someone else has more insight >>> into this. I am suspicious about 2 things: >>> >>> 1.) your use of doall >>> >>> 2.) your use of (thread) >>> >>> It looks to me like you are trying to hack together a kind of pipeline >>> or channel. Clojure has a wealth of libraries that can handle that for you. >>> The main thing you are trying to do is this: >>> >>> (long-running-widget-processor widget)) >>> >>> >>> You go to some trouble to set up workers, all to ensure that >>> long-running-widget-processor is handled in its own thread. >>> >>> I would suggest you look at Lamina: >>> >>> https://github.com/ztellman/lamina >>> >>> In particular, look at pipelines: >>> >>> https://github.com/ztellman/lamina/wiki/Pipelines >>> >>> >>> >>> >>> >>> On Friday, September 5, 2014 1:46:02 AM UTC-4, Beau Fabry wrote: >>>> >>>> Is the kinda ugly constant (doall usage a sign that I'm doing >>>> something silly? >>>> >>>> (let [num-workers 4 >>>> widgets-per-worker (inc (int (/ (count widgets) num-workers))) >>>> bucketed-widgets (partition-all widgets-per-worker widgets) >>>> workers (doall (map (fn [widgets] >>>> (thread >>>> (doseq [widget widgets] >>>> (long-running-widget-processor widget)) >>>> true)) >>>> bucketed-widgets))] >>>> (doall (map <!! workers))) >>>> >>>> https://gist.github.com/bfabry/ad830b1888e4fc550f88 >>>> >>>> All comments appreciated :-) >>>> >>>> Cheers, >>>> Beau >>>> >>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
