pmap isn't an option as the processes kicked off could affect other systems load if we can't control the level of parallelization. futures seem like they'd work quite well (the return value of the jobs is nil, it's a doseq). I might rewrite it with futures at some point. Although it really just seems like a slightly cleaner way of doing exactly what I did with core.async :-)
Thanks Gary! On Wednesday, September 17, 2014 5:31:34 PM UTC+10, Gary Verhaegen wrote: > > Sounds like a job for a future. Something like: > > (->> job-list > (partition-in-sublists 4) > (map #(future (do-job-on-sublist %))) > (mapv deref)) > > This is untested and written on a phone, so might not even be > syntactically correct, but the future calls will create new threds to > execute the do-job functions on the sublists, and the deref call on a > future is blocking. If the result from the futures is not important, I > guess you could save some memory by using map and doall instead of mapv, > though I doubt it would make any difference if you have only 4 sublists. > > Alternatively, have you looked at pmap ? > > On Wednesday, 17 September 2014, Beau Fabry <imf...@gmail.com > <javascript:>> wrote: > >> We don't have streams of data here, the long running tasks have >> side-effects. I would prefer to avoid adding another whole framework just >> to run a few long running jobs in p//. >> >> I have a list of jobs to do, I'm partitioning that list up into 4 sub >> lists to be worked through by 4 p// workers, I then want to block and wait >> until all 4 workers have finished their tasks. >> >> On Wednesday, September 17, 2014 3:27:07 AM UTC+10, larry google groups >> wrote: >>> >>> >>> This does not look correct to me. Perhaps someone else has more insight >>> into this. I am suspicious about 2 things: >>> >>> 1.) your use of doall >>> >>> 2.) your use of (thread) >>> >>> It looks to me like you are trying to hack together a kind of pipeline >>> or channel. Clojure has a wealth of libraries that can handle that for you. >>> The main thing you are trying to do is this: >>> >>> (long-running-widget-processor widget)) >>> >>> >>> You go to some trouble to set up workers, all to ensure that >>> long-running-widget-processor is handled in its own thread. >>> >>> I would suggest you look at Lamina: >>> >>> https://github.com/ztellman/lamina >>> >>> In particular, look at pipelines: >>> >>> https://github.com/ztellman/lamina/wiki/Pipelines >>> >>> >>> >>> >>> >>> On Friday, September 5, 2014 1:46:02 AM UTC-4, Beau Fabry wrote: >>>> >>>> Is the kinda ugly constant (doall usage a sign that I'm doing >>>> something silly? >>>> >>>> (let [num-workers 4 >>>> widgets-per-worker (inc (int (/ (count widgets) num-workers))) >>>> bucketed-widgets (partition-all widgets-per-worker widgets) >>>> workers (doall (map (fn [widgets] >>>> (thread >>>> (doseq [widget widgets] >>>> (long-running-widget-processor widget)) >>>> true)) >>>> bucketed-widgets))] >>>> (doall (map <!! workers))) >>>> >>>> https://gist.github.com/bfabry/ad830b1888e4fc550f88 >>>> >>>> All comments appreciated :-) >>>> >>>> Cheers, >>>> Beau >>>> >>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
