Larry your solution includes the cognitive overhead of another entire library and process model. "future" is part of core, and as I realised when Gary posted the doall's were unnecessary anyway.
On Thursday, September 18, 2014 3:26:36 PM UTC+10, larry google groups wrote: > > > > Thanks for that Larry but I think this is a bit of overkill for my > scenario. > > If I'm counting correctly, your original example has 10 lines of code, and > my example has 11 lines of code (minus the try/catch and the closure and > the namespace declaration). So these 2 solutions are the same length. These > are the 11 lines of code: > > (def ^:private persistence-channel (lamina/channel)) > > (defn persist-this-item [context-wrapper-for-database-call] > (lamina/enqueue persistence-channel > (fn [] (persistence/make-consistent > context-wrapper-for-database-call)))) > > (defn worker [] > (loop [closure-with-item-inside @(lamina/read-channel > persistence-channel)] > (closure-with-item-inside) > (recur @(lamina/read-channel persistence-channel)))) > > (defn start-workers [] > (dotimes [_ 6] > (future (worker)))) > > Maybe its just that I'm familiar with the code, but these 11 lines of code > seem cleaner to me than your 10 lines of code, at least in part because you > are doing stuff like calling (doall) which strikes me as a bit suspicious. > > Do you really feel the 1 extra line of code is overkill? This solution > seems to do what you want, and it's the same length as your solution. > > > > > > On Wednesday, September 17, 2014 8:48:09 PM UTC-4, Beau Fabry wrote: >> >> Thanks for that Larry but I think this is a bit of overkill for my >> scenario. The code I pasted is almost verbatim what we have in our >> production codebase, so the ability to queue new jobs etc is really not >> needed. Cheers though. >> >> On Thursday, September 18, 2014 9:38:47 AM UTC+10, larry google groups >> wrote: >>> >>> >>> > We don't have streams of data here, the long running tasks have >>> side-effects. I would >>> > prefer to avoid adding another whole framework just to run a few long >>> running jobs in p//. >>> >>> >>> I guess I should show you some code, so you can see how simple this is. >>> I'll copy-and-paste some code that I use. >>> >>> One simple way I use Lamina is to save stuff to a database. I don't want >>> the "save" action happening in my main thread, so I put the data in a >>> channel, and I let some workers pull that data off the channel and put it >>> in the database. So what follows is the whole file, this about 30 lines of >>> code, including some try/catch stuff that you probably don't need: >>> >>> (ns loupi.persistence-queue >>> (:require >>> [loupi.persistence :as persistence] >>> [slingshot.slingshot :as ss] >>> [lamina.core :as lamina])) >>> >>> (def ^:private persistence-channel (lamina/channel)) >>> >>> (defn persist-this-item [context-wrapper-for-database-call] >>> (lamina/enqueue persistence-channel >>> (fn [] >>> (ss/try+ >>> (persistence/make-consistent >>> context-wrapper-for-database-call) >>> (catch Object o (ss/throw+ {:type >>> :loupi.supervisor/problem >>> :message "Error in >>> persistence-queue/persist-this-itme." >>> :data o})))))) >>> >>> (defn worker [] >>> (loop [closure-with-item-inside @(lamina/read-channel >>> persistence-channel)] >>> (ss/try+ >>> (closure-with-item-inside) >>> (catch Object o (ss/throw+ {:type :loupi.supervisor/problem >>> :message "Error in >>> persistence-queue/worker." >>> :closure closure-with-item-inside >>> :data o}))) >>> (recur @(lamina/read-channel persistence-channel)))) >>> >>> (defn start-workers [] >>> (dotimes [_ 6] >>> (println "Starting up the persist queue workers.") >>> (future (worker)))) >>> >>> >>> >>> I call (start-workers) when the app starts. When I save something to the >>> database, I call (persist-this-item) and I put a closure on the channel. >>> The workers eventually grab that closure and execute it. >>> >>> Clearly, that closure can do whatever you like. To borrow from your >>> original example, that closure is where you would put: >>> >>> (long-running-widget-processor widget) >>> >>> >>> >>> >>> >>> >>> >>> >>> On Tuesday, September 16, 2014 10:00:07 PM UTC-4, Beau Fabry wrote: >>>> >>>> We don't have streams of data here, the long running tasks have >>>> side-effects. I would prefer to avoid adding another whole framework just >>>> to run a few long running jobs in p//. >>>> >>>> I have a list of jobs to do, I'm partitioning that list up into 4 sub >>>> lists to be worked through by 4 p// workers, I then want to block and wait >>>> until all 4 workers have finished their tasks. >>>> >>>> On Wednesday, September 17, 2014 3:27:07 AM UTC+10, larry google groups >>>> wrote: >>>>> >>>>> >>>>> This does not look correct to me. Perhaps someone else has more >>>>> insight into this. I am suspicious about 2 things: >>>>> >>>>> 1.) your use of doall >>>>> >>>>> 2.) your use of (thread) >>>>> >>>>> It looks to me like you are trying to hack together a kind of pipeline >>>>> or channel. Clojure has a wealth of libraries that can handle that for >>>>> you. >>>>> The main thing you are trying to do is this: >>>>> >>>>> (long-running-widget-processor widget)) >>>>> >>>>> >>>>> You go to some trouble to set up workers, all to ensure that >>>>> long-running-widget-processor >>>>> is handled in its own thread. >>>>> >>>>> I would suggest you look at Lamina: >>>>> >>>>> https://github.com/ztellman/lamina >>>>> >>>>> In particular, look at pipelines: >>>>> >>>>> https://github.com/ztellman/lamina/wiki/Pipelines >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Friday, September 5, 2014 1:46:02 AM UTC-4, Beau Fabry wrote: >>>>>> >>>>>> Is the kinda ugly constant (doall usage a sign that I'm doing >>>>>> something silly? >>>>>> >>>>>> (let [num-workers 4 >>>>>> widgets-per-worker (inc (int (/ (count widgets) num-workers))) >>>>>> bucketed-widgets (partition-all widgets-per-worker widgets) >>>>>> workers (doall (map (fn [widgets] >>>>>> (thread >>>>>> (doseq [widget widgets] >>>>>> (long-running-widget-processor widget)) >>>>>> true)) >>>>>> bucketed-widgets))] >>>>>> (doall (map <!! workers))) >>>>>> >>>>>> https://gist.github.com/bfabry/ad830b1888e4fc550f88 >>>>>> >>>>>> All comments appreciated :-) >>>>>> >>>>>> Cheers, >>>>>> Beau >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.