That's a nice use of core.async :) The key requirement, however, is that a component's dependencies must all initialize before a component can start initializing. That reduces parallelism but not entirely. A component's dependencies can all be initiated in parallel, so long as there aren't any dependencies between each other. I'm sure there must be existing barrier sync algorithms out there that we could use. http://en.wikipedia.org/wiki/Barrier_%28computer_science%29
On Friday, November 8, 2013 12:58:35 AM UTC, Cedric Greevey wrote: > > If you are just visiting every node as a tree, and you have core.async, > then there's > > (let [c (chan)] > (go > (doseq [n (tree-seq ... (depends on tree implementation) ...)] > (>! c n))) > (dotimes [_ (.availableProcessors (Runtime/getRuntime))] > (go > (loop [] > (when-let [n (<! c)] > (expensive-visitation n) > (recur)))))) > > unless there's some trickiness with core.async that I've not "gotten" > here. > Of course, if expensive-visitation returns something instead of having > side effects, you'll need to accumulate some sort of a result in each loop, > and combine these afterward (the dotimes likely becoming a for). > > If the visitation is really expensive, then converting the output of > tree-seq into a vector and using reducers or pmap might also parallelize > things significantly. > > If you need to do more than just visit each node -- if the tree structure > around the node and not just some value in the node needs to be considered > -- then it gets complicated. Something that traverses the tree, but spawns > new threads at each branching (up to some limit perhaps), seems indicated. > > > > On Thu, Nov 7, 2013 at 6:23 PM, Timothy Washington > <twas...@gmail.com<javascript:> > > wrote: > >> Umm I can't think of parallel dependency tree walking algos, off the top >> of my head. Sorry :/ >> >> But niiice. Yeah, we're definitely thinking along the same lines. But >> you're way ahead of me. Now just to be able to isolate that browser repl on >> a per project basis. >> >> >> Sweet >> >> Tim Washington >> Interruptsoftware.ca <http://interruptsoftware.ca/> / >> Bkeeping.com<http://bkeeping.com/> >> >> >> >> On Wed, Nov 6, 2013 at 8:46 PM, Malcolm Sparks <mal...@juxt.pro<javascript:> >> > wrote: >> >>> Hi Tim, >>> >>> It's interesting you're thinking about browser-connected repls. Today I >>> added clojurescript support to Jig, it's pushed to master but I haven't >>> documented the config options yet, but below is an example. Nothing fancy >>> like crossovers yet, it's basically a thin wrapper over ClojureScript's >>> compiler, the config options map directly to the options given to the >>> compiler. >>> >>> :cljs-builder >>> {:jig/component jig.cljs/Builder >>> :jig/project "../my-proj/project.clj" >>> :source-path "../my-proj/src-cljs" >>> :output-dir "../my-proj/target/js" >>> :output-to "../my-proj/target/js/main.js" >>> :source-map "../my-proj/target/js/main.js.map" >>> :optimizations :simple >>> :pretty-print true >>> ;; :clean-build true >>> } >>> >>> Another built-in component can figure out, given the dependencies, what >>> files to serve and on which server to serve it from. >>> >>> :cljs-server >>> {:jig/component jig.cljs/FileServer >>> :jig/dependencies [:cljs-builder :web] >>> :jig.web/context "/js" >>> } >>> >>> which depends on some Ring or Pedestal routing and Jetty or other >>> webserver, but you're likely to have that configured already :- >>> >>> >>> :server >>> {:jig/component jig.web.server/Component >>> :io.pedestal.service.http/port 8000 >>> :io.pedestal.service.http/type :jetty >>> } >>> >>> :web >>> >>> {:jig/component jig.web.app/Component >>> :jig/dependencies [:server] >>> :jig/scheme :http >>> :jig/hostname "localhost" >>> :jig.web/server :server >>> } >>> >>> That's it really. I think Jig makes an ideal platform for building and >>> serving clojurescript, in comparison to Leiningen plugins. Firstly, there's >>> no JVM start-up cost, the point of Jig is to keep the JVM running all the >>> time, so there's no need for the once/auto service that lein-cljsbuild >>> offers. Secondly, serving the resulting Javascript files is straight >>> forward, and built in, so no 'lein ring server' requirement either.. With >>> Leiningen, you either have to build routes to the javascript into your own >>> app, or run lein ring server, but then Leiningen doesn't make it easy to >>> run multiple services concurrently. Thirdly, there's no file-system >>> polling, and services that depend on the clojurescript compilation can be >>> started as soon as the compilation is complete. >>> >>> One thing that lein-cljsbuild does really well is multiple build >>> configs. I've decided to use a single build configuration, to keep >>> implementation easier and a direct mapping to the clojurescript compiler, >>> but of course you can add different builds by just adding (differently >>> configured) components to the Jig config. This would benefit from being >>> able to initialize and start components in parallel, where possible. Unlike >>> cljsbuild, Jig components are started serially, in reverse dependency >>> order. Does anyone know of existing algorithms that can walk a dependency >>> tree in parallel? >>> >>> Feel free to take Jig in different directions though, I'm just letting >>> you know my current thoughts. The single REPL to multiple projects idea >>> might have to be rethought - it seems the prevailing wind is in the >>> direction of multiple REPL connections per editor/IDE. >>> >>> >>> On Thursday, 7 November 2013 01:09:53 UTC, frye wrote: >>> >>>> Too right. >>>> >>>> Yes, wrt the multi-project / classpath thing, running isolated test for >>>> a particular project is only one aspect. I also have an eye to running a >>>> i) >>>> browser-connected repl and ii) debugger for a particular project. So those >>>> things, along with iii) running tests, make very high, the attractiveness >>>> of having that isolation. >>>> >>>> I'll try to put those in github issues. And also pick at the problem >>>> myself when I get some cycles. I think it would add a powerful feature to >>>> the tool. Anyways... :) >>>> >>>> >>>> Tim Washington >>>> Interruptsoftware.ca <http://interruptsoftware.ca/> / >>>> Bkeeping.com<http://bkeeping.com/> >>>> >>>> >>> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com<javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
