Ah, for that you need to "layer" your DAG (directed acyclic graph -- if A and B both depend on C, and are dependencies of D, you don't have a tree). Start with S an empty set and n = 0. Iteratively let Sn be the items with no dependencies not in S, S become S U Sn, and n = n + 1 until no items are not yet in any of the Si. Now initialize, perhaps in parallel, all of the S0; then all of the S1; then all of the S2 ...
On Thu, Nov 7, 2013 at 9:27 PM, Malcolm Sparks <malc...@juxt.pro> wrote: > That's a nice use of core.async :) > > The key requirement, however, is that a component's dependencies must all > initialize before a component can start initializing. That reduces > parallelism but not entirely. A component's dependencies can all be > initiated in parallel, so long as there aren't any dependencies between > each other. I'm sure there must be existing barrier sync algorithms out > there that we could use. > http://en.wikipedia.org/wiki/Barrier_%28computer_science%29 > > On Friday, November 8, 2013 12:58:35 AM UTC, Cedric Greevey wrote: > >> If you are just visiting every node as a tree, and you have core.async, >> then there's >> >> (let [c (chan)] >> (go >> (doseq [n (tree-seq ... (depends on tree implementation) ...)] >> (>! c n))) >> (dotimes [_ (.availableProcessors (Runtime/getRuntime))] >> (go >> (loop [] >> (when-let [n (<! c)] >> (expensive-visitation n) >> (recur)))))) >> >> unless there's some trickiness with core.async that I've not "gotten" >> here. >> Of course, if expensive-visitation returns something instead of having >> side effects, you'll need to accumulate some sort of a result in each loop, >> and combine these afterward (the dotimes likely becoming a for). >> >> If the visitation is really expensive, then converting the output of >> tree-seq into a vector and using reducers or pmap might also parallelize >> things significantly. >> >> If you need to do more than just visit each node -- if the tree structure >> around the node and not just some value in the node needs to be considered >> -- then it gets complicated. Something that traverses the tree, but spawns >> new threads at each branching (up to some limit perhaps), seems indicated. >> >> >> >> On Thu, Nov 7, 2013 at 6:23 PM, Timothy Washington <twas...@gmail.com>wrote: >> >>> Umm I can't think of parallel dependency tree walking algos, off the top >>> of my head. Sorry :/ >>> >>> But niiice. Yeah, we're definitely thinking along the same lines. But >>> you're way ahead of me. Now just to be able to isolate that browser repl on >>> a per project basis. >>> >>> >>> Sweet >>> >>> Tim Washington >>> Interruptsoftware.ca <http://interruptsoftware.ca/> / >>> Bkeeping.com<http://bkeeping.com/> >>> >>> >>> >>> On Wed, Nov 6, 2013 at 8:46 PM, Malcolm Sparks <mal...@juxt.pro> wrote: >>> >>>> Hi Tim, >>>> >>>> It's interesting you're thinking about browser-connected repls. Today I >>>> added clojurescript support to Jig, it's pushed to master but I haven't >>>> documented the config options yet, but below is an example. Nothing fancy >>>> like crossovers yet, it's basically a thin wrapper over ClojureScript's >>>> compiler, the config options map directly to the options given to the >>>> compiler. >>>> >>>> :cljs-builder >>>> {:jig/component jig.cljs/Builder >>>> :jig/project "../my-proj/project.clj" >>>> :source-path "../my-proj/src-cljs" >>>> :output-dir "../my-proj/target/js" >>>> :output-to "../my-proj/target/js/main.js" >>>> :source-map "../my-proj/target/js/main.js.map" >>>> :optimizations :simple >>>> :pretty-print true >>>> ;; :clean-build true >>>> } >>>> >>>> Another built-in component can figure out, given the dependencies, what >>>> files to serve and on which server to serve it from. >>>> >>>> :cljs-server >>>> {:jig/component jig.cljs/FileServer >>>> :jig/dependencies [:cljs-builder :web] >>>> :jig.web/context "/js" >>>> } >>>> >>>> which depends on some Ring or Pedestal routing and Jetty or other >>>> webserver, but you're likely to have that configured already :- >>>> >>>> >>>> :server >>>> {:jig/component jig.web.server/Component >>>> :io.pedestal.service.http/port 8000 >>>> :io.pedestal.service.http/type :jetty >>>> } >>>> >>>> :web >>>> >>>> {:jig/component jig.web.app/Component >>>> :jig/dependencies [:server] >>>> :jig/scheme :http >>>> :jig/hostname "localhost" >>>> :jig.web/server :server >>>> } >>>> >>>> That's it really. I think Jig makes an ideal platform for building and >>>> serving clojurescript, in comparison to Leiningen plugins. Firstly, there's >>>> no JVM start-up cost, the point of Jig is to keep the JVM running all the >>>> time, so there's no need for the once/auto service that lein-cljsbuild >>>> offers. Secondly, serving the resulting Javascript files is straight >>>> forward, and built in, so no 'lein ring server' requirement either.. With >>>> Leiningen, you either have to build routes to the javascript into your own >>>> app, or run lein ring server, but then Leiningen doesn't make it easy to >>>> run multiple services concurrently. Thirdly, there's no file-system >>>> polling, and services that depend on the clojurescript compilation can be >>>> started as soon as the compilation is complete. >>>> >>>> One thing that lein-cljsbuild does really well is multiple build >>>> configs. I've decided to use a single build configuration, to keep >>>> implementation easier and a direct mapping to the clojurescript compiler, >>>> but of course you can add different builds by just adding (differently >>>> configured) components to the Jig config. This would benefit from being >>>> able to initialize and start components in parallel, where possible. Unlike >>>> cljsbuild, Jig components are started serially, in reverse dependency >>>> order. Does anyone know of existing algorithms that can walk a dependency >>>> tree in parallel? >>>> >>>> Feel free to take Jig in different directions though, I'm just letting >>>> you know my current thoughts. The single REPL to multiple projects idea >>>> might have to be rethought - it seems the prevailing wind is in the >>>> direction of multiple REPL connections per editor/IDE. >>>> >>>> >>>> On Thursday, 7 November 2013 01:09:53 UTC, frye wrote: >>>> >>>>> Too right. >>>>> >>>>> Yes, wrt the multi-project / classpath thing, running isolated test >>>>> for a particular project is only one aspect. I also have an eye to running >>>>> a i) browser-connected repl and ii) debugger for a particular project. So >>>>> those things, along with iii) running tests, make very high, the >>>>> attractiveness of having that isolation. >>>>> >>>>> I'll try to put those in github issues. And also pick at the problem >>>>> myself when I get some cycles. I think it would add a powerful feature to >>>>> the tool. Anyways... :) >>>>> >>>>> >>>>> Tim Washington >>>>> Interruptsoftware.ca <http://interruptsoftware.ca/> / >>>>> Bkeeping.com<http://bkeeping.com/> >>>>> >>>>> >>>> -- >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+u...@googlegroups.com. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
