Yeah, that goes in a direction I thought about after my post. I'm going to implement sth. like this and we will see how much code overhead is necessary.
At the time it appears to me that being able to locally dedicate threads to the go mechanism would make this much easier (than a hand controlled implementation) in the same way that Erlangs VM can park everything and utilizes only as many threads as cores exist for everything... Thanks for info regarding pmap, I will check it out. On Tuesday, May 1, 2018 at 3:45:49 PM UTC+2, Justin Smith wrote: > > Just a couple of small points (and not yet a full answer): > > > A node can obviously not pmap over all the child nodes (would spawn > exponential amount of threads) > > pmap is not that naive, it uses a pool sized with the assumption that its > work is CPU bound > > > (2) Made me wonder why I couldn't use the go machinery for this. Parent > nodes that would "wait" for their child nodes to complete would park, > making their thread available to their child nodes. > > This is why async/thread exists, it returns a channel you can park in, and > you can use channels coming into the blocks starting threads to control the > parallelism by hand. > > On Tue, May 1, 2018, 03:41 Leon Grapenthin <grapent...@gmail.com > <javascript:>> wrote: > >> Recently I worked on an algorithm where a distributed tree is (sort of) >> flattened in a way that each node runs a commutative aggregation over all >> of its child nodes calculations. >> >> 1 A node can obviously not pmap over all the child nodes (would spawn >> exponential amount of threads). >> >> 2 If I want to limit the threads using a threadpool, child nodes will >> quickly run out of threads and we can't reuse their parents threads because >> they are blocking, waiting for the child nodes to complete, calculating >> nothing. >> >> It sure could be possible to implement this with an unlimited threadpool >> and some machinery that ensures that no more than N computations are >> happening in parallel. Nonetheless an insane amount of threads would be >> spawned. >> >> (2) Made me wonder why I couldn't use the go machinery for this. Parent >> nodes that would "wait" for their child nodes to complete would park, >> making their thread available to their child nodes. >> >> So of course I could just do it, but the per node computations are >> presumably too heavy for the lightweight go threadpool. >> >> How realistic/useful would it be to have sth. like a (go-with threadpool >> []) in this scenario? >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
