I don't think parallel reducers can handle the conflict issue: two threads making incompatible changes to the vector.
On Monday, January 30, 2017 at 7:01:22 PM UTC-8, Josh Tilles wrote: > > If your goal is “to operate on large data structures in parallel” then > you’ll probably find Clojure’s reducers library > <https://clojure.org/reference/reducers> to be helpful. > > On Monday, January 30, 2017 at 9:38:01 PM UTC-5, Brian Craft wrote: >> >> ans: this scales badly. >> >> There must be a way in clojure to operate on large data structures in >> parallel, no? >> >> On Monday, January 30, 2017 at 6:03:39 PM UTC-8, Brian Craft wrote: >>> >>> Would this not scale badly? When the vector is hundreds of thousands, or >>> millions? >>> >>> On Monday, January 30, 2017 at 5:54:32 PM UTC-8, tbc++ wrote: >>>> >>>> Instead of looking at the state as a ref with a vector in it, think of >>>> it as a vector of refs. That then allows multiple refs to be modified at >>>> once without stepping on other unrelated refs. >>>> >>>> On Mon, Jan 30, 2017 at 5:26 PM, Brian Craft <craft...@gmail.com> >>>> wrote: >>>> >>>>> I'm experimenting with ref, dosync, and alter to run some code in >>>>> parallel, but so far haven't been able to figure out how to structure it >>>>> such that it runs faster, instead of slower. >>>>> >>>>> The problem looks something like this. >>>>> >>>>> The current state is a long vector. Input is a long sequence of >>>>> values. At each step some positions in the vector are populated, if >>>>> empty, >>>>> based on computations over the next value in the sequence. >>>>> >>>>> It's like placing puzzle pieces: search for positions that satisfy a >>>>> constraint, and fill them. >>>>> >>>>> Using threads, I'd like to place the next several values in parallel. >>>>> But running in parallel it's possible that two threads would find >>>>> solutions >>>>> that conflict. In this case, the latter one must continue searching. >>>>> >>>>> Is there a way to express this with clojure? The problem is that I >>>>> can't tell 'dosync' when a previous transaction invalidates the current >>>>> transaction: it just always retries. >>>>> >>>>> e.g. if each thread is doing >>>>> >>>>> >>>>> (dosync >>>>> (let [positions (find-positions next-value @state)] >>>>> (alter state (fn [s] (update-state s positions)))))) >>>>> >>>>> they interfere with each other constantly, because any write to >>>>> 'state' causes the other threads to retry, even if their positions are >>>>> still free. >>>>> >>>>> One really wants to reverse the order of find-positions and dosync >>>>> here, so it computes positions, then tries to commit them, checking in >>>>> the >>>>> transaction that the positions are still free, and continuing the search >>>>> if >>>>> they're not. >>>>> >>>>> I suppose there's some solution involving exceptions. Is there a >>>>> better way to think about this? >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clo...@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+u...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to clojure+u...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> “One of the main causes of the fall of the Roman Empire was >>>> that–lacking zero–they had no way to indicate successful termination of >>>> their C programs.” >>>> (Robert Firth) >>>> >>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
