We really need more information if you're expecting us to offer any help at all. How many items are you updating, what are your update patterns, what is in these refs, are do you need to modify more than one ref in a single "transaction". All this impacts the correct approach. But with such a lack of information we're just taking stabs in the dark.
Timothy On Mon, Jan 30, 2017 at 8:40 PM, Brian Craft <craft.br...@gmail.com> wrote: > I don't think parallel reducers can handle the conflict issue: two threads > making incompatible changes to the vector. > > > On Monday, January 30, 2017 at 7:01:22 PM UTC-8, Josh Tilles wrote: >> >> If your goal is “to operate on large data structures in parallel” then >> you’ll probably find Clojure’s reducers library >> <https://clojure.org/reference/reducers> to be helpful. >> >> On Monday, January 30, 2017 at 9:38:01 PM UTC-5, Brian Craft wrote: >>> >>> ans: this scales badly. >>> >>> There must be a way in clojure to operate on large data structures in >>> parallel, no? >>> >>> On Monday, January 30, 2017 at 6:03:39 PM UTC-8, Brian Craft wrote: >>>> >>>> Would this not scale badly? When the vector is hundreds of thousands, >>>> or millions? >>>> >>>> On Monday, January 30, 2017 at 5:54:32 PM UTC-8, tbc++ wrote: >>>>> >>>>> Instead of looking at the state as a ref with a vector in it, think of >>>>> it as a vector of refs. That then allows multiple refs to be modified at >>>>> once without stepping on other unrelated refs. >>>>> >>>>> On Mon, Jan 30, 2017 at 5:26 PM, Brian Craft <craft...@gmail.com> >>>>> wrote: >>>>> >>>>>> I'm experimenting with ref, dosync, and alter to run some code in >>>>>> parallel, but so far haven't been able to figure out how to structure it >>>>>> such that it runs faster, instead of slower. >>>>>> >>>>>> The problem looks something like this. >>>>>> >>>>>> The current state is a long vector. Input is a long sequence of >>>>>> values. At each step some positions in the vector are populated, if >>>>>> empty, >>>>>> based on computations over the next value in the sequence. >>>>>> >>>>>> It's like placing puzzle pieces: search for positions that satisfy a >>>>>> constraint, and fill them. >>>>>> >>>>>> Using threads, I'd like to place the next several values in parallel. >>>>>> But running in parallel it's possible that two threads would find >>>>>> solutions >>>>>> that conflict. In this case, the latter one must continue searching. >>>>>> >>>>>> Is there a way to express this with clojure? The problem is that I >>>>>> can't tell 'dosync' when a previous transaction invalidates the current >>>>>> transaction: it just always retries. >>>>>> >>>>>> e.g. if each thread is doing >>>>>> >>>>>> >>>>>> (dosync >>>>>> (let [positions (find-positions next-value @state)] >>>>>> (alter state (fn [s] (update-state s positions)))))) >>>>>> >>>>>> they interfere with each other constantly, because any write to >>>>>> 'state' causes the other threads to retry, even if their positions are >>>>>> still free. >>>>>> >>>>>> One really wants to reverse the order of find-positions and dosync >>>>>> here, so it computes positions, then tries to commit them, checking in >>>>>> the >>>>>> transaction that the positions are still free, and continuing the search >>>>>> if >>>>>> they're not. >>>>>> >>>>>> I suppose there's some solution involving exceptions. Is there a >>>>>> better way to think about this? >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>> Note that posts from new members are moderated - please be patient >>>>>> with your first post. >>>>>> To unsubscribe from this group, send email to >>>>>> clojure+u...@googlegroups.com >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/clojure?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to clojure+u...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> “One of the main causes of the fall of the Roman Empire was >>>>> that–lacking zero–they had no way to indicate successful termination of >>>>> their C programs.” >>>>> (Robert Firth) >>>>> >>>> -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
