ans: this scales badly. There must be a way in clojure to operate on large data structures in parallel, no?
On Monday, January 30, 2017 at 6:03:39 PM UTC-8, Brian Craft wrote: > > Would this not scale badly? When the vector is hundreds of thousands, or > millions? > > On Monday, January 30, 2017 at 5:54:32 PM UTC-8, tbc++ wrote: >> >> Instead of looking at the state as a ref with a vector in it, think of it >> as a vector of refs. That then allows multiple refs to be modified at once >> without stepping on other unrelated refs. >> >> On Mon, Jan 30, 2017 at 5:26 PM, Brian Craft <craft...@gmail.com> wrote: >> >>> I'm experimenting with ref, dosync, and alter to run some code in >>> parallel, but so far haven't been able to figure out how to structure it >>> such that it runs faster, instead of slower. >>> >>> The problem looks something like this. >>> >>> The current state is a long vector. Input is a long sequence of values. >>> At each step some positions in the vector are populated, if empty, based on >>> computations over the next value in the sequence. >>> >>> It's like placing puzzle pieces: search for positions that satisfy a >>> constraint, and fill them. >>> >>> Using threads, I'd like to place the next several values in parallel. >>> But running in parallel it's possible that two threads would find solutions >>> that conflict. In this case, the latter one must continue searching. >>> >>> Is there a way to express this with clojure? The problem is that I can't >>> tell 'dosync' when a previous transaction invalidates the current >>> transaction: it just always retries. >>> >>> e.g. if each thread is doing >>> >>> >>> (dosync >>> (let [positions (find-positions next-value @state)] >>> (alter state (fn [s] (update-state s positions)))))) >>> >>> they interfere with each other constantly, because any write to 'state' >>> causes the other threads to retry, even if their positions are still free. >>> >>> One really wants to reverse the order of find-positions and dosync here, >>> so it computes positions, then tries to commit them, checking in the >>> transaction that the positions are still free, and continuing the search if >>> they're not. >>> >>> I suppose there's some solution involving exceptions. Is there a better >>> way to think about this? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clo...@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+u...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+u...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> “One of the main causes of the fall of the Roman Empire was that–lacking >> zero–they had no way to indicate successful termination of their C >> programs.” >> (Robert Firth) >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
