Re: structuring parallel code

1) When you work with numerics, you have to take into account that numerical operations are typically order(s) of magnitude faster and consume much less resources per element than any of those concurrency mechanisms. 2) It is important whether the problem you're working on parallelizable in itse

Re: structuring parallel code

I think java locks may be the only good answer. I can't usefully divide the vector, because the distribution of updates is uniform along the length of it. Perhaps there's a solution with queues, with multiple threads generating potential placements, and a single thread updating the vector and

Re: structuring parallel code

We really need more information if you're expecting us to offer any help at all. How many items are you updating, what are your update patterns, what is in these refs, are do you need to modify more than one ref in a single "transaction". All this impacts the correct approach. But with such a lack

Re: structuring parallel code

I don't think parallel reducers can handle the conflict issue: two threads making incompatible changes to the vector. On Monday, January 30, 2017 at 7:01:22 PM UTC-8, Josh Tilles wrote: > > If your goal is “to operate on large data structures in parallel” then > you’ll probably find Clojure’s re

Re: structuring parallel code

One technique is to batch locks at a coarser granularity. You've explored both ends of the spectrum - 1 lock and N locks. You can also divide the overall vector into any group of refs between 1 and N. If refs are too heavy, there are several other locking mechanisms on the JVM. You could try Cl

Re: structuring parallel code

If your goal is “to operate on large data structures in parallel” then you’ll probably find Clojure’s reducers library to be helpful. On Monday, January 30, 2017 at 9:38:01 PM UTC-5, Brian Craft wrote: > > ans: this scales badly. > > There must be a way i

Re: structuring parallel code

Please define "scales badly" what are you measuring to reach that conclusion? On Mon, Jan 30, 2017 at 7:38 PM, Brian Craft wrote: > ans: this scales badly. > > There must be a way in clojure to operate on large data structures in > parallel, no? > > > On Monday, January 30, 2017 at 6:03:39 PM UT

Re: structuring parallel code

ans: this scales badly. There must be a way in clojure to operate on large data structures in parallel, no? On Monday, January 30, 2017 at 6:03:39 PM UTC-8, Brian Craft wrote: > > Would this not scale badly? When the vector is hundreds of thousands, or > millions? > > On Monday, January 30, 201

Re: structuring parallel code

As you mentioned, what you're battling right now is contention. In your existing code the threads always conflict. No matter what, only one thread's update can succeed, the others will fail and have to re-run. With the refs-in-a-vector approach as long as two threads don't touch the same cell (ref)

Re: structuring parallel code

Would this not scale badly? When the vector is hundreds of thousands, or millions? On Monday, January 30, 2017 at 5:54:32 PM UTC-8, tbc++ wrote: > > Instead of looking at the state as a ref with a vector in it, think of it > as a vector of refs. That then allows multiple refs to be modified at o

Re: structuring parallel code

Instead of looking at the state as a ref with a vector in it, think of it as a vector of refs. That then allows multiple refs to be modified at once without stepping on other unrelated refs. On Mon, Jan 30, 2017 at 5:26 PM, Brian Craft wrote: > I'm experimenting with ref, dosync, and alter to ru

structuring parallel code

I'm experimenting with ref, dosync, and alter to run some code in parallel, but so far haven't been able to figure out how to structure it such that it runs faster, instead of slower. The problem looks something like this. The current state is a long vector. Input is a long sequence of values.