> > What you said holds for reduction but not necessarily a parallel fold (see > clojure.core.reducers/fold). >
Exactly, and that's why stateful transducers are explicitly forbidden in fold and in core.async pipeline functions. This is not related to memory visibility, this is due to the fact that stateful transducers force the reducing process to be sequential. Does it make any sense to parallelize map-indexed ? partition-all ? dedupe ? These kinds of failures are inherently difficult to reproduce unless the > code is in production and you're on vacation. ;) > Couldn't agree more. However, we're all clever people and the Java Memory Model is not magic :) Léo, I definitely agree that you can use unsynchronized mutable stateful > transducers *as long as you can guarantee they'll be used only in > single-threaded contexts.* The problem is at a lower level. The memory model of the JVM > doesn't guarantee that changes to an unsynchronized non-volatile reference > are visible to other threads. > The Java Memory Model allows using unsynchronized variables to share data across threads as long as a memory barrier is set between the writer and the reader. For example, in the case of core.async, the channel lock sets a barrier, and there is also (redundant) barriers for each volatile inside transducers. A transducing process could apply each step of the transduce using a thread > from a pool and also not use a memory barrier > Transducers included in core cannot make the assumption that they will > only be used that way. > Yes, that makes sense that you can't make that assumption. This is the key point : what assumptions a transducer can make ? In my opinion, it is reasonable for a stateful transducer to assume that the transducing context will fulfill the contract of "always passing the result of step n to the first argument of step n+1". This assumption is powerful because it guarantees that there will always be a memory barrier between two successive steps. Proof (reductio ad absurdum) : without a memory barrier, the result of the step n wouldn't be visible to the (potentially different) thread performing the step n+1. So here is my question to the language designers : is it reasonable to assume that ? If yes, that means it's ok to use unsynchronized variables in stateful transducers as long as they stay local. If no, that means we'll use synchronization in all stateful transducers, with an obvious performance penalty and a benefit that remains unclear. On Monday, April 10, 2017 at 7:34:39 PM UTC+2, Alexander Gunnarson wrote: > > Yes, that makes sense that you can't make that assumption. You'd have to > create something like what I was discussing above: > > (defn map-indexed-transducer-base [f box-mutable inc-mutable] > (fn [rf] > (let [i (box-mutable -1)] > (fn > ([] (rf)) > ([result] (rf result)) > ([result input] > (rf result (f (inc-mutable i) input))))))) > > ;; this is the version that Léo would want > (defn map-indexed-transducer-single-threaded [f] > (map-indexed-transducer-base f unsynchronized-mutable-long! > #(unsynchronized-mutable-swap! > % inc)) > > ;; this is the version included in clojure.core > (defn map-indexed-transducer-sequentially-accessed-by-different-threads [f > ] > (map-indexed-transducer-base f volatile! #(vswap! % inc)) > > ;; this works with `fold` and gives you all the indices at least, but in a > nondeterministic order > (defn map-indexed-transducer-concurrently-accessed-by-different-threads [f > ] > (map-indexed-transducer-base f atom #(swap! % inc)) ; or an AtomicLong > variant > > On Monday, April 10, 2017 at 1:06:14 PM UTC-4, Alex Miller wrote: >> >> >> On Monday, April 10, 2017 at 11:48:41 AM UTC-5, Alexander Gunnarson wrote: >>> >>> Léo, I definitely agree that you can use unsynchronized mutable stateful >>> transducers *as long as you can guarantee they'll be used only in >>> single-threaded contexts. * >>> >> >> Transducers included in core cannot make the assumption that they will >> only be used that way. (But you may be able to guarantee that with your >> own.) >> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
