I think you present a key question: what assumptions can a transducer make? We know the standard ones, but what of memory barriers? Based on the current implementation, in terms of concurrency, it seems to make (inconsistent — see also `partition-by`) guarantees that sequential writes and reads will be consistent, no matter what thread does the reads or writes. Concurrent writes are not supported. But *should *sequential multi-threaded reads/writes be supported? This is a question best left to Alex but I think I already know the answer based on his conversation with Rich: it's part of the contract.
I think another key question is, is the channel lock memory barrier part of the contract of a core.async channel implementation? If not, volatiles will be necessary in that context if the memory barrier is ever taken away, and it would make sense that volatiles are used in transducers "just in case" specifically for that use case. But if the channel lock memory barrier *is* part of the contract and not just an implementation detail, then I'm not certain that it's very useful at all for transducers to provide a guarantee of safe sequential multi-threaded reads/writes. On Monday, April 10, 2017 at 2:57:10 PM UTC-4, Léo Noel wrote: > > What you said holds for reduction but not necessarily a parallel fold (see >> clojure.core.reducers/fold). >> > > Exactly, and that's why stateful transducers are explicitly forbidden in > fold and in core.async pipeline functions. > This is not related to memory visibility, this is due to the fact that > stateful transducers force the reducing process to be sequential. > Does it make any sense to parallelize map-indexed ? partition-all ? dedupe > ? > > > These kinds of failures are inherently difficult to reproduce unless the >> code is in production and you're on vacation. ;) >> > > Couldn't agree more. However, we're all clever people and the Java Memory > Model is not magic :) > > > Léo, I definitely agree that you can use unsynchronized mutable stateful >> transducers *as long as you can guarantee they'll be used only in >> single-threaded contexts.* > > > The problem is at a lower level. The memory model of the JVM >> doesn't guarantee that changes to an unsynchronized non-volatile reference >> are visible to other threads. >> > > The Java Memory Model allows using unsynchronized variables to share data > across threads as long as a memory barrier is set between the writer and > the reader. For example, in the case of core.async, the channel lock sets a > barrier, and there is also (redundant) barriers for each volatile inside > transducers. > > > A transducing process could apply each step of the transduce using a >> thread from a pool and also not use a memory barrier > > > >> Transducers included in core cannot make the assumption that they will >> only be used that way. >> > > Yes, that makes sense that you can't make that assumption. > > > This is the key point : what assumptions a transducer can make ? > In my opinion, it is reasonable for a stateful transducer to assume that > the transducing context will fulfill the contract of "always passing the > result of step n to the first argument of step n+1". > This assumption is powerful because it guarantees that there will always > be a memory barrier between two successive steps. > Proof (reductio ad absurdum) : without a memory barrier, the result of the > step n wouldn't be visible to the (potentially different) thread performing > the step n+1. > So here is my question to the language designers : is it reasonable to > assume that ? > If yes, that means it's ok to use unsynchronized variables in stateful > transducers as long as they stay local. > If no, that means we'll use synchronization in all stateful transducers, > with an obvious performance penalty and a benefit that remains unclear. > > > > On Monday, April 10, 2017 at 7:34:39 PM UTC+2, Alexander Gunnarson wrote: >> >> Yes, that makes sense that you can't make that assumption. You'd have to >> create something like what I was discussing above: >> >> (defn map-indexed-transducer-base [f box-mutable inc-mutable] >> (fn [rf] >> (let [i (box-mutable -1)] >> (fn >> ([] (rf)) >> ([result] (rf result)) >> ([result input] >> (rf result (f (inc-mutable i) input))))))) >> >> ;; this is the version that Léo would want >> (defn map-indexed-transducer-single-threaded [f] >> (map-indexed-transducer-base f unsynchronized-mutable-long! >> #(unsynchronized-mutable-swap! >> % inc)) >> >> ;; this is the version included in clojure.core >> (defn map-indexed-transducer-sequentially-accessed-by-different-threads [ >> f] >> (map-indexed-transducer-base f volatile! #(vswap! % inc)) >> >> ;; this works with `fold` and gives you all the indices at least, but in >> a nondeterministic order >> (defn map-indexed-transducer-concurrently-accessed-by-different-threads [ >> f] >> (map-indexed-transducer-base f atom #(swap! % inc)) ; or an AtomicLong >> variant >> >> On Monday, April 10, 2017 at 1:06:14 PM UTC-4, Alex Miller wrote: >>> >>> >>> On Monday, April 10, 2017 at 11:48:41 AM UTC-5, Alexander Gunnarson >>> wrote: >>>> >>>> Léo, I definitely agree that you can use unsynchronized mutable >>>> stateful transducers *as long as you can guarantee they'll be used >>>> only in single-threaded contexts. * >>>> >>> >>> Transducers included in core cannot make the assumption that they will >>> only be used that way. (But you may be able to guarantee that with your >>> own.) >>> >>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
