> > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed).
Totally agree, that's why volatile instead of unsynchronized in transducer sounds like belt and suspenders to me. On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> On Wednesday, December 7, 2016 at 7:00:09 PM UTC+1, Alex Miller wrote: > > I think the reason for some things being volatile and some not is that we > want all of that stuff to be stateful. The volatile ones are making things > stateful by explicitly creating a stateful Clojure container for immutable > values. The ArrayLists are inherently stateful because they are mutable > (done for perf reasons). > > And then I think it is the responsibility of the transducing context to > lock appropriately to ensure visibility (if needed). > > On Wednesday, December 7, 2016 at 11:56:13 AM UTC-6, Alex Miller wrote: >> >> Transducers are expected to be invoked always in a single thread context. >> Everything in core maintains that by just running it in a single thread. >> core.async channels may be invoked from many threads, but only one at a >> time, and channel itself properly locks to make that state visible. That >> said, it does seem odd to me now that partition-by and partition-all have >> state that is not volatile. Thinking about it. >> >> On Wednesday, December 7, 2016 at 11:39:54 AM UTC-6, Léo Noel wrote: >>> >>> Hi ! >>> >>> I'm a bit confused about the official design rules for stateful >>> transducers and transducing contexts, especially about which one should be >>> in charge of memory visibility guarantees. >>> >>> The common practice seems to be using volatiles to hold state in >>> transducers (e.g distinct >>> <https://github.com/clojure/clojure/blob/c0326d2386dd1227f35f46f1c75a8f87e2e93076/src/clj/clojure/core.clj#L4940>) >>> >>> to ensure memory visibility. This usage of volatile is actually the >>> original motivation >>> <http://www.google.com/url?q=http%3A%2F%2Fdev.clojure.org%2Fjira%2Fbrowse%2FCLJ-1512&sa=D&sntz=1&usg=AFQjCNH67nkIeg6VMiMKsTloQt5Xjx7OUg> >>> >>> for adding it in core, and is explained further here >>> <http://stackoverflow.com/questions/31288608/what-is-clojure-volatile> >>> and here <http://insideclojure.org/2014/12/17/distinct-transducer/>. >>> However, some stateful tranducers use unsynchronized mutable state >>> instead (e.g partition-all >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fclojure%2Fclojure%2Fblob%2Fc0326d2386dd1227f35f46f1c75a8f87e2e93076%2Fsrc%2Fclj%2Fclojure%2Fcore.clj%23L7123&sa=D&sntz=1&usg=AFQjCNGKfWMID8XWgWwnghceHaood0NuqQ>), >>> >>> which goes against the previous volatile argument of "safely publish values >>> between threads". >>> In practice, partition-all works fine in e.g core.async because each >>> call to the (maybe stateful) step function is made inside the channel lock. >>> >>> So my question is, does it make any sense at all to have a transducing >>> context that does *not* enforce a *happens-before* order between >>> successive calls to the step function ? >>> If yes, I would be curious to see an example, and that would mean >>> partition-all is broken. >>> If no, what is the point of using volatiles for stateful transducers ? >>> As long as they are not exposed to the outside, unsynchronized variables >>> would do the job and should be slightly faster. >>> >>> Am I missing something ? >>> Thanks >>> >>> Leo >>> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
