On Jan 22, 12:53 am, "Mark H." <mark.hoem...@gmail.com> wrote:
> On Jan 21, 5:21 pm, e <evier...@gmail.com> wrote:
>
> > I would think it would be useful to have something exactly like a stream but
> > that allowed as many iterators as you like but that a mutex prevented any
> > two from consuming the same piece of information.
>
> That might be useful for something, but it's hard to make correct in
> all cases. The reason is that both stream elements and connections
> between stream elements are mutable and shared. Here follows an
> example.
>
> Imagine iterating over a singly-linked list, where you delete each
> node (explicitly, not by losing the reference) _and_ the link to the
> next node right after you get the next node. (For example, a
> directory of files might be implemented as a singly-linked list, and
> you want to delete all the files _and_ the directory itself and print
> all the files deleted, by only making one pass over the directory.)
> You could implement this as a generator which is a closure over a let-
> bound Java object (I've made up some utility function names below):
>
> (defn make-dir-gen [dirname]
> (let [dirstart (directory-list dirname)]
> (fn [eos]
> (loop [dirent dirstart]
> (if (dir-empty? dirent)
> eos
> (let [link (next-link dirent)]
> next (next-entry link)]
> (delete-file! dirent)
> (delete-link! link)
> (recur next)))))))
>
> Now imagine two threads T1 and T2 accessing this generator at the same
> time. Suppose they reach the same node at the same time, and suppose
> that you've protected file deletion and link deletion each
> individually with a mutex (and forbade multiple deletions silently).
> T1 might delete the file and the link and get to "recur next" before
> T2 can call "(next-entry link)". Now T1 goes on but T2 is stuck:
> "link" is _gone_ so "(next-entry link)" goes nowhere.
>
> Protecting the data for access by multiple threads has to be the
> responsibility of the person implementing the generator function,
> because a generator function can have arbitrary side effects and
> iterate over arbitrary mutable data structures with constantly
> changing topology. (Imagine a generator that goes five times around a
> circular list, then breaks the circular link and splices the list into
> another list, leaving a list fragment floating.) It's too hard to try
> to imagine a streams implementation that could "automatically make
> everything thread-safe," or even to imagine what "thread-safe" means
> in the context of general mutable data structures.
>
Those are important things to think about.
There are, in fact, thread semantics for the streams mechanism, as for
the rest of Clojure. Currently, I've made it such that the stream/iter/
seq combination ensures serialized access to the generator, allowing
use of generators over otherwise unsynchronized multi-step activities.
Obviously, stream-backed seqs are safe, being persistent and
immutable. but it is also currently the case that the stream iter is
MT safe as well. All calls to next! on an iter are atomic re: its
generator, i.e. no 2 threads can be in the generator code at once.
That means that, within a private consumer, you could fire up multiple
threads that share the same iter and coordinate to produce a result,
and return the result or a stream upon it.
Used correctly, this is a powerful feature. However, I'm reluctant to
expose/guarantee it, lest people start using iters as connection
points and invalidating the whole stream protection model. A well-
written API should take/return streams/seqs, never iters, so MT use of
an iter should always be an implementation detail of a single stream
step. But I can't enforce that.
For those looking to experiment with such things, feel free to try it
now and provide feedback.
Thanks,
Rich
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
