A followup on these issues, after experimenting a bit with reducers, just for
anyone who may be interested:
- Naive use of reducers in place of seq functions, throughout my program, got
messy and led to some new problems with memory management, I think (or at least
to jerkier execution over time). I realized that reducers probably shouldn't be
used so indiscriminately.
- With more moderate use of reducers, and using fold, I do indeed see
performance improvements relative to sequential code and also relative to the
previous version of my code that used agents for concurrency. I don't actually
seem to be getting much benefit from the way that reducers avoid building
intermediate results -- all of the benefit appears to be due to using a
combination of r/fold and one r/map to process collection elements
concurrently. I've only done small numbers of tests and they all involve
randomness, but I do see improvements on the order of 1/3.
- The best behaving/looking approach for me seems to be to stick with lazy
sequence operations and aggressive de-lazifying for the bulk of my code
(possibly cleaning some things up by defining my own forv, repeatv, etc., or
possibly rewriting some of the core lazy sequence functions to not be lazy in
the first place), but to use r/fold and r/map for concurrency for my top-level
tasks. I'm getting the best results by using reducers only in one place, in my
definition of a concurrent mapping function, which I then use only for
top-level functions (never nested):
(defn pmapall
[f coll]
(if single-thread-mode
(doall (map f coll))
(r/fold 1 r/cat r/append! (r/map f coll))))
This replaces my previous agent-based version:
(defn pmapall
[f coll]
(if single-thread-mode
(doall (map f coll))
(let [agents (map #(agent % :error-handler
(fn [agnt except] (clojure.repl/pst except 1000)
(System/exit 0)))
coll)]
(dorun (map #(send % f) agents))
(apply await agents)
(doall (map deref agents)))))
The one that uses reducers is neater and seems to make my program run faster,
although I'm not really sure why.
In any event, I'm using reducers now only as an alternative concurrency
mechanism and solving my original problems by de-lazification.
-Lee
On Jun 4, 2014, at 7:21 PM, Lee Spector <lspec...@hampshire.edu> wrote:
>
> On Jun 4, 2014, at 1:29 PM, Timothy Baldridge <tbaldri...@gmail.com> wrote:
>
>> Although your original complaint was about clojure seqs being lazy. It
>> should be noted that reducers are also lazy down to the point of a fold or
>> reduce, so I'm not sure what you're really getting there. It wouldn't be
>> hard at all to write map, filter, remove, etc. in terms of list operations.
>> Perhaps that's what you're looking for? If I understand your original
>> complaint, you want map, filter, etc, to be eager.
>>
>
> True, my original concern was (and my main concern still is) to avoid crashes
> (in this case, stack overflow errors) that stem from unexpected (to me)
> consequences of laziness.
>
> Those problems could indeed be solved by rewriting map and filter etc, or
> (easier) by "wringing out the laziness" wherever it arises, e.g. by forcing
> everything to be a vector (or a list, as you suggest). It's a little
> cumbersome to do this everywhere, though, and I was wondering if there was an
> existing library or approach for this issue.
>
> Then, however, it was suggested that I could have my cake and another kind of
> cake too, by using reducers. The suggestion was that this could banish the
> laziness-related issues while also providing significant performance
> improvements. Sounds good to me! I'm still working on getting my current
> project to behave well using reducers, but it seems promising.
>
>>
>> On the other hand, time spent learning how lazy seqs work, and the caveats
>> involved will make it easier for you to understand the code produced by the
>> rest of the community. So perhaps that's the better option.
>
> I agree that it's important to understand lazy sequences, and I do think I
> understand the core concepts reasonably well (and I've implemented lazy
> evaluation in other languages, etc.). But I've also traced some really
> annoying and hard to find bugs to unexpected (and sometimes never fully
> explained) consequences of laziness, so I'd like to find the best ways to
> avoid it when I don't really want it. If that best way turns out to make
> things run faster too then that'd be fantastic.
>
> -Lee
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
