Oops -- something was wrong with my benchmarks, and my "improvements on the order of 1/3" was wrong. I still see improvements with r/fold as compared to my agent-based approach, but the difference now appears to be only something like 1/20.
-Lee On Jun 5, 2014, at 7:19 PM, Lee Spector <lspec...@hampshire.edu> wrote: > > A followup on these issues, after experimenting a bit with reducers, just for > anyone who may be interested: > > - Naive use of reducers in place of seq functions, throughout my program, got > messy and led to some new problems with memory management, I think (or at > least to jerkier execution over time). I realized that reducers probably > shouldn't be used so indiscriminately. > > - With more moderate use of reducers, and using fold, I do indeed see > performance improvements relative to sequential code and also relative to the > previous version of my code that used agents for concurrency. I don't > actually seem to be getting much benefit from the way that reducers avoid > building intermediate results -- all of the benefit appears to be due to > using a combination of r/fold and one r/map to process collection elements > concurrently. I've only done small numbers of tests and they all involve > randomness, but I do see improvements on the order of 1/3. > > - The best behaving/looking approach for me seems to be to stick with lazy > sequence operations and aggressive de-lazifying for the bulk of my code > (possibly cleaning some things up by defining my own forv, repeatv, etc., or > possibly rewriting some of the core lazy sequence functions to not be lazy in > the first place), but to use r/fold and r/map for concurrency for my > top-level tasks. I'm getting the best results by using reducers only in one > place, in my definition of a concurrent mapping function, which I then use > only for top-level functions (never nested): > > (defn pmapall > [f coll] > (if single-thread-mode > (doall (map f coll)) > (r/fold 1 r/cat r/append! (r/map f coll)))) > > This replaces my previous agent-based version: > > (defn pmapall > [f coll] > (if single-thread-mode > (doall (map f coll)) > (let [agents (map #(agent % :error-handler > (fn [agnt except] (clojure.repl/pst except > 1000) (System/exit 0))) > coll)] > (dorun (map #(send % f) agents)) > (apply await agents) > (doall (map deref agents))))) > > The one that uses reducers is neater and seems to make my program run faster, > although I'm not really sure why. > > In any event, I'm using reducers now only as an alternative concurrency > mechanism and solving my original problems by de-lazification. > > -Lee > > > > On Jun 4, 2014, at 7:21 PM, Lee Spector <lspec...@hampshire.edu> wrote: > >> >> On Jun 4, 2014, at 1:29 PM, Timothy Baldridge <tbaldri...@gmail.com> wrote: >> >>> Although your original complaint was about clojure seqs being lazy. It >>> should be noted that reducers are also lazy down to the point of a fold or >>> reduce, so I'm not sure what you're really getting there. It wouldn't be >>> hard at all to write map, filter, remove, etc. in terms of list operations. >>> Perhaps that's what you're looking for? If I understand your original >>> complaint, you want map, filter, etc, to be eager. >>> >> >> True, my original concern was (and my main concern still is) to avoid >> crashes (in this case, stack overflow errors) that stem from unexpected (to >> me) consequences of laziness. >> >> Those problems could indeed be solved by rewriting map and filter etc, or >> (easier) by "wringing out the laziness" wherever it arises, e.g. by forcing >> everything to be a vector (or a list, as you suggest). It's a little >> cumbersome to do this everywhere, though, and I was wondering if there was >> an existing library or approach for this issue. >> >> Then, however, it was suggested that I could have my cake and another kind >> of cake too, by using reducers. The suggestion was that this could banish >> the laziness-related issues while also providing significant performance >> improvements. Sounds good to me! I'm still working on getting my current >> project to behave well using reducers, but it seems promising. >> >>> >>> On the other hand, time spent learning how lazy seqs work, and the caveats >>> involved will make it easier for you to understand the code produced by the >>> rest of the community. So perhaps that's the better option. >> >> I agree that it's important to understand lazy sequences, and I do think I >> understand the core concepts reasonably well (and I've implemented lazy >> evaluation in other languages, etc.). But I've also traced some really >> annoying and hard to find bugs to unexpected (and sometimes never fully >> explained) consequences of laziness, so I'd like to find the best ways to >> avoid it when I don't really want it. If that best way turns out to make >> things run faster too then that'd be fantastic. >> >> -Lee -- Lee Spector, Professor of Computer Science Cognitive Science, Hampshire College 893 West Street, Amherst, MA 01002-3359 lspec...@hampshire.edu, http://hampshire.edu/lspector/ Phone: 413-559-5352, Fax: 413-559-5438 -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
