Hi all,
I've run into an issue with a lazy-seq either being prematurely realized
or having the head unwittingly retained. Reading chapter 5 in The Joy
of Clojure I realize that I am breaking one of the rules (page 150 in my
MEAP version): avoid binding your lazy sequences locally. That said, I
am using Clojure 1.2RC2, and the book states that as of 1.2 the compiler
is smarter about avoiding premature realization issues. I think I may
have stumbled upon a case where it is not handling (or is not able to).
Basically, the entire lazy-seq is being retained in memory when it is
bound locally and then realized in a different thread via a future.
Here is an example of the problem I am facing:
(deftype Foo [])
; The entire lazy-seq is held in memory resulting in an OutOfMemoryError.
(let [foos (take 10000000 (repeatedly #(Foo.)))]
@(future (doseq [foo foos] foo)))
I tried to reproduce this problem in different ways, but as far as I
could tell the above combination of local binding and realization in a
future is the only way the problem crops up. Here is a gist of some of
the other variants that did not cause the problem:
http://gist.github.com/513376
My co-worker, Tim Harper, also did some work with weak references
showing that the objects in the lazy-seq are not being GCed:
http://gist.github.com/510601
Could someone shed some light onto why I am seeing the entire lazy-seq
retained? FWIW, I ran across this issue when using fill-queue like so:
(defn map-queue
"Like map, but populated eagerly via fill-queue
(map-queue odd? (range 1 10) :queue-size 5) ;; will always calculate
5 ahead of the last used item from the lazy sequence"
[f coll & options]
(apply fill-queue (bound-fn [fill] (doseq [value coll] (fill (f
value)))) options))
So in my case coll is a lazy-seq that can't be retained in memory. Any
explanation of what I'm doing wrong and how to better go about my
problem would be greatly appreciated.
Thanks!
-Ben
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
