Hi all,
I'm just getting started with Clojure, and I've got a bit of a
beginner's question regarding memory usage of lazy sequences.
I have an array of data that is too big to fit in memory, so I thought I
would be clever and process it in manageable-sized chunks. My instinct
was to do this using loop/recur using 'take' to bite off the next chunk,
and 'drop' to give me the remainder for the next recursive call. Here's
a contrived example:
;; Process the list one chunk at a time
(defn do-something [biglist]
(loop [rest (drop 1000 biglist)]
(when rest
(doall (take 1000 rest))
(recur (drop 1000 rest)))))
;; Lazily calculate our big data set and pass it along for processing
(do
(do-something
(map (fn [n] (make-array java.lang.Character 10240))
(range 0 100000)))
nil)
When I ran my code it very quickly ran out of memory and fell over.
After thinking about it for a while I've realised it must be because my
'do-something' function call is hanging on to the head of the list, so
as its elements are realised and cached it gradually eats up all my
memory.
Assuming my diagnosis is right, is there some sort of idiomatic way of
dealing with this sort of issue? I suppose that if the JVM allowed for
true tail recursion then this problem wouldn't arise?
Many thanks,
Mark
--
Mark Triggs
<[EMAIL PROTECTED]>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To post to this group, send email to clojure@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
