It doesn't seem very accidental: the namespace binding for 'trees' is retaining the head. You probably want to wrap it in a function:
(defn trees [] ...) -Per On Tue, Mar 23, 2010 at 9:40 PM, aria42 <ari...@gmail.com> wrote: > Hi, > I was experimenting with some code and I had an largish sequence I > did a doseq over. The memory hit the ceiling, which you expect since > even though the head isn't retained GC doesn't happen until you hit > your memory limit (is there a way to change that). Once it hit the > memory limit, 1.2 gigs in this case, the doseq slows to a halt. > > On the other hand if rather than make a long lazy seq, I do it > implicitly in the doseq itself (see FAST snippet below), the > performance is great. Is there anyway to get good performance using a > single lazy seq? Relevant snippets below. > > Thanks, Aria > > (defn lines > "get lines from gz file" > [#^String path] > (-> path > java.io.FileInputStream. > java.util.zip.GZIPInputStream. > java.io.InputStreamReader. > java.io.BufferedReader. > line-seq)) > > ; the tree-from-str does some processing, but doesn't have state > (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") > :when (not (empty? l)) > :let [[t _] (tree/tree-from-str l)]] > t)) > > ; SLOW: hits memory limit and becomes slow b/c of constant > ; GC hits > (doseq [t trees] > (println (str t))) > > ; FAST: low memory > (doseq [l (lines "/usr/local/corpora//NANC/003.gz") > :when (not (empty? l)) > :let [[t _] (tree/tree-from-str l)]] > (println (str t))) > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > > To unsubscribe from this group, send email to > clojure+unsubscribegooglegroups.com or reply to this email with the words > "REMOVE ME" as the subject. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
