Slurp reads the entire file into memory. Maybe it is a combination of a)
the program taking up more of the heap in other parts as it runs and then
b) a particularly large file?
Is there a reason you can't process the files as a line-seq so you don't
have to load the entire thing into memory all at once?
Also, you could just pass a larger -Xmx value on the command line (or in
the lein java-opts) to make the heap larger.
On Fri, Apr 5, 2013 at 9:57 AM, Adrian Muresan
<muresan.adrian...@gmail.com>wrote:
> Hello everyone,
>
> I'm trying to parse a large number of small reports for some data and I'm
> doing this by repeatedly calling the following function (with a for) on
> each of the files:
>
> (defn get-rep [file]
> (let [report (with-open [rdr (io/reader file)](*slurp rdr*))
> reg #";\s+(\d+\.\d+) \s+; (\d+\.\d+) \s+;"
> match (re-matcher reg report)
> found (re-find match)
> fmax-85c (second found)
> rfmax-85c (nth found 2)
> found2 (re-find match)
> fmax-0c (second found2)
> rfmax-0c (nth found2 2)]
>
> {:fmax-85c fmax-85c
> :rfmax-85c rfmax-85c
> :fmax-0c fmax-0c
> :rfmax-0c rfmax-0c}))
>
> My problem is that after a while my program crashes with a jvm heap error,
> it crashes on the slurp line (in bold), although that probably doesn't mean
> much. The slurp line is the only one that I suspect since it's the biggest
> memory consumer in my program.
> Now I'm wondering what's going on and why don't the report files get freed
> from memory. I'm suspecting this might be caused by lazyness at some level,
> but I can't figure out where. Any ideas?
>
> Thanks!
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
