Thanks Laurent, but still behaving the same after change. On Thursday, January 9, 2014 10:59:26 AM UTC, Laurent PETIT wrote: > > Hello, > > In line 9, you retain the head in a var. > > Transform it into a function: > > (defn xml [] > (-> large-file > java.io/input-stream > clojure.data.xml/source-seq)) > > HTH, > > Laurent > > > > 2014/1/9 Peter Ullah <peter...@gmail.com <javascript:>> > >> >> Hi Everyone, >> >> I am trying to write some Clojure to process a large (310M) XML file, in >> doing so, the code seems to consume the heap. I wrote a smaller script to >> simplify the problem but I have now reduced it to a point where I am just >> confused with why the heap is being used up; I'm consuming a lazy sequence >> and not (that I am aware) maintaining a reference to the head. I have also >> removed Leiningen from the equation just in case the REPL was causing any >> references to be kept. >> >> I have managed to process the file as desired by increasing -Xmx and -Xms >> passed to the VM but it doesn't address the underlying problem and my >> confusion. >> >> Here is the code: https://gist.github.com/anonymous/8332220 >> >> Here are some details: >> >> - XML file has 6,321,720 lines. >> - Using clojure.data.xml/source-seq to create a lazy sequence, which >> translates into ~15,000,000 elements in the sequence >> - Keeping a counter to determine how far through the processing I >> have got before failure... (i.e. GC out of memory errors) >> - Increased available and initial heap size to such a point that the >> processing completes. >> - Used VisualVM to profile memory characteristics - see attachment >> >> >> My problem is in understanding why the used heap is increasing and not >> constant. >> >> Any help would be great. thanks. >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com<javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> > >
-- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
