Thanks Laurent, but still behaving the same after change. 

On Thursday, January 9, 2014 10:59:26 AM UTC, Laurent PETIT wrote:
>
> Hello, 
>
> In line 9, you retain the head in a var.
>
> Transform it into a function:
>
> (defn xml [] 
>    (-> large-file
>      java.io/input-stream
>      clojure.data.xml/source-seq))
>
> HTH,
>
> Laurent
>
>
>
> 2014/1/9 Peter Ullah <peter...@gmail.com <javascript:>>
>
>>
>> Hi Everyone, 
>>
>> I am trying to write some Clojure to process a large (310M) XML file, in 
>> doing so, the code seems to consume the heap. I wrote a smaller script to 
>> simplify the problem but I have now reduced it to a point where I am just 
>> confused with why the heap is being used up; I'm consuming a lazy sequence 
>> and not (that I am aware) maintaining a reference to the head. I have also 
>> removed Leiningen from the equation just in case the REPL was causing any 
>> references to be kept.
>>
>> I have managed to process the file as desired by increasing -Xmx and -Xms 
>> passed to the VM but it doesn't address the underlying problem and my 
>> confusion.
>>
>> Here is the code: https://gist.github.com/anonymous/8332220
>>
>> Here are some details:
>>
>>    - XML file has 6,321,720 lines.
>>    - Using clojure.data.xml/source-seq to create a lazy sequence, which 
>>    translates into ~15,000,000 elements in the sequence
>>    - Keeping a counter to determine how far through the processing I 
>>    have got before failure... (i.e. GC out of memory errors)
>>    - Increased available and initial heap size to such a point that the 
>>    processing completes.
>>    - Used VisualVM to profile memory characteristics - see attachment 
>>    
>>
>> My problem is in understanding why the used heap is increasing and not 
>> constant.
>>
>> Any help would be great. thanks.
>>
>>  -- 
>> -- 
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com<javascript:>
>> Note that posts from new members are moderated - please be patient with 
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>

-- 
-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Reply via email to