Look at clojure-contrib/duck_streams.clj (specifically the read-lines function)
I think it should be sufficiently lazy to do the job that you are looking for. (although I don't have any 100mb txt files to test with handy right now...) On Aug 31, 10:44 am, wangzx <wangzaixi...@gmail.com> wrote: > I just want to learn clojure by using it to parse log file and > generate reports. and one question is: for a large text file, can we > use it as a sequence effectively? for example, for a 100M log file, we > need to check each line for some pattern match. > > I just using the (line-seq rdr) but it will cause > OutOfMemoryException. > > demo code > > (defn buffered-reader [file] > (new java.io.BufferedReader > (new java.io.InputStreamReader > (new java.io.FileInputStream file)))) > > (def -reader (buffered-reader "test.txt")) > (filter #(= "some" %) -reader) > > even there is no lines match "some", the filter operation will cause > OutOfMemoryException. > > Is there other APIs like the Sequence but provide stream-like API? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
