Look at clojure-contrib/duck_streams.clj

(specifically the read-lines function)

I think it should be sufficiently lazy to do the job that you are
looking for. (although I don't have any 100mb txt files to test with
handy right now...)

On Aug 31, 10:44 am, wangzx <wangzaixi...@gmail.com> wrote:
> I just want to learn clojure by using it to parse log file and
> generate reports. and one question is: for a large text file, can we
> use it as a sequence effectively? for example, for a 100M log file, we
> need to check each line for some pattern match.
>
> I just using the (line-seq rdr) but it will cause
> OutOfMemoryException.
>
> demo code
>
> (defn buffered-reader [file]
>         (new java.io.BufferedReader
>                 (new java.io.InputStreamReader
>                         (new java.io.FileInputStream file))))
>
> (def -reader (buffered-reader "test.txt"))
> (filter #(= "some" %) -reader)
>
> even there is no lines match "some", the filter operation will cause
> OutOfMemoryException.
>
> Is there other APIs like the Sequence but provide stream-like API?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to