2010/6/24 William Wadsworth <will.wadsworth...@gmail.com>:
> Hi!
>
> On Jun 24, 12:16 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:
>> 2010/6/24 Laurent PETIT <laurent.pe...@gmail.com>:
>>
>>
>>
>> > Hi,
>>
>> > 2010/6/17 William Wadsworth <will.wadsworth...@gmail.com>:
>> >> Hi.
>>
>> >> I have just started learning Clojure and I am really enjoying
>> >> myself. However, I am still getting to grips with the workings of its
>> >> concurrency model, so please excuse me if my question seems too
>> >> obvious.
>>
>> >> I have some code that reads data from a file, parses it and generates
>> >> a CSV file. The flow is as follows:
>>
>> >> <code>
>> >> (use 'clojure.contrib.duck-streams)
>> >> (use 'clojure.contrib.pprint)
>>
>> >> (defn parse-line [s]
>> >> (cl-format nil "~{\"~A\"~^,~}" (into [] (.split s ";"))))
>>
>> >> (let [input-seq (for [line (read-lines "input.dat")]
>> >> (parse-line line))]
>> >> (write-lines "out.CSV" input-seq))
>> >> </code>
>>
>> >> Now the question:
>>
>> >> I would like to generate log entries for any malformed lines occurring
>> >> in the input file. Once of the ways might be to do this in parse-line
>> >> using clojure.contrib.loggingas follows:
>>
>> >> <code>
>> >> (use 'clojure.contrib.logging)
>>
>> >> (defn parse-line [s]
>> >> (let [cs (into [] (.split s ";"))]
>> >> (if (= (count cs) 5)
>> >> (cl-format nil "~{\"~A\"~^,~}" cs)
>> >> (error "Incorrect number of columns in file."))))
>> >> </code>
>>
>> >> Since FOR returns a lazy sequence of the results, is this function
>> >> safe? (Seeing that it is not side-effect free?)
>>
>> > You have to acknowledge that the calls to parse-line will not
>> > necessarily occur during the call to for, but maybe not before calling
>> > write-lines, maybe partly if chunked-seqs are in play (sequences
>> > preloading items in chunks of 32 items for example). And, if there is
>> > an exception that is not thrown in write-lines, your logs may miss the
>> > error logs for some of the remaining items not yet consumed by
>> > write-lines.
>>
>> s/if there is an exception that is not thrown in write-lines/if there is
>> an exception that *is* thrown in write-lines/
>>
>>
>
> That is clear. Thanks.
>
> While on the same topic: using read-lines on a large file (~80MB)
> results
> in an OutOfMemoryException. (I am running the JVM with -client rather
> than
> -server). To me, this is unexpected since read-lines lazily reads the
> file
> contents (and I also presume that the lines are not being cached in
> memory).
> Is there any way of being more memory efficient other than iteratively
> reading
> file contents using using .readLine?
You're probably holding the head. Can you share the code ?
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
