As I mentioned, you want to use the Streaming (or Incremental) module.
As the program now stands the call to `decode` causes 1.5 GB of CSV
data to be read as a `Vector (Vector Int)` before any encoding starts.
-- Johan
On Wed, Aug 21, 2013 at 1:09 PM, Justin Paston-Cooper
wrote:
> Dear All,
>
>
Dear All,
I now have some example code. I have put it on: http://pastebin.com/D9MPmyVd.
vectorBinner is simply of type Vector Int -> Int. I am inputting a 1.5GB
CSV on stdin, and would like vectorBinner to run over every single record,
outputting results as computed, thus running in constant memo
You can use the Incremental or Streaming modules to get more fine
grained control over when new parsed records are produced.
On Thu, Jul 25, 2013 at 11:02 AM, Justin Paston-Cooper
wrote:
> I hadn't yet tried profiling the programme. I actually deleted it a few days
> ago. I'm going to try to get
I hadn't yet tried profiling the programme. I actually deleted it a few
days ago. I'm going to try to get something new running, and I will report
back. On a slightly less related track: Is there any way to use cassava so
that I can have pure state and also yield CSV lines while my computation is
r
On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari wrote:
> Justin Paston-Cooper writes:
>
>> Dear All,
>>
>> Recently I have been doing a lot of CSV processing. I initially tried to
>> use the Data.Csv (cassava) library provided on Hackage, but I found this to
>> still be too slow for my needs. In the
Justin Paston-Cooper writes:
> Dear All,
>
> Recently I have been doing a lot of CSV processing. I initially tried to
> use the Data.Csv (cassava) library provided on Hackage, but I found this to
> still be too slow for my needs. In the meantime I have reverted to hacking
> something together in
