I've had exactly this problem trying to use reducers over a large file that wouldn't fit in memory.
I tried iota, but had the issue that it was still scanning and memory mapping the entire file before it would start doing anything (pulling the whole thing through ram and taking a fair few minutes). How would feeding a line-seq into this compare to iota? And how would that compare to a version of iota tweaked to work in a slightly less eager fashion? Adam On Tuesday, March 12, 2013 1:34:43 PM UTC, Paul Butcher wrote: > > So this turned out to be pretty easy. I've implemented a function called > "foldable-seq" that takes a lazy sequence and turns it into something that > can be folded in parallel. I've checked an example program that uses it to > count words in a Wikipedia XML dump into GitHub: > > https://github.com/paulbutcher/foldable-seq > > The code for foldable-seq is here: > > > https://github.com/paulbutcher/foldable-seq/blob/master/src/wordcount/reducers.clj#L60 > > On my 4-core MacBook Pro, I see a 40 second runtime without parallel-seq, > 13 seconds with. > > -- > paul.butcher->msgCount++ > > Snetterton, Castle Combe, Cadwell Park... > Who says I have a one track mind? > > http://www.paulbutcher.com/ > LinkedIn: http://www.linkedin.com/in/paulbutcher > MSN: pa...@paulbutcher.com <javascript:> > AIM: paulrabutcher > Skype: paulrabutcher > > On 11 Mar 2013, at 13:38, Paul Butcher <pa...@paulbutcher.com<javascript:>> > wrote: > > On 11 Mar 2013, at 11:00, Marko Topolnik <marko.t...@gmail.com<javascript:>> > wrote: > > The idea is to transform into a lazy sequence of eager chunks. That > approach should work. > > > Exactly. Right - I guess I should put my money where my mouth is and see > if I can get it working... > > -- > paul.butcher->msgCount++ > > Snetterton, Castle Combe, Cadwell Park... > Who says I have a one track mind? > > http://www.paulbutcher.com/ > LinkedIn: http://www.linkedin.com/in/paulbutcher > MSN: pa...@paulbutcher.com <javascript:> > AIM: paulrabutcher > Skype: paulrabutcher > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
