Thanks Alan, looking into it. By the way, fold function has [n combinef reducef coll] implementation, where n is number of elements collection is folded by. 512 is just the default.
On Saturday, April 27, 2013 3:51:39 PM UTC+3, Alan Busby wrote: > > On Sat, Apr 27, 2013 at 7:35 PM, Stanislav Yurin > <jus...@gmail.com<javascript:> > > wrote: > >> Actually, what I was trying to do, is to prototype multithreaded i/o >> operation via reducers. And then use fold to regulate number of concurrent >> operations. >> But now something tells me I am doing not very clever thing. >> > > I'm not entirely sure what you mean above, but I've been very happy using > reducers with I/O so far. > > Reducers just have a few tricks you need to be aware of first; > 1. You wont get parallel processing unless the input is a vector. > 2. Each thread gets ~512 elements each, so reducing a vector of 800 > elements will only use two cores. > 3. How you aggregate your final result can greatly impact performance. > (Ex. (r/fold +) is fast, fold-into-vec is slower, etc) > > I wrote a library to use reducers over text files (input) and have found > it to be invaluable for working with giant TSV files. > Link: https://github.com/thebusby/iota/ > > I often then use fold-into-lazy-seq to write the output back to a file. > Link: https://gist.github.com/thebusby/5472980 > > Hope this helps! ;) > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
