Answering my own question, my grouping example failed because of the laziness of the expressions being computed in the pmap: the threads were not evaluating the expressions until they were evaluated (sequentially) by the main thread. Adding a doall gives the result I was hoping for.
On Sunday, October 13, 2013 4:10:16 PM UTC-7, Brian Craft wrote: > > I'm walking a seq of many millions of floats, encoding them for the > persistence layer, and getting sequence ids from the db. So, conceptually > there are two parts: the slow part, and the side-effecting part. Vaguely > like > > (map get-ids (map encode float-seq)) > > which is later reduced while writing to disk. In the get-ids step the > order matters. My first attempt to make the slow part parallel was to use > pmap, > > (map get-ids (pmap encode float-seq)) > > However that's actually slower. I expect this is because even though > "encode" is the bottleneck, it's still faster than the overhead of pmap. I > next tried pmap over groups of floats, a bit like > > (map get-ids (flatten (pmap #(map encode %) (partition-all 20000 > float-seq)))) > > (sorry for any typos, I'm just pseudo-coding here) This was still slower, > which surprised me. I understand the first pmap result, but this one is > puzzling to me. Even if I partition half the length of the seq (so in > theory it can run two threads, each of which will run five or six seconds), > it's no faster than map. Part of this seems to be the overhead of > creating more intermediate seqs. Perhaps I'm misunderstanding what's > happening during partition-all. > > Is there some obvious way to approach this scenario? I looked briefly at > the reducers library, however it was unclear to me how to deal with the > side-effecting portion of the operation. The second (fast) map operation > needs to be done in order. > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
