Well I reply to myself:
with pmap there´s no need to explicitly calculate the batches
The final version of the code is:
(defn parallel-count [iota-file]
(reduce (partial merge-with +)
(pmap (fn [a-subcollection]
(frequencies (words a-subcollection)))
iota-file)))
which is not that dramatically far from the the vanilla version.
Right ?
2013/5/1 Catonano <caton...@gmail.com>
> Meikel,
>
> thank you for your reply
>
>
> 2013/4/30 Meikel Brandmeyer (kotarak) <m...@kotka.de>
>
> Hello,
>>
>> your function does not follow the contract of fold. What you provided is
>> basically the reduce function, but it doesn't work together with fold. For
>> fold you need to merge the different maps you created in the subtasks. So
>> the combine function must look different.
>
>
> Admittedly I had missed the fact that fold expects a reducing function AND
> a combining function.
>
> But the worst thing is that I modified my-frequencies so that it could be
> called with no arguments, but who was going to call i like that ? Not my
> code and not anyone else !!
>
> What was I thinking ?
>
> I´m sorry for the silly question, I was just confused.
>
> Thanks for your help
>
> Additionally you cannot use transients at the moment with fold.
>>
>
> That´s a pity. I suppose I could explicitly split the input vector in
> parts and then call pvalues with a function implementing my word counting
> on each subpart. And then merging the results from each subpart
>
> But the reducers idiom is more beautiful because the modifications to the
> traditional map reduce version would be smaller, with that you haven´t got
> to explicitly split the input and call a specielized version of map.
>
> Oh well...
>
> It was just an exercise after all...
>
> Thanks again Meikel !
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
