Re: Parallel words frequency ranking

2008-12-31 Thread Konrad Hinsen
On 30.12.2008, at 22:24, Mark H. wrote: > On Dec 30, 9:18 am, Mibu wrote: >> In an ideal world, standard functions like map, sort, reduce, filter, >> etc. would know when to parallelize on their own, or even better, the >> compiler will do it for them. > > The former is easier than the latter ;-

Re: Parallel words frequency ranking

2008-12-30 Thread Mark H.
On Dec 28, 7:50 pm, Piotr 'Qertoip' Włodarek wrote: > On 38MB file it takes 28s, compared to 38s of similar but sequential > version. Another good thing is to make a simple performance model, perhaps aided by timings of individual components, before you start parallelizing everything ;-) How lo

Re: Parallel words frequency ranking

2008-12-30 Thread Mark H.
On Dec 30, 9:18 am, Mibu wrote: > In an ideal world, standard functions like map, sort, reduce, filter, > etc. would know when to parallelize on their own, or even better, the > compiler will do it for them. The former is easier than the latter ;-) Even the smartest autoparallelizing compilers

Re: Parallel words frequency ranking

2008-12-30 Thread Mibu
In an ideal world, standard functions like map, sort, reduce, filter, etc. would know when to parallelize on their own, or even better, the compiler will do it for them. Meanwhile, I tried playing with the parallel lib ( http://clojure.org/other_libraries ) which has functions like preduce, psor

Re: Parallel words frequency ranking

2008-12-30 Thread Piotr 'Qertoip' Włodarek
On Dec 29, 7:02 am, Chouser wrote: > > (defn split-string-in-two [s] > >  (let [chunk-size (quot (count s) 2)] > >    [(subs s 0 chunk-size), (subs s chunk-size)])) > > Might this cut a word in half and produce (slightly) incorrect > results? True, I decided to let it be for the sake of simplici

Re: Parallel words frequency ranking

2008-12-29 Thread Mark H.
On Dec 28, 7:50 pm, Piotr 'Qertoip' Włodarek wrote: > Following my recent adventure with words ranking, here's the parallel > version: > > ... > (defn parallel-top-words [in-filepath out-filepath] >   (let [string  (slurp in-filepath) 'slurp' just reads the whole file in at once as a string, rig

Re: Parallel words frequency ranking

2008-12-28 Thread Chouser
On Sun, Dec 28, 2008 at 10:50 PM, Piotr 'Qertoip' Włodarek wrote: > > Following my recent adventure with words ranking, here's the parallel > version: > > (use 'clojure.contrib.duck-streams) Thanks for including your 'use' line -- that's so much better than leaving it implied. Please also consid