Hi all,
I finally found an ideal use-case for pmap, however something very
strange seems to be happening after roughly 30 minutes of execution!
Ok so here is the scenario:
I've got 383 raw scienific papers (.txt) in directory that i'm grouping
using 'file-seq' and so I want to pmap a fn on each element of that seq
(each document). The fn takes a document and a dictionary and annotates
the document with terms found in the dictionary. Basically it uses regex
to tag any occurrences of words that exist in the dictionary. When
pmapping is finished, I should have a list of (annotated) strings that
will be processed serially (doseq) in order to produce a massive file
with all these strings separated by a new-line character (this is how
most adaptive feature generators expect the data to be).
So you can see, this is perfect for pmap and indeed it seems to be doing
extremely well but only for the first 240 papers roughly! all the cpus
are working hard but after approximately 30-40 min cpu utilisation and
overall performance seems to degrade quite a bit...For some strange
reason, 2 of my cores seem to refuse to do any work after these 240
papers which results in a really really slow process. When I start the
process it is going so fast that I cannot even read the output but as I
said after 30-40 min it is getting unbelievably slow! Had the
performance been stable I reckon I need less than 60 min in order to
annotate all 383 papers but with the current behaviour I have no choice
but to abort and restart it passing it the leftovers...
any ideas? are there any issues involved with creating that many futures?
Jim
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
