HI, I come with an update.
Apart from the slow startup time issue, which I can resolve by resuing JVM or enlarge the splits, there's another factor that influence performance a lot, GC. It turns out Clojure will generate a lot of transient objects, which causes frequent young gc, over five times per second. On contrary, pure Java counterpart map-red job will gc only once per second. Also I found a very informative article on this particular issue, I believe many of you have read it: http://berlinbrowndev.blogspot.com/2009/07/jvm-notebook-basic-clojure-java-and-jvm.html So what I am asking is - is there any one use clojure-hadoop in production? How's the speed? GC frequency? I ran a simple map-red job to look for a substring in input line (use .contains) and increment a counter (swap! cnt inc). The result is, processing 4GB data in one mapper, Clojure uses 5'37" and pure Java uses 4'42", it's one minute difference. Although the Clojure code is less and cooler, but the performance loss is not quite worthwhile, IMHO. So I hope here would be some realistic cases that say Clojure is suitable to writing hadoop job. Thanks. On Friday, April 26, 2013 6:05:33 PM UTC+8, Ji Zhang wrote: > > Hi, > > I'm writing map-reduce job with Clojure, yet to find that it seems to be > much slower than a Jave job. > > So I write a simple test case, and upload to gist: > https://gist.github.com/jizhang/5466149 > > At the end of code, there is execution outputs, here are some significant > stats: > > Average time taken by Map tasks: Java 7sec, Clojure 19sec > CPU time spent (ms): Java 244,000, Clojure 1,145,440 > > I'm wondering what slows down the Clojure written map-reduce job. Am I > using it wrong, or it's just an inappropriate senario. > > Any thoughts will be great. Thanks! > > Jerry > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
