On Thu, 1 Jul 2010 11:27:09 -0700 (PDT) j-g-faustus <johannes.fries...@gmail.com> wrote: > On Jul 1, 7:51 pm, Peter Schuller <peter.schul...@infidyne.com> wrote: > > > Is anyone using anything more sophisticated than clojure.core/time for > > > benchmarking clojure code? > Criterium, a benchmarking library for Clojure, seems pretty good: > http://github.com/hugoduncan/criterium
The author responded here. > Based on ideas in this article: > http://www.ibm.com/developerworks/java/library/j-benchmark1.html > > The stackoverflow question where I found it, thanks to Michal Marczyk: > http://stackoverflow.com/questions/3041299/how-to-benchmark-functions-in-clojure > > > Getting *accurate* results can be hard, even with a benchmarking > library. Criterium runs the code 60 times and does statistical > analysis on the results, but I can still get variations above +/-10% > from run to run in the REPL. > > I think benchmarking works best when > * starting a new run each time - i.e. from the command line, a "clean > slate" JVM > * having something that runs long enough to stabilize the JVM - > Criterium wants 1 min or more total runtime. > * running it more than once and checking that results are tolerably > consistent > * looking for differences in orders of magnitude rather than a couple > of percent more or less. Criterium (and the Java code based on the same ideas) has what looks like a serious problem for microbenchmarking: After figuring out how many times to execute the body to make it last a minute, it times the execution of the loop that does this. Which means the reported time includes loop overhead. When I tried the obvious solution (time an empty loop and subtract that value from the original) on top of time, it was sometimes generating negative values, so the empty loop is taking more time than the loop being timed. That's a pretty solid indication that timing the target code was lost in the noise of timing the loop overhead. Possibly trying this fix with all of criterium code to stabilize the JVM would help with the problem. I'd fix it (pretty easy) and try myself, but there's no 1.1 jar and the current sources require 1.2. If nothing else adding code to measure the empty loop and punting if the difference between that and the code loop is statistically insignificant would seem like a good idea. <mike -- Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
