I don't see how the loop is relevant here, at least if the same benchmarking function is used for all the benchmarks you're doing, it should make a difference then since the overhead is the same.
On Jul 1, 2010, at 9:44 PM, Mike Meyer wrote: > On Thu, 1 Jul 2010 11:27:09 -0700 (PDT) > j-g-faustus <johannes.fries...@gmail.com> wrote: >> On Jul 1, 7:51 pm, Peter Schuller <peter.schul...@infidyne.com> wrote: >>>> Is anyone using anything more sophisticated than clojure.core/time for >>>> benchmarking clojure code? >> Criterium, a benchmarking library for Clojure, seems pretty good: >> http://github.com/hugoduncan/criterium > > The author responded here. > >> Based on ideas in this article: >> http://www.ibm.com/developerworks/java/library/j-benchmark1.html >> >> The stackoverflow question where I found it, thanks to Michal Marczyk: >> http://stackoverflow.com/questions/3041299/how-to-benchmark-functions-in-clojure >> >> >> Getting *accurate* results can be hard, even with a benchmarking >> library. Criterium runs the code 60 times and does statistical >> analysis on the results, but I can still get variations above +/-10% >> from run to run in the REPL. >> >> I think benchmarking works best when >> * starting a new run each time - i.e. from the command line, a "clean >> slate" JVM >> * having something that runs long enough to stabilize the JVM - >> Criterium wants 1 min or more total runtime. >> * running it more than once and checking that results are tolerably >> consistent >> * looking for differences in orders of magnitude rather than a couple >> of percent more or less. > > Criterium (and the Java code based on the same ideas) has what looks > like a serious problem for microbenchmarking: > > After figuring out how many times to execute the body to make it last > a minute, it times the execution of the loop that does this. Which > means the reported time includes loop overhead. > > When I tried the obvious solution (time an empty loop and subtract > that value from the original) on top of time, it was sometimes > generating negative values, so the empty loop is taking more time than > the loop being timed. That's a pretty solid indication that timing the > target code was lost in the noise of timing the loop overhead. > > Possibly trying this fix with all of criterium code to stabilize the > JVM would help with the problem. I'd fix it (pretty easy) and try > myself, but there's no 1.1 jar and the current sources require 1.2. > > If nothing else adding code to measure the empty loop and punting if > the difference between that and the code loop is statistically > insignificant would seem like a good idea. > > <mike > -- > Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html > Independent Network/Unix/Perforce consultant, email for more information. > > O< ascii ribbon campaign - stop html mail - www.asciiribbon.org > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
