On Mon, Sep 24, 2012 at 5:22 PM, Tim <timvk...@gmail.com> wrote: > So there's a new benchmark out, seemingly from google. > > It is designed to test performance in web app bottlenecks, especially "DOM, > <canvas> API methods, SVG". > > Paul Irish from Google's Chrome team is in charge of it. He blogged on it > here: > > http://paulirish.com/2012/a-browser-benchmark-that-has-your-back-robohornet/
I'm horrified by this. Quoting my Hacker News comments (http://news.ycombinator.com/item?id=4567796): > Oh god, just when web people were starting to understand how to create good > benchmarks (https://blog.mozilla.org/nnethercote/2012/08/24/octane-minus...), > now we're going back to 1980s microbenchmark hell. > > Doesn't anyone read Hennessy and Patterson any more? The best benchmarks > are real apps, not crappy little microbenchmarks that measure a single thing. > > (Can you hear that thud, thud, thud? It's the sound of me beating my head > against my desk.) Also, one of the tests is basically a no-op executed many times (https://bugzilla.mozilla.org/show_bug.cgi?id=793913#c7). Even better, there's a prime numbers calculation test, apparently to test "math". This is grimly hilarious: Hennessy and Patterson specifically cite the Sieve of Erastosthenes as an example of a toy (and thus crap) benchmark. Sigh. Daniel Buchner is apparently Mozilla's official representative on the RoboHornet "committee of JavaScript experts" (https://github.com/robohornet/robohornet/wiki/Committee-Membership). I don't know what his role is, but the thought of Mozilla officially blessing RoboHornet fills me with dread. While the suite may push us into some useful improvements, I worry that we'll end up implementing some stupid benchmarketing features that we will then carefully have to avoid regressing for the next 10 years. Nick _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
