Maybe this is naive of me, but I for one don't really believe in tweaking benchmarks for the purposes of making Firefox look better.
If we look bad in a benchmark, badmouthing it seems somehow more gentlemanly than stacking it. :) Anyway, I filed a bug on getting rid of the microbenchmarks, which I think we all agree is important, regardless of how that affects Firefox's score. https://github.com/robohornet/robohornet/issues/67 On Tue, Sep 25, 2012 at 10:46 AM, Daniel Buchner <dbuch...@mozilla.com> wrote: > I've got private access to the RoboHornet repo and have been in discussions > with the PM that headed that effort up, do you all want to get some code > committed to help our numbers out? > > - Daniel > > > > On Tue, Sep 25, 2012 at 6:51 AM, Justin Lebar <justin.le...@gmail.com> > wrote: >> >> > (Can you hear that thud, thud, thud? It's the sound of me beating my >> > head >> > against my desk.) >> >> One of the intriguing things about this benchmark is that it's open >> source, and they're committed to changing it over time. >> >> FWIW Paul Irish agrees the sieve is a bad test, although he doesn't >> hate it to the extent you or i would think is deserved. >> https://github.com/robohornet/robohornet/issues/20#issuecomment-8837867 >> So maybe all hope is not lost. >> >> It's really laughable that they count the sieve as a test of >> handlebars.js performance. Instead of, you know, actually using >> handlebars.js. But I again wonder how much we can mold this into >> something interesting. >> >> -Justin >> >> On Tue, Sep 25, 2012 at 1:41 AM, Nicholas Nethercote >> <n.netherc...@gmail.com> wrote: >> > On Mon, Sep 24, 2012 at 5:22 PM, Tim <timvk...@gmail.com> wrote: >> >> So there's a new benchmark out, seemingly from google. >> >> >> >> It is designed to test performance in web app bottlenecks, especially >> >> "DOM, <canvas> API methods, SVG". >> >> >> >> Paul Irish from Google's Chrome team is in charge of it. He blogged on >> >> it here: >> >> >> >> >> >> http://paulirish.com/2012/a-browser-benchmark-that-has-your-back-robohornet/ >> > >> > I'm horrified by this. Quoting my Hacker News comments >> > (http://news.ycombinator.com/item?id=4567796): >> > >> >> Oh god, just when web people were starting to understand how to create >> >> good >> >> benchmarks >> >> (https://blog.mozilla.org/nnethercote/2012/08/24/octane-minus...), >> >> now we're going back to 1980s microbenchmark hell. >> >> >> >> Doesn't anyone read Hennessy and Patterson any more? The best >> >> benchmarks >> >> are real apps, not crappy little microbenchmarks that measure a single >> >> thing. >> >> >> >> (Can you hear that thud, thud, thud? It's the sound of me beating my >> >> head >> >> against my desk.) >> > >> > Also, one of the tests is basically a no-op executed many times >> > (https://bugzilla.mozilla.org/show_bug.cgi?id=793913#c7). >> > >> > Even better, there's a prime numbers calculation test, apparently to >> > test "math". This is grimly hilarious: Hennessy and Patterson >> > specifically cite the Sieve of Erastosthenes as an example of a toy >> > (and thus crap) benchmark. Sigh. >> > >> > Daniel Buchner is apparently Mozilla's official representative on the >> > RoboHornet "committee of JavaScript experts" >> > (https://github.com/robohornet/robohornet/wiki/Committee-Membership). >> > I don't know what his role is, but the thought of Mozilla officially >> > blessing RoboHornet fills me with dread. >> > >> > While the suite may push us into some useful improvements, I worry >> > that we'll end up implementing some stupid benchmarketing features >> > that we will then carefully have to avoid regressing for the next 10 >> > years. >> > >> > Nick >> > _______________________________________________ >> > dev-platform mailing list >> > dev-platform@lists.mozilla.org >> > https://lists.mozilla.org/listinfo/dev-platform > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
