This was not pitched to me by the folks involved as being our official buy-in for this test. They asked me to try and get our people involved and help. I helped get some of the tests working that were crashing in Fx. I thought I was doing something good to help us not get docked for crashing tests.
I can get my name off that list, I wasn't aware this would be used in the press this way. On Tue, Sep 25, 2012 at 7:56 AM, Justin Lebar <justin.le...@gmail.com>wrote: > Maybe this is naive of me, but I for one don't really believe in > tweaking benchmarks for the purposes of making Firefox look better. > > If we look bad in a benchmark, badmouthing it seems somehow more > gentlemanly than stacking it. :) > > Anyway, I filed a bug on getting rid of the microbenchmarks, which I > think we all agree is important, regardless of how that affects > Firefox's score. > > https://github.com/robohornet/robohornet/issues/67 > > On Tue, Sep 25, 2012 at 10:46 AM, Daniel Buchner <dbuch...@mozilla.com> > wrote: > > I've got private access to the RoboHornet repo and have been in > discussions > > with the PM that headed that effort up, do you all want to get some code > > committed to help our numbers out? > > > > - Daniel > > > > > > > > On Tue, Sep 25, 2012 at 6:51 AM, Justin Lebar <justin.le...@gmail.com> > > wrote: > >> > >> > (Can you hear that thud, thud, thud? It's the sound of me beating my > >> > head > >> > against my desk.) > >> > >> One of the intriguing things about this benchmark is that it's open > >> source, and they're committed to changing it over time. > >> > >> FWIW Paul Irish agrees the sieve is a bad test, although he doesn't > >> hate it to the extent you or i would think is deserved. > >> https://github.com/robohornet/robohornet/issues/20#issuecomment-8837867 > >> So maybe all hope is not lost. > >> > >> It's really laughable that they count the sieve as a test of > >> handlebars.js performance. Instead of, you know, actually using > >> handlebars.js. But I again wonder how much we can mold this into > >> something interesting. > >> > >> -Justin > >> > >> On Tue, Sep 25, 2012 at 1:41 AM, Nicholas Nethercote > >> <n.netherc...@gmail.com> wrote: > >> > On Mon, Sep 24, 2012 at 5:22 PM, Tim <timvk...@gmail.com> wrote: > >> >> So there's a new benchmark out, seemingly from google. > >> >> > >> >> It is designed to test performance in web app bottlenecks, especially > >> >> "DOM, <canvas> API methods, SVG". > >> >> > >> >> Paul Irish from Google's Chrome team is in charge of it. He blogged > on > >> >> it here: > >> >> > >> >> > >> >> > http://paulirish.com/2012/a-browser-benchmark-that-has-your-back-robohornet/ > >> > > >> > I'm horrified by this. Quoting my Hacker News comments > >> > (http://news.ycombinator.com/item?id=4567796): > >> > > >> >> Oh god, just when web people were starting to understand how to > create > >> >> good > >> >> benchmarks > >> >> (https://blog.mozilla.org/nnethercote/2012/08/24/octane-minus...), > >> >> now we're going back to 1980s microbenchmark hell. > >> >> > >> >> Doesn't anyone read Hennessy and Patterson any more? The best > >> >> benchmarks > >> >> are real apps, not crappy little microbenchmarks that measure a > single > >> >> thing. > >> >> > >> >> (Can you hear that thud, thud, thud? It's the sound of me beating my > >> >> head > >> >> against my desk.) > >> > > >> > Also, one of the tests is basically a no-op executed many times > >> > (https://bugzilla.mozilla.org/show_bug.cgi?id=793913#c7). > >> > > >> > Even better, there's a prime numbers calculation test, apparently to > >> > test "math". This is grimly hilarious: Hennessy and Patterson > >> > specifically cite the Sieve of Erastosthenes as an example of a toy > >> > (and thus crap) benchmark. Sigh. > >> > > >> > Daniel Buchner is apparently Mozilla's official representative on the > >> > RoboHornet "committee of JavaScript experts" > >> > (https://github.com/robohornet/robohornet/wiki/Committee-Membership). > >> > I don't know what his role is, but the thought of Mozilla officially > >> > blessing RoboHornet fills me with dread. > >> > > >> > While the suite may push us into some useful improvements, I worry > >> > that we'll end up implementing some stupid benchmarketing features > >> > that we will then carefully have to avoid regressing for the next 10 > >> > years. > >> > > >> > Nick > >> > _______________________________________________ > >> > dev-platform mailing list > >> > dev-platform@lists.mozilla.org > >> > https://lists.mozilla.org/listinfo/dev-platform > > > > > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
