I know the principal Google PM, Alex K, who heads up RoboHornet - he has been extremely helpful with our Web Components initiative. I believe he had good intentions with RoboHornet, and his personal posts (and those of Paul Irish) did not claim Google had Mozilla's official organizational support for the benchmark/tests (it was supposed to be by-devs-for-devs)
I believe Alex would be very amenable to changes to the tests, general benchmark strategy, and our contributions. I can help make sure our concerns are addressed - I'd be more than happy to do so. - Daniel On Sep 25, 2012 5:11 PM, "Nicholas Nethercote" <n.netherc...@gmail.com> wrote: > On Wed, Sep 26, 2012 at 9:37 AM, Kevin Gadd <kevin.g...@gmail.com> wrote: > > Has anyone reached out directly to the guys running the benchmark to > > try and get a feel for what their motivations are here? They may > > actually be willing to do the work necessary to make this a good > > benchmark. > > Justin filed https://github.com/robohornet/robohornet/issues/67. The > most relevant part of their response was this: > > 'I just added an answer in the FAQ about "micro-benchmarks": > https://github.com/robohornet/robohornet/wiki/FAQ . In summary, > although these are small, succinct tests, our hope is that they don't > have the failings normally associated with "microbenchmarks" because > they are motivated by specific pain points and the tests can evolve > with browsers to ensure that they continue testing what we want them > to test. Of course, there are problems with the specific tests today, > which we want to improve as quickly as possible now that we have good > feedback on them.' > > It's wishful thinking, IMO. Even flawless microbenchmarks are a poor > foundation for a benchmark suite. So unless they're willing to throw > out everything they have and start from scratch, things won't get much > better. > > Interestingly, Microsoft has strongly criticized RoboHornet on the > exact same grounds: > > http://news.cnet.com/8301-1023_3-57519568-93/life-beyond-javascript-googles-abuzz-over-robohornet-test/ > . > They created something called "RoboHornet Pro", which runs RoboHornet > in parallel with a weird graphical thing that looks likethe one from > "The Matrix". I *think* it's meant to be something of a joke; if you > ignore it and just read what they've said, the message is clear, e.g.: > "Like all micro-benchmarks, RoboHornet is a lab test that only focuses > on specific aspects of browser performance" and "Synthetic benchmarks > are never the right choice for the customer." > > The RoboHornet people responded to Microsoft's comments with: > > "Feedback like this is extremely valuable in this early alpha phase of > the project, but RoboHornet aims to be an independent benchmark driven > by Web developers, not a marketing tool for browser vendors. We're > eager to work with people from every part of the Web ecosystem to > accomplish this goal." > > Which doesn't sound like they're serious about making big changes. > > Also from that article: 'Google declined to comment, saying the > project is an "independent benchmark"' > > Nick > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
