On Wed, Sep 26, 2012 at 9:37 AM, Kevin Gadd <kevin.g...@gmail.com> wrote: > Has anyone reached out directly to the guys running the benchmark to > try and get a feel for what their motivations are here? They may > actually be willing to do the work necessary to make this a good > benchmark.
Justin filed https://github.com/robohornet/robohornet/issues/67. The most relevant part of their response was this: 'I just added an answer in the FAQ about "micro-benchmarks": https://github.com/robohornet/robohornet/wiki/FAQ . In summary, although these are small, succinct tests, our hope is that they don't have the failings normally associated with "microbenchmarks" because they are motivated by specific pain points and the tests can evolve with browsers to ensure that they continue testing what we want them to test. Of course, there are problems with the specific tests today, which we want to improve as quickly as possible now that we have good feedback on them.' It's wishful thinking, IMO. Even flawless microbenchmarks are a poor foundation for a benchmark suite. So unless they're willing to throw out everything they have and start from scratch, things won't get much better. Interestingly, Microsoft has strongly criticized RoboHornet on the exact same grounds: http://news.cnet.com/8301-1023_3-57519568-93/life-beyond-javascript-googles-abuzz-over-robohornet-test/. They created something called "RoboHornet Pro", which runs RoboHornet in parallel with a weird graphical thing that looks likethe one from "The Matrix". I *think* it's meant to be something of a joke; if you ignore it and just read what they've said, the message is clear, e.g.: "Like all micro-benchmarks, RoboHornet is a lab test that only focuses on specific aspects of browser performance" and "Synthetic benchmarks are never the right choice for the customer." The RoboHornet people responded to Microsoft's comments with: "Feedback like this is extremely valuable in this early alpha phase of the project, but RoboHornet aims to be an independent benchmark driven by Web developers, not a marketing tool for browser vendors. We're eager to work with people from every part of the Web ecosystem to accomplish this goal." Which doesn't sound like they're serious about making big changes. Also from that article: 'Google declined to comment, saying the project is an "independent benchmark"' Nick _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
