> >> It seems like another alternative might be to run Octane in Talos, > >> instead of v8_7. > >> > >> It seems like Talos has two advantages over AWFY (correct me if I'm wrong): > >> > >> 1. Easy for developers to schedule jobs via try (maybe less of a concern > >> with a benchmark like this, where I suspect results are more > >> reproducible locally?) > > > > I believe there was talk of adding try support for AWFY (there already is > > for AWSY). Of course that's not actually done yet, I just want to point out > > it's not particularly hard and AWSY's version could be adapted rather > > easily. > >
Running Octane in Talos would be useful, but we would be duplicating efforts. While that is not a bad thing, will we get value from it. We do get value in self serving support with try and tools like mozci for backfilling and retriggering. The issue in the bug also points out that the developers who care about Octane use AWFY and already detect these regressions before talos does (or at least a sheriff finds it and files a bug). This specific topic of upgrading V8 to Octane for Talos should be discussed in bug 1174671. > Talos already runs on non-virtualized hardware. I don't see any inherent > reason we couldn't rework AWSY as a Talos test. In general it feels to > me like we should be running performance tests on relops-supported > infrastructure where possible, as opposed to adhoc systems. I would agree that the more we can run on managed systems the better. While all Talos jobs are run on non-virtualized hardware today, we do run on a shared pool of hardware with the unittests. One difference between AWFY and Talos is that the numbers are so much more stable, even in the browser version (AWFY runs a js shell as well as a browser). I believe this is attributed to the type of hardware, the environment, or the fact that a specific test is run on a specific machine. > > In general it would be great if we could consolidate the various perf tests > > (AWFY, AWSY, Talos, Raptor, etc) under one umbrella (at least from an end > > user perspective). So you could go to trychooser and choose a "Perf" option > > that would have various subsets like: "JS Engine", "Memory Usage", "Layout > > Latency", "Mobile Launch Time", etc. > > This is a worthwhile goal- Simplifying the interface to the tools over the next few quarters to allow for common sheriffing, and self serving will make big strides. > > 1. It assumes that all test machines of a particular class will be > uniform, at least per test. For example, Autophone tracks the > performance of something like 9 different Android devices seperately > (see: http://phonedash.mozilla.org/) -- that's not something Perfherder > was designed to do. As mentioned earlier in this comment AWFY runs the same test on the same machine- the numbers are more reliable, but there is no further evidence that is the cause of the noise in Talos, I suspect it is a factor. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
