2017-06-26 19:40 GMT+02:00 Ehsan Akhgari <ehsan.akhg...@gmail.com>: > On 06/26/2017 09:52 AM, Armen Zambrano Gasparnian wrote: >> >> On Friday, 23 June 2017 10:05:42 UTC-4, Ehsan Akhgari wrote: >>> >>> On Fri, Jun 23, 2017 at 4:19 AM, Chris Peterson <cpeter...@mozilla.com> >>> wrote: >>> >>>> >>>> On 6/23/17 12:17 AM, Ehsan Akhgari wrote: >>>> >>>> But to speak of a more direct measurement of performance, let's look at >>>> our progress on Speedometer V2 >>>> >>>> <https://github.com/WebKit/webkit/tree/master/PerformanceTests/Speedometer>. >> >> Ehsan, were you comparing against http://speedometer2.benj.me? > > I used this instance for these measurements. >> >> Or this http://browserbench.org/Speedometer/? > > This is still Speedometer V1, FWIW.
Both websites indicate Speedometer V1 but are currently running Speedometer V2: - The hosted version (under benj.me) is a direct clone of the Webkit repository PerformanceTests/Speedometer with changes for running under AWFY. I've manually updated the title's version number. - For browserbench.org, see https://github.com/WebKit/webkit/commit/ae7c2b286458944f61b33eea85fb89f263245b52#diff-0858c771e4c688cca20fc04d800816aa . I've opened a pull request for updating their version number as well. > >> Or using the AWFY code? (It uses local proxy) >> >>>> Today, I measured our progress so far on this benchmark by comparing >>>> Firefox 53, 54, 55.0b3 (latest beta as of this writing) and the latest >>>> Nightly, all x64 builds, on the reference hardware >>>> <https://www.amazon.com/dp/B01K1IO3QW>. This is the result (numbers are >>>> the reported benchmark score, higher is better): >>>> >>>> [image: Speedometer improvements] >>>> >>>> >>>> How do these Speedometer V2 scores map to the results on AWFY? AWFY >>>> shows >>>> many Speedometer sub-tests, but no score in the range of 70.21. AWFY >>>> machine #36 is the reference hardware. >>>> >>>> >>>> https://arewefastyet.com/#machine=36&view=breakdown&suite=speedometer-misc >>>> >>> Armen has been investigating the difference. The Speedometer benchmark >>> is >>> extremely sensitive to anything else that is going on on the machine at >>> the >>> time you are running the tests, see for example >>> https://bugzilla.mozilla.org/show_bug.cgi?id=1373396#c3 where he >>> discovered >>> that turning off the "Shut off display after 15 minutes" setting improves >>> our benchmark score by about 10 points or so! I think there are other >>> investigations ongoing to dig into the remaining difference as well. >>> >> Currently, machine #36 is testing against non-PGO builds from inbound (to >> catch regressions). >> I'm looking into adding mozilla-central PGO builds to the mix. >> I assume PGO builds can run slightly faster than non-PGO builds. >> >> Once we have PGO builds we will be able to see if there are anymore >> machine configration changes required (I doubt it). >> >> Direct link to the speedometer score on machine #36: >> >> https://arewefastyet.com/#machine=36&view=single&suite=speedometer-misc&subtest=score >> >>> (Also note that there is a speed difference on Speedometer between >>> Nightly >>> and Beta, where Nightly with the same code will be a bit slower than Beta >>> since some Nightly specific features do show up in Speedometer profiles >>> currently, for example things like bug 1375568 are currently Nightly >>> only, >>> and there are also debugging checks like < >>> >>> https://searchfox.org/mozilla-central/rev/3291398f10dcbe192fb52e74974b172616c018aa/ipc/chromium/src/base/pickle.h#26> >>> that show up a bit in profiles as well. I think that explains why 55.0b3 >>> is scoring so high comparing to 56.0a1 there.) >>> >>> Cheers, >>> -- >>> Ehsan >> >> _______________________________________________ >> dev-platform mailing list >> dev-platform@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-platform > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
