On Wed, Oct 17, 2018 at 2:03 PM Emilio Cobos Álvarez <emi...@crisal.io> wrote: > > On 10/17/18 11:56 AM, James Graham wrote: > > On 17/10/2018 10:12, James Graham wrote: > >> On 17/10/2018 01:23, Emilio Cobos Álvarez wrote: > >>> Hi Philip, > >>> > >>> Do you know how do reftests run in order to get that data? > >>> > >>> I'm particularly curious about this Firefox-only failure: > >>> > >>> css/selectors/selection-image-001.html > >>> > >>> It passes both on our automation and locally. I'm curious because I > >>> was the author of that test (whoops) and the Firefox fix (bug 1449010). > >>> > >>> Does it use the same mechanism than our automation to wait for image > >>> decodes and such? Is there any way to see the test images? > >> > >> It's using the same harness as we use in gecko, so it should be giving > >> the same results, but of course it's possible that there's some > >> difference in the configuration that could cause different results for > >> some tests. > >> > >> Unfortunately there isn't yet a way to see the images; because of the > >> number of failures per run, and the number of runs, putting all the > >> screenshots in the logs would be prohibitively large, but there is a > >> plan to start uploading previously unseen screenshots to wpt.fyi [1] > > > > OK, I investigated this and it turns out that we accidentally started > > uploading tbpl-style logs with screenshots for full runs when we turned > > on taskcluster for PRs. So the screenshot is available through > > > > https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://taskcluster-artifacts.net/U6OIGr7ZTjurDYjy_KgyCg/0/public/results/log_tbpl.log > > Thanks! So it looks that the reftest screenshots are taken on inactive > windows? > > We don't respect ::selection for inactive windows, so the failure now > makes sense. > > Still I think there's something fishy there, but it may be related to > the widget toolkit that is on wpt's CI or something...
Thanks James for accidentally storing screenshots in Taskcluster logs and figuring out how to use them with reftest-analyzer, that's great and I'll pass along this tip to blink-dev as well :D _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
