Re TART regressions and Gavin's concerns - as always, we should not trust the 
numbers blindly.

The first thing we need is probably taking few windows machines with different 
performance characteristics and compare tab animation perf on those machines, 
especially on the cases where TART shows regression (TART measures 10 cases of 
animation - like when opening a new tab, closing a tab, in several DPIs, etc), 
and preferably listen to subjective assessments on this from more than one 
person.

We should also remember that tests tend to be more reliable when the stuff they 
measure is in their "comfort zone". The more specialized the test is - the more 
it expects the test subject to behave within tighter constraints, and vice 
verse - the higher level the test is, the more it could treat its subject like 
a black box, and cares less about internal details.

TART happens to be quite specialized and OMTC is a major shift in graphics 
implementation. Even if TART is already running and providing useful results 
with OMTC on OS X, it could still be out of its "comfort zone" with OMTC on 
windows.

This is true about all tests. The more specialized the test is - the more it 
needs to be kept aligned with test subject, and the more it could produce less 
reliable results when it isn't.

So, we should take the regression numbers with a grain of salt, and try to make 
sure first that the results are still good, and if they are, see what we can do 
about it, etc.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reply via email to