I think this is generally a good idea. I have a few questions before
jumping in to agree though.
1. Is the current testharness.js API the documentation at the beginning
of <http://w3c-test.org/resources/testharness.js>? If that is the case,
the API looks a lot heavier weight than the default mochitest API we
use. In that case, would it make sense for us to have a compatibility
layer which translates our mochitest APIs to the equivalent
testharness.js API calls? (I'm not 100% sure if that conversion would
be straightforward.)
2. Is there any support for running reftest-style tests in a framework
that is reusable by other browsers? If not, can we move to propose the
reftest framework to the appropriate standards bodies so that it can be
adopted by other browsers? Our reftest framework has been carefully
designed to be Gecko-agnostic, and is much superior to the equivalent
testing framework that WebKit has (not sure about other browser
engines). Furthermore, the files loaded by this framework are not
loaded in a privileged context with APIs such as SpecialPowers, which
makes a large number of them portable to other browser engines.
I think it makes sense for us if we can start this effort on the reftest
framework, since that has a much lower barrier to entry, and ultimately
this effort would be valuable only if other browser engines start to use
our tests (and hopefully share theirs with us as well).
Cheers,
Ehsan
On 12-08-16 5:31 AM, Aryeh Gregor wrote:
Mozilla has a long-standing policy that with certain limited
exceptions, all code changes must be accompanied by a test. Following
this policy has given us an excellent and steadily growing regression
test-suite. Some of these tests are very specific to Mozilla, but a
substantial fraction test our conformance to web standards that other
UAs implement. Unfortunately, new tests are as a rule written in very
Mozilla-specific formats. I think we would do a better job of
fulfilling our mission to advance the web if we made more of an effort
to write our tests in such a way that other UAs could more easily use
them, where relevant. I believe we could do this without
significantly increasing the burden on test-writers.
To that end, I suggest the following course of action:
1) Decide on guidelines for whether a test is internal or reusable.
As a starting point, I suggest that all tests that are regular
webpages that don't use any Mozilla-specific features should be
candidates for reuse. Examples of internal tests would be tests
written in XUL and unit tests. In particular, I think we should write
tests for reuse if they cover anything that other browsers implement
or might implement, even if there's currently no standard for it.
Other browsers should still be able to run these tests, even if they
might decide not to follow them. Also, tests that currently use
prefixed web-exposed properties should still be made reusable, since
the properties should eventually be unprefixed.
2) Write an introduction to testharness.js targeted at people familiar
with mochitest. testharness.js is the de facto standard testing
harness in the web standards world, and we already can run such tests
as mochitests automatically (see dom/imptests/), so JavaScript tests
meant to be usable by other browsers should be written in that format.
3) Require that all new tests that qualify as reusable must be written
in testharness.js format rather than mochitest format if possible.
(Reftests and crashtests can remain as-is, IMO. Some mochitests might
not be possible to rewrite as testharness.js yet, e.g., those using
SpecialPowers, so I guess they can stay mochitests for now.)
4) Require that all new tests that qualify as reusable must be checked
into a specific new directory created for this purpose, rather than
someplace near the code as they are currently. Reviewers need to
eventually start giving r- for tests written in the wrong format or
put in the wrong place, although it would make sense to phase the
requirement in over time and not be too strict at first. Test writers
do not have to bother actually publishing them -- they just have to
write them in the correct format and put them in the correct directory
in the source tree.
5) Make sure someone is keeping an eye on the reusable-tests
directory, and submitting the tests as appropriate to somewhere where
they can be easily reused. This might involve submitting some of them
to standards bodies for formal approval. Other tests might not
currently follow any standard, but could still be imported by other
browsers to test for crashes or assertions, or to flag possible
regressions. Those tests might not be moved anywhere special, but
should still be easier for other browsers to reuse than they are now.
I think that the above won't make anything much harder for our coders,
but will be a big step forward for web testing -- especially if our
example motivates other browsers to do the same. It needs a little
bit of infrastructure work, but nothing much. (1) and (2) are easy,
and I suspect someone like Ms2ger will be happy to handle (5). But
(3) and (4) need broad agreement from module owners/peers. Does
anyone have any objections to them? Once we have a system like this
in place, we can try to persuade other browser vendors to use our
tests, and provide their tests under similar terms.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
