Mozilla has a long-standing policy that with certain limited exceptions, all code changes must be accompanied by a test. Following this policy has given us an excellent and steadily growing regression test-suite. Some of these tests are very specific to Mozilla, but a substantial fraction test our conformance to web standards that other UAs implement. Unfortunately, new tests are as a rule written in very Mozilla-specific formats. I think we would do a better job of fulfilling our mission to advance the web if we made more of an effort to write our tests in such a way that other UAs could more easily use them, where relevant. I believe we could do this without significantly increasing the burden on test-writers.
To that end, I suggest the following course of action: 1) Decide on guidelines for whether a test is internal or reusable. As a starting point, I suggest that all tests that are regular webpages that don't use any Mozilla-specific features should be candidates for reuse. Examples of internal tests would be tests written in XUL and unit tests. In particular, I think we should write tests for reuse if they cover anything that other browsers implement or might implement, even if there's currently no standard for it. Other browsers should still be able to run these tests, even if they might decide not to follow them. Also, tests that currently use prefixed web-exposed properties should still be made reusable, since the properties should eventually be unprefixed. 2) Write an introduction to testharness.js targeted at people familiar with mochitest. testharness.js is the de facto standard testing harness in the web standards world, and we already can run such tests as mochitests automatically (see dom/imptests/), so JavaScript tests meant to be usable by other browsers should be written in that format. 3) Require that all new tests that qualify as reusable must be written in testharness.js format rather than mochitest format if possible. (Reftests and crashtests can remain as-is, IMO. Some mochitests might not be possible to rewrite as testharness.js yet, e.g., those using SpecialPowers, so I guess they can stay mochitests for now.) 4) Require that all new tests that qualify as reusable must be checked into a specific new directory created for this purpose, rather than someplace near the code as they are currently. Reviewers need to eventually start giving r- for tests written in the wrong format or put in the wrong place, although it would make sense to phase the requirement in over time and not be too strict at first. Test writers do not have to bother actually publishing them -- they just have to write them in the correct format and put them in the correct directory in the source tree. 5) Make sure someone is keeping an eye on the reusable-tests directory, and submitting the tests as appropriate to somewhere where they can be easily reused. This might involve submitting some of them to standards bodies for formal approval. Other tests might not currently follow any standard, but could still be imported by other browsers to test for crashes or assertions, or to flag possible regressions. Those tests might not be moved anywhere special, but should still be easier for other browsers to reuse than they are now. I think that the above won't make anything much harder for our coders, but will be a big step forward for web testing -- especially if our example motivates other browsers to do the same. It needs a little bit of infrastructure work, but nothing much. (1) and (2) are easy, and I suspect someone like Ms2ger will be happy to handle (5). But (3) and (4) need broad agreement from module owners/peers. Does anyone have any objections to them? Once we have a system like this in place, we can try to persuade other browser vendors to use our tests, and provide their tests under similar terms. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
