On Tue, Oct 9, 2012 at 2:43 AM, Aryeh Gregor <a...@aryeh.name> wrote: > On Sat, Oct 6, 2012 at 6:25 AM, Jonas Sicking <jo...@sicking.cc> wrote: >> In general, testharness.js seems to be more optimized for producing a >> result report which measure how close an implementation is to >> implementing a feature, than it is optimized for making it easy to >> write tests. > > I think it's actually optimized for Opera's internal testing needs as > much as anything, since James Graham is the one who wrote it. Opera > has a much more elaborate testing infrastructure than we do. I think > they actually have stuff like databases instead of grepping plaintext > output, and the ability to run individual test files, and > automatically running new test files X times to record all > intermittent failures. If we just layer it on top of mochitest, we > can't take advantage of the greater structure, so the added complexity > is wasted. James has indicated that features like "each file runs the > same set of tests no matter how many fail" are essential to working > properly with Opera's test setup. > >> I believe many developers right now end up spending as much time >> writing tests as they do implementing features. That is a very high >> cost, but something that is definitely worth it. However we should be >> working towards lowering that cost rather than increasing it. >> >> Rather than trying to convince developers that testharness.js would in >> fact not increase the cost of writing tests, I think we should try to >> get W3C to adjust testharness.js such that it's easier to write tests >> for it. If we make it as easy to write W3C tests as it is to write >> mochitests then I would absolutely agree with your proposal. I would >> imagine that would also make it easier to get other browser vendors to >> do the same, as well as members of the webdevelopment community. > > I think testharness.js would definitely increase the costs of writing > tests. I think the increase in interop would be worth it, if other > browsers start using our tests. It's all good and well if we > implement a feature correctly, but if other browsers implement it > differently, it's not very useful to authors. This is why we want to > submit tests to the W3C to start with.
I agree that at a large scale, the additional value from writing tests that are sharable with other browser vendors and with the W3C community is technically worth the overhead that testharness.js brings. However, for someone working against a looming deadline, the cost of writing sharable tests and risk missing the deadline can be much higher than having those tests not be run by other browser vendors. Additionally, I think that the more overhead the harness has, the less thorough the tests will be. I've definitely noticed that the tests from Opera for CORS has been a lot less thorough than the tests that I wrote myself. I absolutely think that there's a stronger bias against writing more comprehensive tests the more work is needed to write those tests. No amount of "rules" forcing developers to write tests against testharness.js will remove that bias. So I'd much rather spend the additional effort to create a test harness which is optimized for getting comprehensive tests than take the additional cost to developers and the reduction in test quality that comes with a overly heavyweight harness. I do understand that Opera might have various requirements for what fits in their test harness, but I think if the goal is to create a harness "for the web" then we should optimize for what the web needs. Which is lots of tests and comprehensive ones. >> Another problem that I think we'd have is that many of our tests use >> generators and yield. This *dramatically* cuts down on the complexity >> of writing complex tests which has lots of asynchronous callbacks. For >> example [1][2] would have been much harder to write without them. I >> think our approach here could be migrate these tests to use ES6 based >> generators as soon as we have them implemented in gecko, and then >> submit them to W3C as soon as enough browsers implement ES6. >> >> I don't think that we should be telling people to not use generators >> in the meantime. My experience is that rewriting tests to use >> generators both cuts down on the test writing time, and makes it much >> less likely that the test ends up with intermittent orange bugs. > > If it's a pain to write a particular file in testharness.js, it can be > kept as mochitest. In my experience, quite a lot of tests boil down > to like ten lines, which would take about three minutes more to write > using testharness.js than mochitest. Also, a test that's based on > testharness.js but uses some Gecko-only features would be easier to > make portable later than a test that's based on mochitest and also > uses Gecko-only features. This doesn't match my experience at all. My experience is that writing tests has a high cost and results in fairly complex test files. / Jonas _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
