On Wed, Oct 10, 2012 at 12:46 AM, Jonas Sicking <jo...@sicking.cc> wrote: > However, for someone working against a looming deadline, the cost of > writing sharable tests and risk missing the deadline can be much > higher than having those tests not be run by other browser vendors.
Which is why we have rolling releases, right, so things can be pushed off a bit if necessary to get them right? If individual projects' managers set deadlines that don't prioritize portable testing, then of course those projects will have trouble getting portable tests done, but that's true for any possible goal. Many software projects don't require good testing or QA at all, and as a result developers write half-baked features so they can meet deadlines that don't allow enough time to do things right. That doesn't mean we should avoid doing things right! Naturally, we do have to prioritize our resources, and sometimes it will make more sense to not bother with reusable tests if they're too much work. That doesn't mean we shouldn't aim to *ever* write reusable tests. > Additionally, I think that the more overhead the harness has, the less > thorough the tests will be. I've definitely noticed that the tests > from Opera for CORS has been a lot less thorough than the tests that I > wrote myself. I absolutely think that there's a stronger bias against > writing more comprehensive tests the more work is needed to write > those tests. No amount of "rules" forcing developers to write tests > against testharness.js will remove that bias. I wrote a lot of tests using testharness.js that I hope you'll agree are pretty thorough, such as Range and Selection tests: http://hg.mozilla.org/mozilla-central/file/5cca0408a73f/dom/imptests/webapps/DOMCore/tests/approved http://hg.mozilla.org/mozilla-central/file/5cca0408a73f/dom/imptests/editing/selecttest On the other hand, I've seen *manual* tests (= no framework overhead at all) submitted to the W3C by Microsoft that are so superficial as to be worse than useless, not to mention some that were wrong. That you write better tests than other people can't be blamed on the framework! In practice: I've written many tests, both complicated and simple, in both Mochitest and testharness.js. Once you get used to them, they're about equally easy to write, although testharness.js winds up being slightly longer. If you're writing any sort of nontrivial test and you're familiar with the test harness you're using, 90% of your effort is going to be thinking about what exactly to test and how to test it, not actually typing stuff. That said, of course, Mozilla hackers *are* familiar with Mochitest but not testharness.js, and adopting testharness.js in parallel with Mochitest would require people to be familiar with both. That is certainly a minus. > So I'd much rather spend the additional effort to create a test > harness which is optimized for getting comprehensive tests than take > the additional cost to developers and the reduction in test quality > that comes with a overly heavyweight harness. I think making Yet Another Test Harness is not the right way to go. testharness.js has traction in the standards world, it's well documented, it already works in all browsers, at least Mozilla and Opera (possibly MS/WebKit too) can already use testharness.js tests internally, and there are people from every major browser who are at least somewhat familiar with it. It may not be perfect, but it doesn't make sense to reinvent the wheel still again unless there are really compelling specific flaws that can be identified. > This doesn't match my experience at all. My experience is that writing > tests has a high cost and results in fairly complex test files. It depends. If you're writing tests for an entire feature, then yes, it's very complicated. I've done that -- Range, Selection, and editing, for instance. But most routine bugs are simple things and the tests are just a few lines long. Even if we only shared those tests, it would be a step forward. On Wed, Oct 10, 2012 at 6:41 AM, Boris Zbarsky <bzbar...@mit.edu> wrote: > On 10/10/12 12:23 AM, Ian Bicking wrote: >> >> Here's how I think you'd write a simple XHR test in both: >> >> // SimpleTest aka MochiTest >> req = new XMLHttpRequest(); >> req.open("GET", "/example.json"); > > > How did example.json get there? > > What if you need to test CORS? > > With mochitest at this point you're doing some SJS work and whatnot. > >> // testharness > > > And here you have to go and do whatever is appropriate to your server setup. > Which is not part of testharness. Which is why the CORS tests imported from > Opera to the W3C ended up all broken, because they did not configure the > server correctly. testharness.js absolutely does not support all the same features as Mochitest. It also doesn't support taking screenshots, for instance, which makes it useless for certain things. But we can still use it for the things it does support, like self-contained DOM tests. Although yes, this means using a mix of two different testing frameworks, which is annoying, especially if you start in testharness.js and then realize it needs to be a Mochitest. But there are plenty of types of tests that can easily be testharness.js. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
