1. The `Assert.*` namespace is optional and may be omitted. This module is also present in the addon-sdk and used _with_ that namespace, usually with a lowercase `assert.*`. Please pick whatever suits your fancy.
2. testharness.js, Mochitest, XPCShell’s head.js and other suite-runners that we use in-tree are needlessly monolithic. They mix defining assertion methods, test setup, test definition, test teardown in one silo and dogmatically impose a test definition style. Their lack of modularity costs us flexibility in adopting and/ or promoting TDD development. They are, however, great for writing regression tests and that’s what we use them for. 3. Mochitest is a framework we adopted years ago (I don’t know how many, though) and the number of upstream changes/ improvements have been limited to none. In fact, I don’t see Mochitest used by any JS community in the wild anymore and it’s not like I haven’t tried looking. The saddening result, however, is that our status quo is a testing framework that we are reluctant to update or change. It is written in a different coding style than Mozilla’s, which makes it alien enough to present a bar for change. Consider the boilerplate needed for writing chrome Mochitests: each .xul test file I open redefines an `ok()` method that is a shorthand to `window.opener.wrappedJSObject.SimpleTest.ok` and use _only_ that assertion method to test our code. We need to improve there, I think, and I’m not sure if Mochitest is the way to go here. We can bikeshed even more about the ‘Simple’ in ‘SimpleTest’, but I’m keeping my cool ;-) 4. None of the test-suites promote modularity and needlessly dictate a reporting style. What I mean by this is that there’s no way to hook different reporting styles in a test runner to promote TDD, for example. What does automation use to detect test failures? TAP[1] is something efficient and optimised for machine-reading, but we parse output lines in a format that is far from an industry standard. We humans delve through a whole bunch of scroll back to find the test case/ assertion we’re interested in. We even rely on our tooling to repeat all the failing tests at the end of a run. 5. Assertion semantics are indeed poorly specified, across the board. Our switch from `do_check_matches()` to `deepEqual()` even revealed a buggy implementation there, which we didn’t know about. Apart from that, it was largely undocumented, not fully covered by unit tests except for the pathological cases. I’m actually a bit scared of what I’ll find in Mochitest[3] Type coercion is something specifiable, but I’m not sure whether that is something `ok`, `equal` and family should implement guards for. If there’s a wish/ call for more specific assertion methods like `is_truthy`, `is_falsy` and variants for all possible coercion traps, I think there’s room in Assert.jsm to add those. We are in the sad status quo that all assertion methods in all test suites are underspecified to some degree. The fact that these methods are an integral part of each suite makes it harder to change that. Assert.jsm breaks away from that approach to make these improvements possible to a wider audience. If we agree that more spec’ing is needed, we might as well fork the spec[2] to MDN and collectively work it out. 6. Having various definitions of assertion methods spread out across different suites makes maintaining them a bit harder. I’m specifically thinking about `deepEqual()` here; I was thinking about adding more robust support for Iterables that can be found in ‘new' JS primitives like Set and Map. Wouldn’t it be nice if we could add support for recursively iterating over their values and asserting scalars for equality? 7. Names of assertion methods are an excellent reason for bikeshedding. The main reason for the amount of time it took for the spec[2] to be formalised was exactly this, IIRC. Never mind that, like I said before: I’m fine with forking the spec and adding aliases for each assertion method if need be. I mostly care about the fact that we can implement them in one place. With a lengthy post like this, it often happens that one steps on ones toe. I apologise for that in advance. Mike. [1] http://en.wikipedia.org/wiki/Test_Anything_Protocol [2] http://wiki.commonjs.org/wiki/Unit_Testing/1.1 [3] https://bugzilla.mozilla.org/show_bug.cgi?id=1018226 On 03 Jun 2014, at 10:52, James Graham <ja...@hoppipolla.co.uk> wrote: > On 03/06/14 00:24, Chris Peterson wrote: >> On 6/2/14, 3:42 PM, Ehsan Akhgari wrote: >>> 2. I also value consistency more than my personal preferences, and based >>> on that, using the existing APIs in some tests and the new APIs in other >>> tests (even if we agreed that #1 above doesn't matter) is strictly worse >>> than the status quo. >> >> btw, in the mozilla.dev.tech.javascript-engine.internals fork of this >> thread, bz and David Bruant pointed out that W3C's testharness and >> TC39's test262 each use yet another set of assertion function names. Any >> tests we import from those test suites will need glue code to integrate >> with our test harness(es). > > In fact, for testharness.js tests (and the W3C web-platform-tests in > general) the plan is to have a dedicated test harness (bug 945222). This > is already up and running on tbpl on Cedar and will be turned on for > mozilla-central as soon as the intermittents are under control (Linux is > looking good, Windows has some issues with WebVTT tests, OSX shows a > little more variability). > > As a result, in the near future we won't need glue code between > testharness.js tests and other kinds of tests. > > FWIW I think the main problem with the CommonJS assertions is their > their semantics. For example: > > * Test assertions shouldn't silently type-cast, but ok, equal and > notEqual all do that. Their brevity compared to strictEqual and > notStrictEqual means that they are likely to be much more widely used. > > * notStrictEqual and notDeepEqual are terrible names that actively > mislead about what the functions do. > > * deepEqual has, as far as I can tell, underspecified semantics. I can't > tell if it is supposed to recurse into nested objects and, if so, > whether there is supposed to be be any loop prevention (the spec talks > about "equivalent values" for every key, without defining what this means). > > * throws doesn't seem to provide any way to test that the type or > properties of the thrown object. > > I know we have test harnesses using assertion functions that already > have some of these problems, but I don't see the benefit of adding a new > set of methods with exactly the same issues. > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
