On 03/06/14 12:27, Mike de Boer wrote: >>> 4. None of the test-suites promote modularity and needlessly dictate >>> a reporting style. What I mean by this is that there’s no way to hook >>> different reporting styles in a test runner to promote TDD, for >>> example. What does automation use to detect test failures? TAP[1] is >>> something efficient and optimised for machine-reading, but we parse >>> output lines in a format that is far from an industry standard. We >>> humans delve through a whole bunch of scroll back to find the test >>> case/ assertion we’re interested in. We even rely on our tooling to >>> repeat all the failing tests at the end of a run. >> >> Yes, the way we deal with test output has historically been suboptimal. >> We are in the process of fixing that as we speak. We have developed a >> json-based protocol [1] for test output. This is already being used for >> web-platform-tests, and for FirefoxOS certsuite. There is current work >> in progress for converting mochitest and marionette tests to this >> format. Other suites will follow. > > Do you have bug numbers where I can follow that work in progress? It sounds > awesome!
Yes, sorry I should have included some. The metabug for structured logging is bug 916295 The Mochitest work is happening in bug 886570 and Marionette in bug 956739. Treeherder is using pivotal tracker rather than bugzilla, so I don't have a bug number for that one, but the "story" for basic integration is [1]. >> Changing the semantics of things that people are already using seems >> like a uphill battle. I think if you wanted to introduce a common set of >> assertion names across Mozilla harnesses, starting from CommonJS rather >> than starting from a discussion of actual requirements was the wrong >> approach. > > That’s not what we’re doing here! Changing semantics is a non-goal, merging > assertion methods into one re-usable module is. > > Taking the CommonJS spec as an umbrella for these simple assertion methods is > a causality and I think it helps provide a common, immediate understanding > for new contributors who’d like to write test for the code they contribute. > > Sure, the semantics of `do_check_matches()` changed. But that method was only > used in two locations, its use not promoted in any wiki page and its > implementation lossy. I was under the impression that you were proposing landing CommonJS method support and then forking the commonjs spec to improve the semantics. I may have misunderstood. I think I would need some evidence to back up the hypothesis that new contributers are unusually likely to know CommonJS compared to the N other test frameworks that exist. For example most automation contributers have a better working knowledge of Python than Javascript and would be more comfortable with something that looks like Python's unittest module. I imagine for front end developers there would be different biases. For platform developers the story is likely to be different again. In any case, learning new assertion names isn't something that strikes me as being a particularly high barrier to entry. With testharness.js the only complaint I recall about the names is that they favour explicitness over brevity. Confusion for people moving from other js test frameworks has more often come from the differences in high-level philosophy. [1] https://www.pivotaltracker.com/s/projects/749519/stories/70575430 _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
