On 11/08/2012 01:10 PM, Aryeh Gregor wrote:
On Wed, Nov 7, 2012 at 4:13 PM, James Graham <jgra...@opera.com> wrote:
There is an experimental branch with this mode in; it isn't production
quality yet. I am still unsure that it's a good idea; in particular I think
it encourages people to write multiple tests on the same page in such a way
that if one fails the whole suite stops running. This seems very unpleasant
to deal with; remember that unlike typical Mozilla tests these will not
necessarily all pass in their unmodified form, and that modifying imported
tests inline creates a headache when you want to merge changes.
In Mochitest, a failed assert doesn't stop execution of the file, so
even if fails aren't marked inline, the results are still likely to be
useful. An unexpected result just prints out a TEST-UNEXPECTED-PASS
or TEST-UNEXPECTED-FAIL line to the console instead of TEST-PASS or
TEST-EXPECTED-FAIL, which is then slurped out by various line-based
parsing tools and reported as a failure. (Yes, our infrastructure is
not so sophisticated.)
In testharness.js, a failed assert causes the whole test to fail.
This is not very useful if the whole file is one giant test that tests
lots of different things. If a failed assert wasn't fatal in this
experimental new mode, on the other hand, it would still be pretty
useful.
I think that would be possible to arrange, but you couldn't do that and
retain an invariant on the number of tests run in the face of arbitrary
(i.e. not asserts) lines of code failing. This would make it very hard
to compare results between runs. If you allowed the file as a whole to
have a status you could distinguish between "this file ran without
unexpected exceptions" and "this file had an unexpected exception", but
it still seems like it would be annoying if, when implementing an API,
tests for an unimplemented method caused a single "fail" result rather
than detailed information about the parts that passed.
I know Mozilla use a system where all the tests in a file should pass,
but I don't see how that will work well when you don't control the
tests. If you are manually editing every file when you import it, I
imagine that updating tests will be so time consuming that it will be
tempting not to bother. How do you plan to address this?
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
