On Sunday 22 May 2011, Ralf Wildenhues wrote: > Hi Stefano, and sorry for the long delay, > No problem, you had warned me in due time about such possible delays this month; so there's really no need to apologize.
> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST: > > Now that my GSoC application "automake - Interfacing with a test protocol > > like TAP or subunit" has been officially accepted, I'd like to start > > discussing with the community some early, high-level design and interface > > decisions. > > > > 1. Reuse parallel-tests "framework" > > ----------------------------------- > > > > The new TAP/SubUnit support should reuse as much of the current > > parallel-tests implementation and semantics as possible. In particular, > > it should be able to run different test scripts in parallel, generate a > > `.log' file for each test script and a "summarizing" `test-suite.log' > > file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT > > and AM_COLOR_TESTS and the environment variable VERBOSE, and support > > different extensions for the test scripts, with extension-specific "log > > compilers" and flags (the stuff enabled by TESTS_EXTENSIONS, > > <ext>_LOG_COMPILER, etc.). > > Sounds all sane. > > > The XFAIL_TESTS variable might be still supported for the sake of > > backward-compatibility (see below for the details), but it should be > > deprecated, since TAP and SubUnit offer better and more granular ways > > to express expected failures. > > OK. > > In another mail: > > Thinking again about this, it might be worth trying to be even more > > consistent > > with the existing parallel-tests functionality, and use an > > `ext_TEST_PROTOCOL' > > variable (or similar) instead of a global `tests-protocol' option. With > > some > > tweaking to the post-processing of `.log' files done in `lib/am/check.am' > > (to > > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a > > more > > consistent API. > > > > I've started experimenting with this idea, and I'm not seeing any obvious > > shortcoming right now. I'm hoping I'll be able to post some experimental > > patches soon enough. > > Allowing to specify that per-test is a good idea for transitioning test > suites. > About this, in my first two "tentative" patches: <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html> I've taken an even more general approach, allowing the developer to define and use his own program(s) to: 1. launch his test scripts, 2. interpreter their results, 3. display these results on screen, and 4. format and generate the log files. All of this is attainable simply by assigning a variable `LOG_WRAPPER' (and extensione-specific counterparts of it), and, well, obviously providing a real "driver" script that obeys to minimal command-line interface (so that it can grasp the options the Automake-generated Makefiles passes to it). Then we will hopefully be able to implement our TAP/SubUnit parsers on the top of this feature (thus making it indirectly more tested, which is always good for a new feature). > I hope to look into the posted patches later today. > About this, please note that I might be AFK until this evening. So have no haste. > > 2. New automake option `tests-protocol' > > --------------------------------------- > > > > The Tap/SubUnit support in the Automake-generated testsuite drivers > > should be enabled by a new (argument-requiring) option `tests-protocol', > > that will be used to specify the level of support for, detection of, and > > enforcing of SubUnit/TAP streams. > > > > The possible values for `tests-protocol' will be: > > - tests-protocol=tap > > All test scripts are expected to use the TAP protocol. > > - tests-protocol=subunit > > All test scripts are expected to use the SubUnit protocol. > > - tests-protocol=adaptive > > The way you describe "adaptive", it sounds like it should rather be > named something like "detect" or "detected" or so. > I'd like to withdraw this proposal now that we can define per-extension test protocols. Having our hypotetical "client developer" rename his test scripts as they get converted to use TAP/SubUnit is IMO better than we having to implement in Automake a probably non-trivial "metaparser" that could end up being scarcerly used anyway. WDYT? > > Each test script is expected to print on its first line of output > > which protocol it uses (the exact format of this special line is > > still to be determined); if this line is unrecognized, the driver > > should assume that the test script uses no protocol. Also, in > > this case, we should continue to honour XFAIL_TESTS. All of this > > should help to maximize backward-compatibility. > > > > > 3. Console output from the test driver > > --------------------------------------- > > > > This output should remain as close as possible to the one already > > provided by the current parallel-tests driver. The following example > > should help to clarify what I mean. > > > > Assume we are using `tests-protocol=adaptive', and let TESTS be defined > > to "pass.test skip.test subunit.test tap.test". Here, `pass.test' and > > `skip.test' are test scripts that use no protocol (and that exit with > > status `0' and `77' respectively), `subunit.test' is a test script using > > the SubUnit protocol and containing four testcases (one passing, one > > failing, one skipped and one which incurs in an expected failure), and > > `tap.test' is a test script using the TAP protocol which runs two > > successful testcases, then encounters an internal error and bails out > > (using the "Bail out!" directive). > > > > With such a setup, this is the output I'd expect from "make check": > > PASS: pass.test > > SKIP: skip.test > > PASS: subunit.test [testcase name/description] > > FAIL: subunit.test [testcase name/ description] > > SKIP: subunit.test [testcase name/description] [reason for skipping] > > XFAIL: subunit.test [testcase name/description] [failure reason] > > PASS: tap.test [testcase name/description] > > PASS: tap.test [testcase name/description] > > ERROR: tap.test [reason for the bailout] > > > > Of course, the `color-tests' option should make the above output properly > > colorized; the attached html file shows what colors I'd expect. > > Wrt. to the details of the output, I wouldn't fix things too early. > Two things two consider here: > - if tests are run in parallel, you want to avoid intermixing output > from different tests as much as possible, > I disagree. Once the test script name is printed in each line which report the result of one of its test cases, I see no issue with intermixed lines. I.e., I don't believe that this: PASS: pass.test PASS: subunit.test [foo] PASS: tap.test [t1] SKIP: skip.test PASS: tap.test [t2] FAIL: subunit.test [bar] SKIP: subunit.test [baz] [reason for skipping] ERROR: tap.test [reason for the bailout] XFAIL: subunit.test [quux] [failure reason] is any less clear than this: PASS: pass.test SKIP: skip.test PASS: subunit.test [testcase name/description] FAIL: subunit.test [testcase name/ description] SKIP: subunit.test [testcase name/description] [reason for skipping] XFAIL: subunit.test [testcase name/description] [failure reason] PASS: tap.test [testcase name/description] PASS: tap.test [testcase name/description] ERROR: tap.test [reason for the bailout] XXX > but even more than that, you want to avoid intermixing within lines. > OTOH, I do believe this is a real concern, to be carefully addressed and tested for. Thanks for bringing this up. > Gathering output and > printing it all at once can help here, but the downside is possibly > losing output if tests are interrupted, or the driver doesn't finish > correctly. > - some sort of bypass of the (currently hidden) output is helpful, > some gnulib-using packages do that already by redirecting stderr > before running the test, and printing a reason for skipping from > within the test. > [thinking about this put on the TODO list] > I think the basic premise to take into account is that the test driver > author (you) may not know all requirements a future test author may > have. > That's why I like my current idea of allowing the client developers to provide their own testsuite driver. > > 4. RST support and HTML generation: should be dropped? > > ------------------------------------------------------ > > Good question. You could do a poll on the automake list (in a separate > mail & thread, to gain visbility). Or just defer the implementation > for later. > > In another mail: > > BTW, I also hope that new interface I'm planning will make it easier to > > implement HTML report generation also for TAP and SubUnit tests, with a > > consistent reuse of the existing code. In which case my considerations > > above will become moot. > > Cool. > The downside of this is that it might place additional burdens on the writer of test drivers. Hmmm... maybe a new option "no-html-test" or similar is warranted? > Thanks! > Ralf > Thanks for the review. Regards, Stefano
