Hello automakers. Now that my GSoC application "automake - Interfacing with a test protocol like TAP or subunit" has been officially accepted, I'd like to start discussing with the community some early, high-level design and interface decisions.
JFTR, you can read my GSoC application here: <http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/slattarini/1> And here are some useful links where you can find more information about TAP and SubUnit: - The documentation for the TAP format understood by the perl module `Test::Harness' (kind of "de-facto standard"): <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod> - A wiki on the TAP protocol and format: <http://testanything.org/wiki/index.php/Main_Page> - A C-based project implementing both a TAP producer and a TAP consumer: <http://www.eyrie.org/~eagle/software/c-tap-harness/> - The SubUnit homepage: <https://launchpad.net/subunit> - A blog entry about SubUnit usage in Samba: <http://jelmer.vernstok.nl/blog/archives/262-subunit-usage-in-Samba.html> First Tentative Decisions and Questions on the User Interface ============================================================= 1. Reuse parallel-tests "framework" ----------------------------------- The new TAP/SubUnit support should reuse as much of the current parallel-tests implementation and semantics as possible. In particular, it should be able to run different test scripts in parallel, generate a `.log' file for each test script and a "summarizing" `test-suite.log' file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT and AM_COLOR_TESTS and the environment variable VERBOSE, and support different extensions for the test scripts, with extension-specific "log compilers" and flags (the stuff enabled by TESTS_EXTENSIONS, <ext>_LOG_COMPILER, etc.). The XFAIL_TESTS variable might be still supported for the sake of backward-compatibility (see below for the details), but it should be deprecated, since TAP and SubUnit offer better and more granular ways to express expected failures. 2. New automake option `tests-protocol' --------------------------------------- The Tap/SubUnit support in the Automake-generated testsuite drivers should be enabled by a new (argument-requiring) option `tests-protocol', that will be used to specify the level of support for, detection of, and enforcing of SubUnit/TAP streams. The possible values for `tests-protocol' will be: - tests-protocol=tap All test scripts are expected to use the TAP protocol. - tests-protocol=subunit All test scripts are expected to use the SubUnit protocol. - tests-protocol=adaptive Each test script is expected to print on its first line of output which protocol it uses (the exact format of this special line is still to be determined); if this line is unrecognized, the driver should assume that the test script uses no protocol. Also, in this case, we should continue to honour XFAIL_TESTS. All of this should help to maximize backward-compatibility. 3. Console output from the test driver --------------------------------------- This output should remain as close as possible to the one already provided by the current parallel-tests driver. The following example should help to clarify what I mean. Assume we are using `tests-protocol=adaptive', and let TESTS be defined to "pass.test skip.test subunit.test tap.test". Here, `pass.test' and `skip.test' are test scripts that use no protocol (and that exit with status `0' and `77' respectively), `subunit.test' is a test script using the SubUnit protocol and containing four testcases (one passing, one failing, one skipped and one which incurs in an expected failure), and `tap.test' is a test script using the TAP protocol which runs two successful testcases, then encounters an internal error and bails out (using the "Bail out!" directive). With such a setup, this is the output I'd expect from "make check": PASS: pass.test SKIP: skip.test PASS: subunit.test [testcase name/description] FAIL: subunit.test [testcase name/ description] SKIP: subunit.test [testcase name/description] [reason for skipping] XFAIL: subunit.test [testcase name/description] [failure reason] PASS: tap.test [testcase name/description] PASS: tap.test [testcase name/description] ERROR: tap.test [reason for the bailout] Of course, the `color-tests' option should make the above output properly colorized; the attached html file shows what colors I'd expect. 4. RST support and HTML generation: should be dropped? ------------------------------------------------------ I haven't really looked into this yet, so I don't know how much additional work (if any!) would be to continue to support generation of HTML reports from the testsuite-generated `.log' files in the new TAP/SubUnit testsuite driver. Still, one question that bugs me is: how much is this HTML report generation used in Real Life? Is it really worth to support it out of the box, or could we leave it to the interested used to write their own rules (maybe in a reusable `*.am' fragment)? -*-*- Comments, ideas and criticism are welcome! Regards, Stefano
