Hello Robert. On Wednesday 06 July 2011, Robert Collins wrote: > Very sorry for the slow response, been EBUSY with real-life. > > On Sun, May 22, 2011 at 11:42 PM, Stefano Lattarini > <stefano.lattar...@gmail.com> wrote: > > On Sunday 22 May 2011, Ralf Wildenhues wrote: > >> Hi Stefano, and sorry for the long delay, > >> > > No problem, you had warned me in due time about such possible delays this > > month; so there's really no need to apologize. > > > >> * Stefano Lattarini wrote on Fri, Apr 29, 2011 at 11:21:06AM CEST: > >> > Now that my GSoC application "automake - Interfacing with a test protocol > >> > like TAP or subunit" has been officially accepted, I'd like to start > >> > discussing with the community some early, high-level design and interface > >> > decisions. > >> > >> > >> > 1. Reuse parallel-tests "framework" > >> > ----------------------------------- > >> > > >> > The new TAP/SubUnit support should reuse as much of the current > >> > parallel-tests implementation and semantics as possible. In > >> > particular, > >> > it should be able to run different test scripts in parallel, generate a > >> > `.log' file for each test script and a "summarizing" `test-suite.log' > >> > file, honour the make variables AM_TESTS_ENVIRONMENT, TESTS_ENVIRONMENT > >> > and AM_COLOR_TESTS and the environment variable VERBOSE, and support > >> > different extensions for the test scripts, with extension-specific "log > >> > compilers" and flags (the stuff enabled by TESTS_EXTENSIONS, > >> > <ext>_LOG_COMPILER, etc.). > >> > >> Sounds all sane. > >> > >> > The XFAIL_TESTS variable might be still supported for the sake of > >> > backward-compatibility (see below for the details), but it should be > >> > deprecated, since TAP and SubUnit offer better and more granular ways > >> > to express expected failures. > >> > >> OK. > >> > >> In another mail: > >> > Thinking again about this, it might be worth trying to be even more > >> > consistent > >> > with the existing parallel-tests functionality, and use an > >> > `ext_TEST_PROTOCOL' > >> > variable (or similar) instead of a global `tests-protocol' option. With > >> > some > >> > tweaking to the post-processing of `.log' files done in > >> > `lib/am/check.am' (to > >> > generate `$(TEST_SUITE_LOG)'), this might allow greater code reuse and a > >> > more > >> > consistent API. > >> > > >> > I've started experimenting with this idea, and I'm not seeing any obvious > >> > shortcoming right now. I'm hoping I'll be able to post some experimental > >> > patches soon enough. > >> > >> Allowing to specify that per-test is a good idea for transitioning test > >> suites. > >> > > About this, in my first two "tentative" patches: > > <http://lists.gnu.org/archive/html/automake-patches/2011-05/msg00093.html> > > I've taken an even more general approach, allowing the developer to define > > and use his own program(s) to: > > 1. launch his test scripts, > > 2. interpreter their results, > > 3. display these results on screen, and > > 4. format and generate the log files. > > All of this is attainable simply by assigning a variable `LOG_WRAPPER' > > (and extensione-specific counterparts of it), and, well, obviously > > providing a real "driver" script that obeys to minimal command-line > > interface (so that it can grasp the options the Automake-generated > > Makefiles passes to it). Then we will hopefully be able to implement > > our TAP/SubUnit parsers on the top of this feature (thus making it > > indirectly more tested, which is always good for a new feature). > Note that a slightly-improved version of this design has now been implemented (and documented) in the public branch 'test-protocols'.
> If the subunit parser just gets all the output from the test script, > More than that: it gets the *command* that should be used to run the test script (this is usually just the test script path, but might be a little more complicated in case the test has an associated LOG_COMPILER). This way, the driver can also get the script exit status and standard error (and possibly other informations, e.g., the test execution time). > you might want to use the subunit parser itself: > You mean the existing python implementation, right? Well, honestly I was planning to do exactly that (and similarly for TAP, using the TAP::Parser module), since I've seen I won't be able to have portable TAP/SubUnit parsers by the midterm evaluation deadline. Also, such parsers would provide an existing (albeit temporary) implementation I can write test cases against, which will offer a way to make the future "portable" parsers more compatible with the existing "reference" implementations. > folk in the > bootstrap-set couldn't (obviously), but most other environments can > use high level languages freely. (I don't object to more parser > implementations existing, I'm just thinking about reuse where > possible. > Agreed, that would be a good idea. And the current design should make it quite easy to implement. > > [SNIP] > > >> > >> > 3. Console output from the test driver > >> > --------------------------------------- > > [SNIP] > > >> Wrt. to the details of the output, I wouldn't fix things too early. > >> Two things two consider here: > >> - if tests are run in parallel, you want to avoid intermixing output > >> from different tests as much as possible, > >> > > I disagree. Once the test script name is printed in each line which report > > the result of one of its test cases, I see no issue with intermixed lines. > > I.e., I don't believe that this: > > > > PASS: pass.test > > PASS: subunit.test [foo] > > PASS: tap.test [t1] > > SKIP: skip.test > > PASS: tap.test [t2] > > FAIL: subunit.test [bar] > > SKIP: subunit.test [baz] [reason for skipping] > > ERROR: tap.test [reason for the bailout] > > XFAIL: subunit.test [quux] [failure reason] > > > > is any less clear than this: > > > > PASS: pass.test > > SKIP: skip.test > > PASS: subunit.test [testcase name/description] > > FAIL: subunit.test [testcase name/ description] > > SKIP: subunit.test [testcase name/description] [reason for skipping] > > XFAIL: subunit.test [testcase name/description] [failure reason] > > PASS: tap.test [testcase name/description] > > PASS: tap.test [testcase name/description] > > ERROR: tap.test [reason for the bailout] > > > >> but even more than that, you want to avoid intermixing within lines. > >> > > OTOH, I do believe this is a real concern, to be carefully addressed and > > tested for. Thanks for bringing this up. > > For Both TAP and subunit the test script running needs to feed into a > single parser: > This is not possible with the current implementation/design of third-party test drivers in Automake. Every test script listed in $(TESTS) is passed *separately* to dedicated instance of the proper test driver; this means that the output of *each* test script in $(TESTS) will be parsed by a different instance of the TAP (or SubUnit) parser. > the issue with intermingling then is all about how the > parsers structure their output. > > Ideally you would use a threadsafe structure. > > For subunit, *if* you were working in python you could use the > existing multithreaded logic testrepository uses, which can mingle > multiple streams correctly, preserving datestamps - a single ui parser > then handles all the combined output. > While all this seems nice and sensible, it is unfortunately incompatible with the basic design choice of the new Automake third-party test drivers support, which is the ability of share as much code and semantics as possible with the pre-existing 'parallel-tests' driver, and to easily associate a `.log' file to each test script. > > > >> > 4. RST support and HTML generation: should be dropped? > >> > ------------------------------------------------------ > >> > >> Good question. You could do a poll on the automake list (in a separate > >> mail & thread, to gain visbility). Or just defer the implementation > >> for later. > >> > >> In another mail: > >> > BTW, I also hope that new interface I'm planning will make it easier to > >> > implement HTML report generation also for TAP and SubUnit tests, with a > >> > consistent reuse of the existing code. In which case my considerations > >> > above will become moot. > >> > >> Cool. > >> > > The downside of this is that it might place additional burdens on the > > writer of test drivers. Hmmm... maybe a new option "no-html-test" or > > similar is warranted? > > At least for subunit, and I believe TAP afficiondos will agree for > TAP, you'd generally run subunit2html [made upname] to generate a html > report. An example of such a formatter is subunit2junithtml, and I > believe samba have a perl based html formatter for subunit already > (Jelmer will know more about that). > For the moment, I've managed to provide an interface that is unobtrusive and doesn't break the existing RST->HTML conversion; i.e., test results are registerd in the log files using the `:test-result:' reStructuredText field (some small additional contraints are required if the RST->HTML conversion is to be supported, but they are not required by any other testsuite harness feature, so that the autohr of a test driver won't be forced to implement them; this is an intended behaviour, and is now tested by the Automake's own testsuite). Regards, Stefano
