On 04/01/2012 02:50 AM, Bob Friesenhahn wrote:
> On Sun, 1 Apr 2012, Stefano Lattarini wrote:
>
>> On 03/31/2012 11:47 PM, Bob Friesenhahn wrote:
>>>
>>> how does one re-execute just one test in order to see the details
>>> of how it failed?
>>>
>> With Automake, one shouldn't -- he should write his test to be verbose
>> enough so that a perusing of the logs is sufficient to reveal the reason
>> of the failure.
>
> Ok, now I hear that there are still log files similar to parallel-tests
> (presumably using the identical facility).
>
Yes. And also (newly-introduced) "trs" (Test ReSults) files that store
some metadata about the tests' runs (note however that the format of
these files is still highly exeprimental and subject to change).
> It does not seem that Russ's C TAP Harness offers this quite wonderful
> feature since the instructions advise to re-run just the one test to
> see more verbose output (similar to what I have experienced from perl).
>
>> You mean running only a given test case in a given TAP test script, or all
>> the test cases in a given TAP test script? Currently, there is no
>> transparent
>> way way to do the former (which is hardly surprising), but for the latter,
>> you would use the same interface you use for "simple" tests:
>>
>> make check TESTS=granular.tap
>>
>> or even:
>>
>> make check TEST_LOGS=granular.log
>
> So Automake TAP still supports the same parallel-tests facility but
> treats files with the ".tap" extension as being TAP test scripts?
>
Not by default; you must declare which extensions should use the TAP
driver, with the definition of a proper "<ext>_LOG_DRIVER" variable.
For example, if your testsuite contains "simple tests" written as shell
script (and having a '.sh' extension) and TAP tests written as perl
scripts (and having a '.pl' extension), your Makefile.am might look
like this:
# Tell which program should run the .sh scripts.
SH_LOG_COMPILER = $(SHELL) -ex
# Tell which program should run the .pl scripts
PL_LOG_COMPILER = $(PERL) -MTAP::More -w
# Tell which program should interpret the outcomes for the .pl scripts
PL_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) \
$(top_srcdir)/build-aux/tap-driver.sh
In the next Automake version, I hope to be able to refine the above
to something like this:
SH_LOG_COMPILER = $(SHELL) -ex
PL_LOG_COMPILER = $(PERL) -MTAP::More -w
PL_LOG_PROTOCOL = tap
Of course, the use of "<ext>_LOG_DRIVER" variables will still be
supported, to allow developers to define their own drivers if they
want to (e.g., if they want to support a protocol like SubUnit that
is not supported by default by Automake).
> That sounds like a useful approach.
>
I'm glad to hear this.
> An issue with using the TAP facility to reduce the number of tests
> (to fix command line length issues) is that it reduces the amount
> of available test parallelism.
>
That's a risk if TAP is over-used or abused; it's up to the developer
to strike a balance. You can take a look at the TAP usage in the
automake's own testsuite for few examples.
Regards,
Stefano
