# from James Keenan via RT
# on Friday 21 September 2007 13:44:
>On Thu Sep 20 13:03:31 2007, [EMAIL PROTECTED] wrote:
>>http://perlsix.org/svn/parrot/view/branches/unified_testing/BRANCH_TODO
>>...
>> 5) Add processing of the output of Configure --test.
>
>When I began working on writing tests for the Perl 5 components of
>Parrot last November, we made the decision to not incorporate all the
>tests I was writing under the 'make test' target.
This is perfectly fine.
>... some take a long time to run
>...some of the tests were only meaningful if run post-configuration,
>pre-build...
Those are fine reasons to exclude tests from a given target.
>So I'm unclear as to what the 'unification' you're talking about would
>mean for these test suites. Can you clarify?
I think "unification" is primarily needed in the sense of unifying how
the results are reported. Do not be alarmed -- the fact that
TAP::Parser has "a parser" (and an aggregator) as separate objects from
the harness means there is more than one way to do it.[1]
Also, "unification" in the sense that the shebang line for the tests
tell the harness ~how to execute the test, not where to find the parrot
and etc. It is intended that the test shebangs will function against
an installed parrot and/or hll.
And, also "unification" might involve specifying/organizing the tests
(e.g. which are "developer", which are "exhaustive", etc) as e.g. a
YAML file rather than capturing `$some_t_harness --files`.[2] If this
can be completely expressed in data and/or directory layout, the amount
of support code required to select a set of tests to run is reduced or
eliminated.
The details are in the details, err... footnotes.
[1] For example, a smoke server might be logging the TAP results from
`./Configure.pl --test`, then doing some post-processing and reporting
on those results in conjunction with the rest of the TAP tree. The
time at which those tests are run need not be dictated by the single
harness, but it would be nice to unify the config/declaration of which
tests are which and why -- possibly allowing a smoker to run them
outside of the context of ./Configure so long as the state of the
universe required by these tests is known.
[2] This is not strictly necessary, but from my adventures in reading a
few $some_t_harness files, the list of files does not usually require
computation, and yet can be difficult to deduce from a quick read vs a
data file. Perhaps some environment variables or etc. become involved
and the list of tests actually does require computation. At the least,
the unified harness will not require each t/harness to execute
Test::Harness::runtests(@tests)[3] and so the code is reduced simply to
the task of returning a manifest of files to be run. Perhaps
t/manifest, (which (yes) might still need to be executed) but does not
actually run the tests.[4] But maybe a nice directory layout would do?
[3] "unification" actually conflicts with "each t/harness calling
runtests()" in that the options (such as archiving the TAP) would need
to be communicated in a rather roundabout way.
[4] Whatever best serves the goal of being able to reliably report on
the results of all of the tests run, how long they (each and all) took,
and (hopefully cleanly) selects which to run (or not run) when.
--Eric
--
Moving pianos is dangerous.
Moving pianos are dangerous.
Buffalo buffalo buffalo buffalo buffalo buffalo buffalo.
---------------------------------------------------
http://scratchcomputing.com
---------------------------------------------------
