On Mon, Jul 10, 2006 at 03:55:25PM -0700, Darren Duncan wrote:
> I believe that the current Test.pm already qualifies as a baby-Perl
> implementation, as overall its functionality is quite simple, and it
> uses very little of the language (especially since its use of
> junctions was removed a few weeks ago).
Test.pm currently uses the following non-trivial language features:
- implicitly, the ability to load a module
- function export
- function named args
- parameter default values
- &caller
- $?CALLER::CALLER::POSITION
- &sub.goto
- %*ENV
- END {...}
Some of these can be given up on in a prospective Test::Simple, if only
because they in themselves shouldn't be the first things an implementation
should strive to provide for their own sake.
> Have a look at 01-sanity again. Since I moved 08-test out of that
> folder (and into 02-test-pm) none of those 01-sanity tests use
> Test.pm at all.
>
> My impression of 01-sanity is that it checks that the Perl 6
> implementation has the basic language functions that Test.pm has as a
> prerequisite, such as being able to print, knowing basic
> conditionals, supporting basic subroutines and modules.
>
> If 01-sanity runs, then the normal Test.pm should be useable, and
> 02-test-pm checks that Test.pm itself actually works. When that
> runs, then the other tests can be performed as usual.
Some things are missing in 01-sanity/ if the particular implementation
of our current Test.pm is the par for tests there.
Clearly more tests should be added, but we can permit fledgling
implementations to use their own worse-is-better Test modules. So these
tests should perhaps go into 02-test-pm/, not 01-sanity/.
> I would say for starters that anything which is written in Perl 6
> will be shared by all implementations, and that includes Test.pm.
>
> Considering its few needs, any Perl that can't run the current
> Test.pm is hardly useful, so they would be getting up to the level of
> supporting Test.pm very quickly. And in the process of getting to
> that level, the 01-sanity tests can be tried without Test.pm.
(Responded to above.)
> >The other question that comes to mind is how to manage SKIP and TODO
> >tests for release
>
> All unconditional skips/todos in the test suite should be eliminated
> entirely, and be allowed to fail. If hangs are a concern, then the
> test harness (presumably not written in Perl 6) can just have a
> timeout function and kill any tests that run for too long (the time
> threshold can be changed per user in config.yml).
Can you suggest a portable implementation for killing hanged tests?
> The proper way to have skips/todos is for them to be conditional.
>
> Ideally, one of the first things that any Perl 6 implementation would
> provide is something akin to a %FEATURES (or some other named) global
> or environmental or config variable, which a Perl 6 script can query
> at runtime to see whether that implementation CLAIMS to support
> certain features. Only features that are claimed to be supported are
> tested, to yield a ok/not ok, and only those features claimed to be
> unsupported, either explicitly, or implicity by not being mentioned
> at all, will be skipped.
>
> A brand new Perl 6 implementation will declare an empty %FEATURES,
> meaning that everything is unsupported, and so everything conditional
> skips (which would be most tests, ideally). As the implementation
> matures, it starts adding items to %FEATURES, and so skips are
> replaced by passes or fails.
>
> Of course, certain sanity items will have to work before %FEATURES
> does, such as the ability to print, basic conditionals and boolean
> expressions, the ability to read from a variable, etc. But no
> support for subroutines or external modules will be needed in order
> to do this, so the threshold is lower than for any type of Test.pm.
>
> The %FEATURES variable would be populated in the logical fashion,
> derived partly or entirely from explicit declarations in the
> implementation code itself, and partly or not at all from user
> configuration or environment as is applicable. Entirely from code
> may be better so that this doesn't make the CONFIG or ENV or whatever
> vars redundant, or they could possibly overlap as makes sense, but my
> emphasis is on the code.
>
> If we are going to go with this idea, and I give it my highest level
> of recommendation, it would stand to reason that the possible
> features to be tested for will be standardized and documented such as
> in a Synopsis.
>
> I could start enumerating such a list if no one else wants to, but it
> would be long and would stand for a thought out design, like the
> tests themselves.
What sort of granularity are you thinking of for this variable? Some
test files have one subtest fail for a reason somewhere deep in the
internals. These reasons tend to be implementation specific and lack
good names: "pending AST refactoring" is both spatially and temporally
imprecise, for it affects many surface features and yet there may well
be further refactorings in the future. And yet this, for example, is
one of the things blocking many tests from being fixed for the next
version of Pugs. Nearly all these features sorta work today, but not
100% correctly. What does and does not work from the surface looks quite
arbitrary. How would you set %FEATURES for them?
> More broadly speaking, I think that a revisiting of the test
> hierarchy with a goal to it being able to scale better (see TASKS of
> Pugs), and a capabilities/features list could be done somewhat hand
> in hand. Features that tend to be implemented together can also be
> grouped together for testing, and tests could be grouped generally in
> dependency order, such that if feature A needs to be implemented
> before feature B can be implemented, feature A will be tested first.
> A trivial case of such organizing has started with 01-sanity and
> 02-test-pm being checked first, but that's as far as it goes so far.
I agree that's a good goal, and that sanity tests are the first place to
do it. What's next up?
> >As a Pugs hacker I can say that the immediacy of access to t/ is a
> >great benefit, and I would hate to lose that. But what's the best way
> >of granting that benefit to other implementations without crippling it
> >for anyone?
>
> From the point of view of a *user* of Perl 6, one that gets it from
> CPAN, getting this to work is just a matter of packing the tests in
> such a way that they can be "installed" some how and be visible to
> the packages for Pugs or v6 or whatever.
>
> This may require something unusual, because afaik, only "modules"
> typically get installed and become visible to other packages, while
> test scripts don't.
As a first approximation, this doesn't have to be installable in a
system location.
--
Gaal Yahas <[EMAIL PROTECTED]>
http://gaal.livejournal.com/
