At 6:53 PM +0300 7/10/06, Gaal Yahas wrote:
On Sat, Jul 08, 2006 at 12:47:17AM -0700, Darren Duncan wrote:
As briefly discussed on #perl6 ...
As briefly replied there before being jethandled...
As further discussed there ...
Perhaps we need a baby-Perl Test::Simple for new implementations, and
by convention have 01-sanity use only those functions present there. The
drawback being somewhat duplicated effort and another distribution worry.
I believe that the current Test.pm already qualifies as a baby-Perl
implementation, as overall its functionality is quite simple, and it
uses very little of the language (especially since its use of
junctions was removed a few weeks ago).
Have a look at 01-sanity again. Since I moved 08-test out of that
folder (and into 02-test-pm) none of those 01-sanity tests use
Test.pm at all.
My impression of 01-sanity is that it checks that the Perl 6
implementation has the basic language functions that Test.pm has as a
prerequisite, such as being able to print, knowing basic
conditionals, supporting basic subroutines and modules.
If 01-sanity runs, then the normal Test.pm should be useable, and
02-test-pm checks that Test.pm itself actually works. When that
runs, then the other tests can be performed as usual.
The first obvious question that this enumeration presents is: where to
cut? Does every implementation use its own Test.pm? Pugs was able to
start using one after 23 days, but that was a simpler module then than
it is today, and a fresh implementation may find it hard to get started
if its first objective is to implement so much Perl 6 simultaneously.
I would say for starters that anything which is written in Perl 6
will be shared by all implementations, and that includes Test.pm.
Considering its few needs, any Perl that can't run the current
Test.pm is hardly useful, so they would be getting up to the level of
supporting Test.pm very quickly. And in the process of getting to
that level, the 01-sanity tests can be tried without Test.pm.
The other question that comes to mind is how to manage SKIP and TODO
tests for release mangement. Currently Pugs, assuming globality, simply
plunges into every t/ file before a release and marks forsaken (for this
release) tests TODO, and commits. But surely different implementations
have their different bugs. In the case of TODO we can fix that easily:
we already have a mechanism for force_todoing tests "at a distance",
though currently that distance is at the head of the test file (but it's
a simple matter to make that a distro-level file). The deeper technical
challenge is how to maintain SKIPs, which after all, are often used
before hanging tests. skip_all we can manage; skip we cannot, until at
least when we have a powerful debugger.
All unconditional skips/todos in the test suite should be eliminated
entirely, and be allowed to fail. If hangs are a concern, then the
test harness (presumably not written in Perl 6) can just have a
timeout function and kill any tests that run for too long (the time
threshold can be changed per user in config.yml).
The proper way to have skips/todos is for them to be conditional.
Ideally, one of the first things that any Perl 6 implementation would
provide is something akin to a %FEATURES (or some other named) global
or environmental or config variable, which a Perl 6 script can query
at runtime to see whether that implementation CLAIMS to support
certain features. Only features that are claimed to be supported are
tested, to yield a ok/not ok, and only those features claimed to be
unsupported, either explicitly, or implicity by not being mentioned
at all, will be skipped.
A brand new Perl 6 implementation will declare an empty %FEATURES,
meaning that everything is unsupported, and so everything conditional
skips (which would be most tests, ideally). As the implementation
matures, it starts adding items to %FEATURES, and so skips are
replaced by passes or fails.
Of course, certain sanity items will have to work before %FEATURES
does, such as the ability to print, basic conditionals and boolean
expressions, the ability to read from a variable, etc. But no
support for subroutines or external modules will be needed in order
to do this, so the threshold is lower than for any type of Test.pm.
The %FEATURES variable would be populated in the logical fashion,
derived partly or entirely from explicit declarations in the
implementation code itself, and partly or not at all from user
configuration or environment as is applicable. Entirely from code
may be better so that this doesn't make the CONFIG or ENV or whatever
vars redundant, or they could possibly overlap as makes sense, but my
emphasis is on the code.
If we are going to go with this idea, and I give it my highest level
of recommendation, it would stand to reason that the possible
features to be tested for will be standardized and documented such as
in a Synopsis.
I could start enumerating such a list if no one else wants to, but it
would be long and would stand for a thought out design, like the
tests themselves.
More broadly speaking, I think that a revisiting of the test
hierarchy with a goal to it being able to scale better (see TASKS of
Pugs), and a capabilities/features list could be done somewhat hand
in hand. Features that tend to be implemented together can also be
grouped together for testing, and tests could be grouped generally in
dependency order, such that if feature A needs to be implemented
before feature B can be implemented, feature A will be tested first.
A trivial case of such organizing has started with 01-sanity and
02-test-pm being checked first, but that's as far as it goes so far.
As a Pugs hacker I can say that the immediacy of access to t/ is a
great benefit, and I would hate to lose that. But what's the best way
of granting that benefit to other implementations without crippling it
for anyone?
From the point of view of a *user* of Perl 6, one that gets it from
CPAN, getting this to work is just a matter of packing the tests in
such a way that they can be "installed" some how and be visible to
the packages for Pugs or v6 or whatever.
This may require something unusual, because afaik, only "modules"
typically get installed and become visible to other packages, while
test scripts don't.
(I get around this perceived limitation with Rosetta by implementing
all the backend-shared tests as Rosetta::Validator::* modules, which
install with the framework and can be invoked like a module. But I
don't know if something like that is feasible for a general Perl 6
test, so something else may be required.)
As for hackers working from version control, where immediacy of
access to and seamless integration of updates of otherwise separately
distributed components is important, I don't know exactly what to
suggest as a solution but perhaps something that lets you checkout
multiple SVN into a shared space where the pieces can see each other
as if they were one big distro, and you can just test how the parts
work together naturally. I don't know how to implement this, though.
As you can see, there are possibly wider and wider scopes of problems to
solve here and we can certainly choose not to attack every problem now.
But yes, let's get the discussion going (thanks, Darren, for starting
this thread) and come up with a plan, so that we can be ambitious and
realistic together.
I think it is very important to share whatever resources can be
shared, for easier development and better quality results. If this
causes a few hiccups short term, like test fails rather than skips,
or all tests temporarily skipping, I think it is still worth it.
-- Darren Duncan
