On Sat, Jul 08, 2006 at 12:47:17AM -0700, Darren Duncan wrote: > As briefly discussed on #perl6 ...
As briefly replied there before being jethandled... > I propose that we make like Sun and its Java VM validation suite, > and start distributing the Perl 6 test suite as its own distribution > on CPAN Here are a number of issues. 1. Naming > Audrey suggested that it could be named v6::tests ... I previously to > that suggested Perl6::Certification ... and other suggestions are > welcome. Please let's not call it certification. That gives foreground to a question of authority which I think is (certainly at this stage, possibly later too) a needless distraction. 2. Modularity > From that point onward, neither the Perl6::Pugs nor v6 distros would > include their own test suites, or such would be extremely basic. > Their 'make test' and 'make smoke' would instead call out to > v6::tests. > > (A substantial part of the testing and/or smoking harness should > probably also be included, so people that use it get more consistent > formatted results that can be compared, though some details will need > to be in the individual implementation distros, as only they know how > they are invoked. Whether that is written in Perl 6 or Perl 5 or > something else is less of an issue short term.) The Pugs test system has grown organically and has functionality at the following levels, most of which currently reside in the Pugs repo: A) A Perl 6 Test.pm that borrows heavily from Perl 5 TAP goodies: ok, is, like, cmp_ok, isa_ok, is_deeply, dies_ok, skip, and related functions; as well as conventions for extra TAP info supplying message coordinates and skip/todo reasons, used in reports [ext/Test/ in the Pugs repo]. B) A Perl 5 Straps harness that keeps more metadata around for machine processing than Test::Harness' default "make smoke", and which has the extra bonus of working faster on SMP machines [originally util/yaml_harness.pl in the Pugs repo, which has since been refactored to mostly be a pugs-specific front-end to Test::TAP::Model on CPAN]. C) Local report generators: a colorful TAP results generator [Tetst::TAP::HTMLMatrix on CPAN and a frontend in util/testgraph.pl] and a FIT-and-Project-Xanadu inspired cross-referencer into arbitrary locations in the spec and the tests [util/catalog-tests.pl]. D) A smoke server system for transmitting and viewing other smoker's reports [code in util/smokeserv/ ; service maintained by Ingo Blechschmidt]. E) Scripts and cron jobs on feather to autosmoke pugs every revision and submit results [currently maintained by Juerd, I believe]. (The slides at http://perlcabal.org/~gaal/pugstest/start.html demonstrate most of these features.) The first obvious question that this enumeration presents is: where to cut? Does every implementation use its own Test.pm? Pugs was able to start using one after 23 days, but that was a simpler module then than it is today, and a fresh implementation may find it hard to get started if its first objective is to implement so much Perl 6 simultaneously. Perhaps we need a baby-Perl Test::Simple for new implementations, and by convention have 01-sanity use only those functions present there. The drawback being somewhat duplicated effort and another distribution worry. The other question that comes to mind is how to manage SKIP and TODO tests for release mangement. Currently Pugs, assuming globality, simply plunges into every t/ file before a release and marks forsaken (for this release) tests TODO, and commits. But surely different implementations have their different bugs. In the case of TODO we can fix that easily: we already have a mechanism for force_todoing tests "at a distance", though currently that distance is at the head of the test file (but it's a simple matter to make that a distro-level file). The deeper technical challenge is how to maintain SKIPs, which after all, are often used before hanging tests. skip_all we can manage; skip we cannot, until at least when we have a powerful debugger. 3. Generality > On a separate but related matter, I suggest also separating anything > else that isn't Pugs specific, that is common to both it and v6, from > the Pugs distro and distributing those separately. This includes the > contents of: examples/ , ext/, some of docs/, some of util/, and so > on. > > These various distros can be distributed on their own schedule, as > frequent or infrequent as makes sense, just as v6 and Moose et al are. > > As to whether any of the above are still maintained within Pugs SVN > or moved to separate SVN, that is a possibly orthogonal matter. As a Pugs hacker I can say that the immediacy of access to t/ is a great benefit, and I would hate to lose that. But what's the best way of granting that benefit to other implementations without crippling it for anyone? And a different tack on all of this: at YAPC I heard from Jesse Vincent that work has been done in Best Practical to refactor the test system for a non-Perl 6 project, or rather, collection of products. Their effort is called Chimps and its repo lives at: http://svn.bestpractical.com/cgi-bin/index.cgi/bps/browse/Test-Chimps/trunk/ It is still very much work in progress! But I did see a demo and it's looking good. They are focusing on a multi-product smoke server built on our stuff (thanks for the compliment, guys!). (Yuval Kogman had also been putting some work on that before hearing of Chimps, but I think he put that on low flame due to unvocal demand.) The Best Practical stuff will be open source and they sound happy to both receive and give back code contributions. So, I think there's a very big overlap between several different users. I'd love for more cooperation and a single code base (though of course sometimes different smokeserver services) for - Pugs hackers during development - Other Perl 6 implementors - @Larry as people starting from a spec doc and exploring how well a particular feature is tested and supported by the implementations (the game is "follow the superscript Ts": http://perlcabal.org/~gaal/pugstest/t_index/Synopsis/S06.html#175425272 ). - A "product" company with more than one report subject, possibly not 100%-public - CPAN hackers? - CPAN users? As you can see, there are possibly wider and wider scopes of problems to solve here and we can certainly choose not to attack every problem now. But yes, let's get the discussion going (thanks, Darren, for starting this thread) and come up with a plan, so that we can be ambitious and realistic together. -- Gaal Yahas <[EMAIL PROTECTED]> http://gaal.livejournal.com/
