On Thu, Jun 26, 2008 at 11:36:05AM +0200, Moritz Lenz wrote: > This is a more or less random collection of thoughts and questions about > the rakudo testing infrastructure. > > First of all I feel that it's in a rather good shape.
I agree -- it's looking better all of the time. Many thanks to you, Auzon, and particle for the excellent work! > I put up a chart of number of tests on http://rakudo.de/ and plan to > update it regularly (via cron job once the admin installs Text::CSV_XS). Shiny! I'll make good use of the charts :-). > [...] it would be useful to have an option that makes localtest more > verbose. Specifically if a script dies, it's not obvious after which > test it died. I'm not exactly sure what is meant here -- do you want to know which test file died, or a specific test in the test file? Isn't that normally produced by the test harness summary? (It is on my system.) > I have two scripts in my local tools/ directory, both of which aren't > platform independent (only tested on linux so far). One writes a patch > for automagically unfudging tests (aka autounfudge.pl), the other > generates the chart on rakudo.de. What should I do with them? I think they can go in the tools/ subdirectory for now. > isa_ok in pugs and rakudo seem to work differently. Many tests call > isa_ok with a string as the second argument: > isa_ok(0, 'Int'); > while rakudo only understands the bare type name: > isa_ok(0, Int); > It seems to me that the latter is more intuitive, but I'd still like > some feedback before changing the tests on a larger scale. I can't find any examples in the Synopses where a string is used for 'isa' testing... but I can also see why we'd want to allow strings. I think we need clarification from p6l here. > There seems to be a thorough confusion about numeric types. for example > some tests read like this: is (1.1).WHAT, 'Num'; and then in a different > file is (1.1).WHAT, 'Rat'; That raises two questions for me > 1) should we test for the value of .WHAT at all (unless for testing WHAT > of course)? It seems to me that a compiler should have the freedom to > return a different subtype (for example 1 could be a PositiveInt as long > as PositiveInt is an Int; that's a contrived example, but you get the > picture) > 2) How do we know which numeric type is a class and which is a role? Is > there an explicit spec about the types of number literals? That could > have some impact on type checking in the tests. (Maybe I should prod p6l > with that). This is another place where we probably need guidance from p6l. > I've attached a list of skip messages from spectest_regression, roughly > sorted into categories, some messages commented by me (in parenthesis). Excellent. Note that the tools/test_summary.pl script also produces the skip messages from spectest_regression, and I'll likely add todo messages shortly. > Finally I'd like to point out some annoyances with rakudo. It's rather > frustrating (for example while fudging a large test file) to get an > error message that can't easily be mapped to a location in the source file. > [...] > I guess the various Past:: nodes already have line information attached > - would it be possible to print that whenever something fails that > involves Past nodes? Reporting line number information for errors at runtime is RT#53082, which in turn currently depends on RT#43269. I don't have a feeling that setline/setfile will be fixed in Parrot anytime soon, which means that we probably need to come up with a workaround for PAST. I'm also waiting for an opportunity to refactor PGE to use Cursor objects, which will be able to more efficiently compute line number information for the PAST nodes. Otherwise we end up doing a lot of redundant calculation for line numbers, (which can get expensive on utf8 strings in Parrot). Thanks! Pm
