On Sat, Oct 11, 2008 at 2:24 AM, Allison Randal <[EMAIL PROTECTED]> wrote: > Moritz Lenz wrote: >> >> jerry gay wrote: >> A combined harness is much better in terms of reporting. > > Yes. > a combined harness is much easier now that we require T::H 3. anybody with TH3-shaped tuits who could take a look at rakudo's and/or parrot's test harness with an eye on combining harness results would certainly be rewarded by increased karma and high praise.
>>> the tests we expect to pass reliably should be the default tests we >>> run. we expect all spectest_regression tests to pass reliably. the >>> default test target should always be named 'test'. it seems natural >>> that we add spectest_regression to the 'test' makefile target. >>> additionally, this would have possibly prevented the 74 failures >>> post-mdd-merge, since allison didn't know about the additional test >>> target in the makefile. >> >> well, if reading the README is too much even for our architect then we >> shouldn't assume that anybody else does ;-) > > Another thing that would be helpful for languages in trunk is something like > a TESTME file. It should briefly say exactly what steps a core developer > should take to test that their changes haven't broken the language, and if > failures are expected or all tests should pass. Also, anything strange like > having multiple test harnesses running in sequence instead of aggregating > the results in one report. (That one caught me on Rakudo's 'make test' too. > I thought all the test were passing, and then found that the final "All > tests pass" report was hiding earlier failures in a different summary.) The > README is quite verbose and intended for people who want to use the > language. Even after reading it, it's not straightforward to decide what to > test and whether failures are relevant. > i'll try to address rakudo's documentation this weekend at pittsburgh perl workshop. this seems like a great task for new contributors, working alongside old hands. > What would be really ideal is if core developers could just run 'make > languagetest' in the repository root and get a single report of all the > language failures, and know for sure that any failures are their > responsibility. But, we're a long way away from that. > TH3 makes this a lot easier than previously. ~jerry
