Jerome wrote: > > I'm sorry, I've been very busy for 2 weeks, and did not have the time to > continue things regarding this topic. :-( > I hope to have some more time this week-end, but I'm not even sure...
It happens. :) > So, very quickly: > 1. The main goal is to have unified languages tests, that is, make > language-tests would get a list of all the test files for all of the > languages, and then run only _one_ harness with all those files. This > would allow for one nice overview of all the tests, and tinderboxen > would benefit from this too: they were reporting success as long as the > last language was reporting succes. > > 2. In order for this to work, all the languages should have commit to > having a t/harness file in their directory, responsible to get back a > list of all the test files when called with a -files argument. (Except > if we consider that *.t files are test files, but this may be too > strict) If the language implementer wants to use this harness to also > launch test for the current language only, this is of course welcome I agree with the idea of allowing for a 'make languages-test', but I'm not sure I agree that means Parrot::Test has to be smart enough to run every language's tests directly (more below). [...] > Now, here's my view for the future of languages testing: > 1. I really think the scheme one language_output_is function and one > global harness is the way to go. This does not forbid having one > harness per language: indeed, each language still _needs_ one harness > script to fetch the list of test files. I'm not clear on why we need a separate language_output_is function. P6C::TestCompiler is using the standard output_is, it only overrides &generate_pbc_for so it can compile the p6 source down to pasm before it passes it off to Parrot::Test::generate_pbc_for to compile it down to pbc. I do agree that keeping the language-specific harnesses is very important. (I'll tear my hair out if I have to run the test suite for all the languages every time I want to test Perl 6 :). > 2. Will's patch needs to be updated in (at least) two ways: a) put back > pasm files in test directories I agree. > and b) remove the need for a > Parrot::Test::<Language> module. It should _allow_ it, but should also > provide a convenient default for languages that do not need fancy > testing things. (This should be the case of all the "little" languages > where running a $parrot languages/$lang/$lang.pasm $code_file should be > enough) Hmm... I'd lean toward keeping all the language-specific code in the languages/* directories. Core Parrot shouldn't have to carry extra weight for every language it ever ships with, or might ever need to test. Dropping a language from a specialized distribution should be as simple as dropping the irrelevant languages/* directory. > 3. All the languages needs to be updated to take this scheme into > account once Parrot::Test is modified accordingly. > > That's it, I don't know if I made myself clear? (tricky subject to work > on - and thus to explain :o) ) It seems clear, but if my responses seem like I don't understand, please clarify. > As we saw, it seems that Perl 6's test suite does weird things. Can you > explain what those weird things are, so we can take them into account > in the Plan? The only major thing it does is override &generate_pbc_for and pass it to &generate_functions. > Also, what do you think of the scheme? (and when I say you, I also mean > the p6i list) Knowing that the scheme will allow (currently it > _forces_) you to extend Parrot::Test with Parrot::Test::Perl6 (or > whatever language) Would it be possible to define a set of conventions instead of adding to the weight of Parrot::Test? Say, if a language has a t/harness, it should return X results in Y format. Cardinal may write tests in Ruby, and Pirate may write tests in Python, but it should still be possible to drop them into languages/ and test them with the rest of the languages. > I'll try to work on it saturday, but maybe we should currently revert > the patch on Parrot::Test so I can have some more time re-working the > languages testing stuff, without annoying Leo and you. (at least) It's Leo's call, but sounds reasonable to me. I don't know that we'll reach a final solution this weekend, and I'd like to work on other P6C stuff. Thanks! Allison
