On Tue, May 03, 2005 at 10:36:40AM -0400, Stevan Little wrote:
> The ongoing search to find a good model for TODO tests in Pugs
> continues ...
>
> To start with, the idea of some kind of "switch" to turn all failures
> into TODOs has been removed. It was a bad idea from the start and since
> it never actually was implemented, I have ditched it entirely.
Good.
> Second, I would like to move away from the force_todo() usage, as it
> has a number of obvious problems. However, I cannot dismiss it's
> usefulness as a quick-fix during release preparation, so i will not yet
> remove it.
That also sounds good. If not sure, wait a bit.
> It was determined (on #perl6 and various emails) that what we really
> need in Pugs is a multi-layered TODO model. In Pugs we have several
> different kinds of TODO tests, along with the plain old TODO. Here is a
> list of some of the more common ones and examples of how the attributes
> might look;
This sounds like a really good plan: specify why a test is TODO.
> - TODO
>
> :todo, :todo(1)
>
> Sometimes, it just doesn't fit anywhere else, so in that case it is
> just TODO.
So, vanilla.
> - TODO bug
>
> :todo<bug>, todo => 'bug', :todo('bug')
>
> This is either an outright bug in Pugs, or it is a feature which
> /should/ be implemented, but for some reason (usually some kind of
> seemingly unrelated bug) it is not. This is a good "layer" for edge
> cases tests and other such ugly corners.
So this is a test which should be working, but isn't. Still, the
definition for this one is a little hazy in my mind.
> - TODO feature
>
> :todo<feature>, todo => 'feature', :todo('feature')
>
> These are tests written for perl6 features which are not yet
> implemented in Pugs. Objects is the perfect example of this.
This is what TODO is all about: this is going to be implemented, but
isn't yet, so I'm marking it TODO until it is implemented.
So maybe the 'bug' one means: this was implemented, but now seems to be
unimplemented (because internals changed somehow, etc.).
> - TODO parsefail
>
> :todo<parsefail>, todo => 'parsefail', :todo('parsefail')
>
> These are tests which fail to parse correctly. These are usually tested
> with eval_ok() or eval_is().
So this is like 'feature', except that the parser does not understand
the syntax yet. Maybe it could be called 'syntax', which might make
more sense.
> - TODO hardfail
>
> :todo<hardfail>, todo => 'hardfail', :todo('hardfail')
>
> These are tests which parse okay, but die with an uncatchable error. I
> will be experimenting with using the "is lazy" trait to catch and
> not-evaluate these (and therefore not kill pugs).
This one is hazy in my mind, too. I'm guessing it means the syntax
parses, but is still not understood, so the interpretter dies when
trying to execute. It's the difference between understanding (the
syntax) and comprehending (the meaning). English doesn't really have
good words to describe it. Dutch uses 'verstaan' (for words/syntax) and
'begrijpen' (for comprehension/meaning).
> My goal was to support these different kinds of TODOs with Test.pm, and
> also make them visible through Test::Harness (and eventually in our
> smoke test). In exploring Test::Harness I noticed that TODO tests can
> also have a "reason", just like SKIP tests do (although Test::Harness
> seems to ignore them) and I have exploited this feature to accomplish
> my goal.
Very nice.
> So currently (in the latest revision), Test.pm will support the
> following code:
>
> ok(bad_function(), "... testing bad_function", :todo<bug>);
>
> this will produce the following TAP output:
>
> not ok 1 - ... testing bad_function # TODO bug
> # Failed (TODO bug) test (-e line 3, column 1-46)
> # Got: undef
>
> I have also altered our local copy of Test::Harness (in inc/) so that
> if you were to run this with 'prove' or 'make test' you would see
> something like this:
>
> t/my_test_file....ok
> 1/5 TODO bug test
> 2/5 TODO tests
> 2/5 TODO feature tests
> All tests successful.
>
> This allows a simple top-level overview of not only the number of TODO
> tests, but of each specific TODO test "layer". Ideally we can support
> this level of TODO granularity in the test smoke interface as well.
That looks really useful. We might just be able to leave the TODO tests
as TODO, once we have all the reasons in place.
> Now all that is left is for people to start changing all the :todo
> tests to :todo<some_attribute>.
Can you put the definitions of each attribute in one of the README
files?
> As always comments, question and suggestions are always welcome.
And gladly given. Good work, Stevan.
-kolibrie
