Moritz Lenz wrote:
<snip>
So I'd like to hear your opinions: do you think
adverb-based testing is a good idea? If you don't like
it, do you see any other good way to tackle the
problems I mentioned above?
After reading everything in this thread to date and in order to
structure my thoughts, I wrote up some assertions a suggestion.
1) A perl6 implementation is to be certified by its ability to pass the
test suite. The test functionality is implemented in the implementation
(the implicit recursion has been noted in other threads). This means the
behaviour of the test functionality must be specified and verifiable.
Hence the assertion that perl6 is to be specified by a suite of
documents and tests seems to me to imply that test functionality must be
an inherent part of the language specification.
2) Test functionality for the compiler must be as simple to implement as
possible, so that it can be incorporated into the implementation at an
early stage.
3) Although test functionality for more complex software, eg.,
event-driven GUIs, could be constructed from simple specified test
functionality, more complex forms will be developed to isolate the
features that need testing and to provide the diagnostics.
In other words, test functionality sufficient for the compiler may not
be adequate for module testing. But other functions can be developed in
Test modules that can be hooked into a general testing approach.
4) Testing software is different from debugging or running software.
Running is about providing functionality to the user. Debugging is about
getting expected behaviour and discovering why behaviour exhibited is
not what is expected / specified. Testing is about demonstrating that
the functionality provided is the functionality expected / specified
under all specified conditions.
It seems to me that the ethos of testing could be much wider than just a
part of the development stage. From a risk-management perspective,
mission-critical software (especially when it is complex and large)
should be tested regularly against a standard test suite, because random
errors may occur in the software (eg., a power surge subtly corrupts the
contents of hard disk storage), and particularly after any upgrade of
hardware or ancillary software, or any other environmental change, to
say nothing of changes (upgrades) in the software itself.
Indeed, the inclusion of test functionality in the language and the
focus on test suites as part of perl6 culture would make software
written in perl6 extremely desirable in risk-sensitive companies.
Consequently, it seems to me that the following might be useful:
a) a global variable $*TESTING which defaults to FALSE (or should it be
$?TESTING ?)
It could be set lexically so that specific software / modules can be
tested without triggering tests in other used modules.
b) When $*TESTING is TRUE, any TEST block is executed.
c) Within a TEST block, the tenary <?? !!> is defined slightly
differently, thus for
<perl6 boolean expression> ?? <diagnostic if expression is TRUE> !!
<diagnostic if expression is FALSE>;
<perl6 boolean expression> is guaranteed to return Boolean::FALSE if an
exception or failure condition is encountered when evaluating it.
Some advantages of this approach over :OK<>:
- no new behaviour outside of a TEST block is defined, no change to
adverbs or boolean operators.
- any expression that leads to a boolean result (note that :OK is
suggested to be defined only on boolean operators) can be included in
the expression, eg., an entire block.
- The programer has control over both the "True" diagnostic, as in the
:OK<> syntax, but also over the 'False' diagnostic, thus allowing a
degree of introspection on the component of the expression, which the
programmer has more knowledge about than the compiler.
- Since the variables used in the boolean expression are available to
the programmer for both diagnostics, there is no need for special magic
to generate the failure diagnostic, which seems to be the situation with
:OK<>.
- Since it is the programmer that defines the False diagnostic, no extra
autogenerated macros are needed.
- The minimum that is needed for a test would be to specify a 'true'
diagnostic and the $! error variable, eg.,
TEST { 2 == 2 ?? say 'constants are constants' !! say $! };
d) A TEST block is specified to react to exceptions / failures in a
different manner than in normal blocks. Uncaught exceptions are
discarded at the end of a block. Thus compiler / module / software
failures do not stop the software from continuing, unless specifically
required by the programmer to do so within the block.
e) Other functions that are useful in test suites, such as plan, could
be defined later as wrappers around "?? !!"
Regards,
Richard
