Thanks for the replies everyone!
On 7/3/25 21:40, Jacob Bachmeyer wrote:
On 7/3/25 18:11, Russ Allbery wrote:
Karl Berry <k...@freefriends.org> writes:
[...]
This confusion comes from not clearly thinking through what the test
*is*. A proper test verifies conformance to a specification. The
example from the DejaGnu manual is a specification "The sun shall
shine." and a test "The sun is shining." which passes or fails depending
on whether it is day or night.
Well, my original thinking was writing tests with invalid input for a given
function, and have it expectedly "not ok". This makes some sense because
one would expect a function given bogus (or perhaps edge case) arguments
to fail. The discrepancy is whether to report a failure as "ok", or to
mark it
as "not ok" with a directive for the harness.
In this case, the specification is "The syscall shall return an error if
given an invalid argument." and the test provides an invalid argument
and expects an error return code. The test *passes* ("ok" in TAP) *if*
*and* *only* *if* the syscall reports failure when given an invalid
argument.
[...]
TAP *should* *not* have semantics for that. TAP scripts are supposed to
be runnable directly or through trivial harnesses like piping the output
through `grep 'not ok'`. If the meaning of "ok" and "not ok" could be
inverted on a test-by-test basis, there would be room for endless
confusion.
Yeah this makes more sense, thanks for the insight. It does make sense that
changing the meaning of ok/not ok is not a good idea, though the binary
choice
does feel limiting.
A renewed proposal could be adding an " # XFAIL " directive, that simply
moves the counting of the test to the XFAIL: row of the output of `make
check`.
That would accomplish both the ability to programtically deem tests as
"XFAIL"
and would preserve the semantics of ok/not ok. Please let me know would be
worthwhile.
Infact my original motivation is/was to see if I could make use of the
various
rows put out by `make check`, which admittedly is rather cosmetic.
