On 7/5/25 14:36, Soham wrote:
Thanks for the replies everyone!
On 7/3/25 21:40, Jacob Bachmeyer wrote:
[...]
TAP *should* *not* have semantics for that. TAP scripts are supposed
to be runnable directly or through trivial harnesses like piping the
output through `grep 'not ok'`. If the meaning of "ok" and "not ok"
could be inverted on a test-by-test basis, there would be room for
endless confusion.
Yeah this makes more sense, thanks for the insight. It does make sense
that
changing the meaning of ok/not ok is not a good idea, though the
binary choice
does feel limiting.
The binary choice is *intended* to be limiting. Either the behavior
meets the specification ("ok") or it does not meet the specification
("not ok").
The only other options are failures in the testsuite itself (DejaGnu
"UNRESOLVED"; TAP "Bail out!"), an incomplete test (DejaGnu "UNTESTED";
TAP "TODO"), or lack of local support rendering the test meaningless
(DejaGnu "UNSUPPORTED"; TAP "SKIP").
A renewed proposal could be adding an " # XFAIL " directive, that simply
moves the counting of the test to the XFAIL: row of the output of
`make check`.
That would accomplish both the ability to programtically deem tests as
"XFAIL"
and would preserve the semantics of ok/not ok. Please let me know
would be
worthwhile.
As long as the understanding is that an XFAIL test is a test that
*should* pass but currently fails due to bugs here or elsewhere.
-- Jacob
