15.3.3.2 Log files generation and test results recording
:test-global-result:
This is used to declare the "global result" of the script. Currently,
the value of this field is needed only to be reported (more or less
verbatim) in the generated global log file $(TEST_SUITE_LOG), so it's quite
free-form. For example, a test script which run 10 test cases, 6 of which
pass and 4 of which are skipped, could reasonably have a PASS/SKIP value for
this field, while a test script which run 19 successful tests and one failed
test could have an ALMOST PASSED value. What happens when two or more
:test-global-result: fields are present in the same .trs file is undefined
behaviour.
There is no indication of how to treat XFAIL, XPASS, SKIP alone or in
combination. Since a tap-driver.sh accepts '--expect-failure yes' as input
option it is quite conceivable multiple test lines can yield these results.
Only PASS/FAIL are addressed.
If
TEST_EXTENSTIONS = .tap
TAP_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) ./tap-driver.sh
TAP_LOG_DRIVER_FLAGS = --expect-failure yes
TESTS = test.tap
What is the exit status for test line combinations of SKIP, XPASS, and
XFAIL. Suppose the test plan is 1..N and there is a combination of SKIP,
XPASS and XFAIL, that is, PASS => XPASS and FAIL => XFAIL, what is the exit
status and what is the the :test-global-result: value. The :test-result:
values are SKIP, XPASS, XFAIL.
