GraphicsMagick uses a custom TAP-test script (patterned on the example
one from Automake) where there is support for deciding if the test is
expected to fail based on a list of features. For example, the support
for a file format may depend on an optional library and that library is
a "feature". The test_command_fn() function has this usage description:
# Test a simple command where pass/fail depends on command exit status
#
# Usage:
# test_command_fn description [ -F 'feat1 feat2 ...'] command
#
# Where:
# description -- test description
# -F -- optional space-delimited required features
# command -- command to execute
A typical test expression looks like:
test_command_fn "MNG ${type}" -F PNG ${MEMCHECK} ${rwfile} -filespec
"out_${type}_%d" "${SRCDIR}/input_${type}.miff" MNG
so in the above one can see that "PNG" is a feature (in this case
depending on libpng being present).
This strategy has been working well for a great many years.
The ability to supply a custom test functions script which is matched to
the software being tested is quite useful.
The idea of adding a directive that certain tests are expected to fail
is interesting, but the immediate feature-based approach seems much
easier (and more efficient) than trying to add indirect support, such as
via an algorithm or database lookup.
Bob
On 6/30/25 20:23, Jacob Bachmeyer wrote:
On 6/30/25 16:41, Soham wrote:
Hello,
I recently was using the tap driver provided by automake, and noticed
that there currently is no way to declare a test as xfail or xpass
apart from marking it as "todo". The test ouput however does produce
rows for XPASS and XFAIL, and the existing TRS format already
supports both.
Since the TAP specification allows for custom directives [1], I was
wondering if there was interest in a patch adding these to
`lib/tap-driver.sh`. I would add "XPASS" and "XFAIL" directives, and
if the test did not pass/fail as expected, it would result in an ERROR.
Perhaps Autotest uses ERROR differently, but the better way to think
about it is expected-vs.-unexpected results. A test marked XFAIL that
fails is expected to fail, but a test marked XFAIL that passes is a
surprise XPASS.
A better solution is to add a directive that declares that certain
tests are expected to fail. The TAP driver then matches the actual
results with the expected results.
-- Jacob
