Autoconf’s Autotest has notion of expected-to-fail (aka XFAIL) tests and it
served my needs very well about past 18-20 years.
Sent from mobile
On Tue, 1 Jul 2025 at 22:24, Bob Friesenhahn <
graphicsmagick.proj...@gmail.com> wrote:
> GraphicsMagick uses a custom TAP-test script (patterned on the example
> one from Automake) where there is support for deciding if the test is
> expected to fail based on a list of features. For example, the support
> for a file format may depend on an optional library and that library is
> a "feature". The test_command_fn() function has this usage description:
>
> # Test a simple command where pass/fail depends on command exit status
> #
> # Usage:
> # test_command_fn description [ -F 'feat1 feat2 ...'] command
> #
> # Where:
> # description -- test description
> # -F -- optional space-delimited required features
> # command -- command to execute
>
> A typical test expression looks like:
>
> test_command_fn "MNG ${type}" -F PNG ${MEMCHECK} ${rwfile} -filespec
> "out_${type}_%d" "${SRCDIR}/input_${type}.miff" MNG
>
> so in the above one can see that "PNG" is a feature (in this case
> depending on libpng being present).
>
> This strategy has been working well for a great many years.
>
> The ability to supply a custom test functions script which is matched to
> the software being tested is quite useful.
>
> The idea of adding a directive that certain tests are expected to fail
> is interesting, but the immediate feature-based approach seems much
> easier (and more efficient) than trying to add indirect support, such as
> via an algorithm or database lookup.
>
> Bob
>
> On 6/30/25 20:23, Jacob Bachmeyer wrote:
> > On 6/30/25 16:41, Soham wrote:
> >> Hello,
> >>
> >> I recently was using the tap driver provided by automake, and noticed
> >> that there currently is no way to declare a test as xfail or xpass
> >> apart from marking it as "todo". The test ouput however does produce
> >> rows for XPASS and XFAIL, and the existing TRS format already
> >> supports both.
> >>
> >> Since the TAP specification allows for custom directives [1], I was
> >> wondering if there was interest in a patch adding these to
> >> `lib/tap-driver.sh`. I would add "XPASS" and "XFAIL" directives, and
> >> if the test did not pass/fail as expected, it would result in an ERROR.
> >
> > Perhaps Autotest uses ERROR differently, but the better way to think
> > about it is expected-vs.-unexpected results. A test marked XFAIL that
> > fails is expected to fail, but a test marked XFAIL that passes is a
> > surprise XPASS.
> >
> > A better solution is to add a directive that declares that certain
> > tests are expected to fail. The TAP driver then matches the actual
> > results with the expected results.
> >
> >
> > -- Jacob
> >
> >
> >
>
>
