One of the most vexing aspects of GCC development is dealing with failures in the various testsuites. In general, we are unable to keep failures down to zero. We tolerate some failures and tell people to "compare your build against a clean build".
This forces developers to either double their testing time by building the compiler twice or search in gcc-testresults and hope to find a relatively similar build to compare against. Additionally, the marking mechanisms in DejaGNU are generally cumbersome and hard to add. Even worse, depending on the controlling script, there may not be an XFAIL marker at all. So, while we would ideally keep NO failures in the testsuite, the reality is that we are content with having KNOWN failures. For a given set of failures out of 'make check', I would like to have a simple filtering mechanism that prunes the known failures out. Desired features: - List of known failures lives in SVN. - Each target can have its own list. - Supports ignoring FAIL, UNRESOLVED and XPASS results. - Supports pattern matching to glob sets of failures. - Co-exists with the existing XFAIL support in DejaGNU. - Supports flaky tests. - Supports timestamps to avoid having tests in a knonw-to-fail state forever. In terms of implementation, this filter could be part of 'make check'. We'd pipe make check's output to it and it would decide whether to emit FAIL/UNRESOLVED/XPASS lines based on the black list. I could also make this a post-check filter that runs on all the generated <tool>.sum files. The filter could live in <src>/contrib and be used on demand. I am not thrilled about the prospect of implementing this in DejaGNU directly. Thoughts? Thanks. Diego.
