On Wed, Sep 7, 2011 at 5:28 PM, Diego Novillo <dnovi...@google.com> wrote: > One of the most vexing aspects of GCC development is dealing with > failures in the various testsuites. In general, we are unable to > keep failures down to zero. We tolerate some failures and tell > people to "compare your build against a clean build". > > This forces developers to either double their testing time by > building the compiler twice or search in gcc-testresults and hope > to find a relatively similar build to compare against. > > Additionally, the marking mechanisms in DejaGNU are generally > cumbersome and hard to add. Even worse, depending on the > controlling script, there may not be an XFAIL marker at all. > > So, while we would ideally keep NO failures in the testsuite, the > reality is that we are content with having KNOWN failures. For a > given set of failures out of 'make check', I would like to have a > simple filtering mechanism that prunes the known failures out. > > Desired features: > > - List of known failures lives in SVN. > - Each target can have its own list. > - Supports ignoring FAIL, UNRESOLVED and XPASS results. > - Supports pattern matching to glob sets of failures. > - Co-exists with the existing XFAIL support in DejaGNU. > - Supports flaky tests. > - Supports timestamps to avoid having tests in a knonw-to-fail > state forever. > > In terms of implementation, this filter could be part of 'make > check'. We'd pipe make check's output to it and it would decide > whether to emit FAIL/UNRESOLVED/XPASS lines based on the black > list. > > I could also make this a post-check filter that runs on all the > generated <tool>.sum files. The filter could live in > <src>/contrib and be used on demand. > > I am not thrilled about the prospect of implementing this in > DejaGNU directly. > > Thoughts?
I think it would be more useful to have a script parse gcc-testresults@ postings from the various autotesters and produce a nice webpage with revisions and known FAIL/XPASSes for the target triplets that are tested. That's been a long time on my TODO list, but my web/script FU is weak enough that I've been pushing that back. Maybe you have some web-stuff-capable folks at Google even? ;) Richard. > > Thanks. Diego. >
