On Thu, Sep 8, 2011 at 8:31 PM, Richard Guenther <richard.guent...@gmail.com> wrote: > On Wed, Sep 7, 2011 at 5:28 PM, Diego Novillo <dnovi...@google.com> wrote: >> One of the most vexing aspects of GCC development is dealing with >> failures in the various testsuites. In general, we are unable to >> keep failures down to zero. We tolerate some failures and tell >> people to "compare your build against a clean build". >> >> This forces developers to either double their testing time by >> building the compiler twice or search in gcc-testresults and hope >> to find a relatively similar build to compare against. >> >> Additionally, the marking mechanisms in DejaGNU are generally >> cumbersome and hard to add. Even worse, depending on the >> controlling script, there may not be an XFAIL marker at all. >> >> So, while we would ideally keep NO failures in the testsuite, the >> reality is that we are content with having KNOWN failures. For a >> given set of failures out of 'make check', I would like to have a >> simple filtering mechanism that prunes the known failures out. >> >> Desired features: >> >> - List of known failures lives in SVN. >> - Each target can have its own list. >> - Supports ignoring FAIL, UNRESOLVED and XPASS results. >> - Supports pattern matching to glob sets of failures. >> - Co-exists with the existing XFAIL support in DejaGNU. >> - Supports flaky tests. >> - Supports timestamps to avoid having tests in a knonw-to-fail >> state forever. >> >> In terms of implementation, this filter could be part of 'make >> check'. We'd pipe make check's output to it and it would decide >> whether to emit FAIL/UNRESOLVED/XPASS lines based on the black >> list. >> >> I could also make this a post-check filter that runs on all the >> generated <tool>.sum files. The filter could live in >> <src>/contrib and be used on demand. >> >> I am not thrilled about the prospect of implementing this in >> DejaGNU directly. >> >> Thoughts? > > I think it would be more useful to have a script parse gcc-testresults@ > postings from the various autotesters and produce a nice webpage > with revisions and known FAIL/XPASSes for the target triplets that > are tested. > > That's been a long time on my TODO list, but my web/script FU is > weak enough that I've been pushing that back.
I have something along those lines for the Linaro releases: http://ex.seabright.co.nz/helpers/testcompare/gcc-linaro-4.6-2011.08/logs/armv7l-natty-cbuild162-ursa1-cortexa9r1/gcc-testsuite.txt?base=gcc-linaro-4.6-2011.07-0 and a lower level diff-on-sum-files for each commit: http://builds.linaro.org/toolchain/gcc-linaro-4.5+bzr99541~rsandifo~lp823708-4.5/logs/armv7l-natty-cbuild181-ursa4-armv5r2/testsuite-diff.txt http://builds.linaro.org/toolchain/gcc-linaro-4.6+bzr106801~ams-codesourcery~merge-from-fsf-20110908-4.6/logs/x86_64-natty-cbuild181-oort1-x86_64r1/testsuite-diff.txt They're both a hack and only work against local files. The code is available at: https://launchpad.net/tcwg-web and: https://launchpad.net/cbuild They're both similar to contrib/compare_results but webified and hooked into our auto builders. -- Michael
