On Wed, 20 Feb 2019 at 14:51, Jakub Jelinek <ja...@redhat.com> wrote: [..]
> The aim is to avoid regressions, not zero FAIL rate, which e.g. for the > guality testcase is not really possible as the testing matrix is too large > for debug info coverage, -O levels x targets x ISA choices x GDB issues > and it is impossible to encode that into xfails. > Plus, the rpm builds test both normally and with additional > -fstack-protector-strong as that is what the whole distro does, while > normally the latter isn't included. So, some FAILs might be once with > normal options and once with -fstack-protector-strong, other tests only > FAIL > with the latter e.g. when testing asan and having UB in the test to test it > how asan handles it and -fstack-protector-strong may stand in a way of > that. > There are some flakey tests too (mostly in the go testsuite). > > So, I have a directory with years of testsuite results from koji/brew > builds, essentially build logs from all the architectures massaged through > sed -n -e '/^==*TESTING/,/^==*TESTING END/p' > and compare that regularly for new builds. > OK. Jakub so what kind of method you are using to recognise that something is wrong with gcc/gdb/binutils if none of those packages test suites seems may exit with non-zero exit code and some failures are perfectly OK? (Or maybe logic of the suite should be negated?) Do you have some sort of script/tool extracting results from build logs? or it is some other method? Syntax of the content /^==*TESTING/,/^==*TESTING END/p' looks a bit complicated .. How to trace those results in more or less automated way? kloczek -- Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
