On 21/09/17 02:27, Joseph Myers wrote: > On Wed, 20 Sep 2017, Segher Boessenkool wrote: > >>> - buildbot can notify people if the build fails or if there's a test >>> regression. Notification can be sent to IRC and email for example. What >>> would people prefer to have as the settings for notifications? >> >> Just try it! IRC is most useful I think, at least for now. But try >> whatever seems useful, if there are too many complaints you can always >> turn it off again ;-) > > We have the gcc-regression list for automatic regression notifications > (but as per my previous message, regression notifications are much more > useful if someone actually does the analysis, promptly, to identify the > cause and possibly a fix). >
Yes, the gcc-regression list. Will add a notifier to email the list. > My glibc bots only detect testsuite regressions that change the exit > status of "make check" from successful to unsuccessful (and regressions > that break any part of the build, etc.). That works for glibc, where the > vast bulk of configurations have clean compilation-only testsuite results. > It won't work for GCC - you need to detect changes in the results of > individual tests (or new tests which start out as FAIL and may not > strictly be regressions but should still be fixed). Ideally the expected > baseline *would* be zero FAILs, but we're a long way off that. > Yes, with GCC is slightly more complex but it should be possible to calculate regressions even in the presence of non-zero FAILs. Thanks for your comments, -- Paulo Matos
