On Mon, Nov 11, 2013 at 06:38:15AM -0800, Andi Kleen wrote: > Jeff Law <l...@redhat.com> writes: > > > Thoughts or comments? > > If noone tests java completely then it will quickly bitrot won't it? > > So ideally some bot would still regularly build/test it. > If you don't do that you could as well just remove the code. > > The underlying problem seems to be the requirement for each > contributor to run all the tests themselves. Over time the gcc > test suite has become longer and longer, so it becomes a bigger > burden. But then they still don't test everything, because > of optional components (e.g. not everyone has Ada installed). > Also some tests are just unreliable and do not give good results. > Also if patch breaks ia64 it will be hardly noticed.
There already are plenty of machines at compile farm so problem lies in writing a bot. > Perhaps it would make more sense to just split the required tests : > > - quick tests everyone is expected to run before commit (bootstrap + > test suite <10/20/30min?) > That could be some subset of what runs today. Maybe bootstrap > and most of the C/C++ test suite. > - This should be only tests that always pass, not the usual noise that > is there today (e.g. not that guality random generator) > - longer tests that run in centralized bots > - bots automatically bisect regressions > - clear policies that if someone causes a bot regression > they are on the hook to fix it in a given time, or the patch gets > reverted. > - Get rid of all tests that are not reliable. > - With bots it would be also quite feasible to have even > longer tests, that run hours. For example you could add > more slow static code analysis steps. > - With bots you can also have nice central dash boards that give the current > state, instead of the current "hunt random message on gcc-testresults" > This also reminds me similar proposal of shifting burden of testcase writing to users. Problem with adding user testcase is paperwork, so there would be separate codebase with user tests. It would work that if in bugzilla there is a bug with a reliable reproducer then commiter add reproducer with bug number to database. It would trigger a bot to do bisection what change introduced problem. These will be checked by bots and when there is a failure on closed bug it will be reopened. Also having a bot would allow adding benchmarks that take hours until variance gets to reasonable level.
