On 10/06/15 20:22, Norbert Thiebaud wrote: > - a test auto-induced one: when a test is unstable and produce random > failures based on circumstances... the infamous 'heisenbugs' > and heisenbug can be a systemic/design problem or can be a real bug > that is hard to trigger. either way these are not useful, and in fact > harmful in a ci context; because the human nature is 'If you can't > reproduce it is not a bug' > so the later category of real hard to trigger bug is always labeled > 'systemic error' and ignored anyway... and it make people numb to > errors... > For automated testing, trust is paramount: heisenbug test failure are > the enemy, false non-failure is bad but actually less painful
Would it make sense to have a server dedicated to Heisenbugs? If a test triggers a heisenbug, disable it on most of them but try to instrument the heisenbug tester up the wazoo so that when it fails, there's a pile of logs to try and work out what went wrong. And of course, it doesn't spam failure reports generally unless someone asks for them (the person to whom the bug is assigned should get them, of course), but it does save those logs for forensic analysis. Cheers, Wol _______________________________________________ LibreOffice mailing list LibreOffice@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice
