On Sun, Apr 1, 2012 at 9:39 AM, Christopher Jones < christopher.jo...@oracle.com> wrote:
> > > On 3/31/12 2:38 PM, Alexey Shein wrote: > >> 1 апреля 2012 г. 0:27 пользователь Johannes Schlüter >> <johan...@schlueters.de> написал: >> >>> On Sat, 2012-03-31 at 13:21 +0500, Alexey Shein wrote: >>> >>>> By constantly publishing >>>> newsletter with failed / xfail bugs you're telling them "That's our >>>> current problems. Maybe you could help us with them". This way we >>>> could convert that discussing energy into some good patches. >>>> >>> >>> While many people will simply filter them out. At least that's my >>> experience with such automated mails in different projects ;-) >>> >>> johannes >>> >> >> Okay, let's find it out. I've created a poll here: >> https://wiki.php.net/xfail_**poll <https://wiki.php.net/xfail_poll>. >> Please, leave your voice. I'll close the poll in a week. Thank you. >> >> > Is there anything in the Jenkins work that makes this discussion irrelevant > (or more relevant)? What other ways should we be running & reviewing test > failures? > > currently XFAILs are handled as passing tests. I'm working on a solution for having email notifications about new test failures. There are 3 option, which I'm evaluating: 1, one can set thresholds if using the xunit plugin ( https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin) instead of the built-in junit plugin, which allows to fail the build if a new test failure is introduced. 2, the email-ext plugin allows you to set trigger for sending an email if a "regression"(new test failure), and also allows sending email for "improvement" (which is triggered when a previously failing test starts passing. 3, creating a custom post-build action, where we compare the current build's test result with the previous reports, and send out emails/mark the build failed if a new test failure is introduced. The problem with 1 is that it doesn't likes our junit.xml. First I debuged out that it doesn't support nested test suites, but that should be fixed with https://issues.jenkins-ci.org/browse/JENKINS-8460 I upgraded the plugin, but still no joy, I have to debug further. The problem with 2 is that the trigger only happens based on comparing the previous and current build's test result numbers. So if you introduce a new test failure and also fix a previously failing test, it won't fire the "regression" trigger. 3. seems to be the best solution, but requires the most work on our part. When we implement this (using one method, or another) we could change so that the XFAILs are handled from the jenkins POV as test failures, as they will be listed as such, but you still can see what are the ones failing since ages and what are the new ones. Jenkins provides the following informations regarding test failures: you can see the test failures introduced with a particular build through the build Status page ( http://ci.qa.php.net/job/php-src-5.3-matrix-build/architecture=x86,os=linux-debian-6.0/452/ for example) here you have a link to the Test Result page, plus the number of the failing tests, plus the sum of the test changes with that build (+-) under that line, Jenkins list the test failures introduced with that build under that you have "Show all failed tests >>>" which will expand the test list to include all failing tests, not just the new ones. jenkins also provides us with a Test Result Trend on the project page ( http://ci.qa.php.net/job/php-src-5.3-matrix-build/ for example), where you can toggle between showing all tests or just the failing ones. you can also see the history for a build for a config through the History link ( http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=x86,os=linux-debian-6.0/651/testReport/history/? for example) you can also see the total test results for a specific build per config ( http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=x86,os=linux-debian-6.0/651/testReport/junit/?) or aggregated ( http://ci.qa.php.net/job/php-src-5.4-matrix-build/651/testReport/?) you can also see a single test result history via selecting a test and clicking on the History link ( http://ci.qa.php.net/job/php-src-5.4-matrix-build/651/architecture=x86,os=linux-debian-6.0/testReport/php-src.ext.libxml/tests/004_phpt___libxml_set_streams_context__/history/? for example) here you can also see the status for that test in each build, so you can see when was that introduced or fixed. for example you can see that the libxml_set_streams_context() started failing after build number 641, where you can see the Changes mentions a libxml bugfix: http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=x86,os=linux-debian-6.0/641/changes having the email notification in place could make those kind of changes more visible. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
