On Thu, Nov 3, 2011 at 11:33 PM, Rasmus Lerdorf <ras...@lerdorf.com> wrote:
> On 11/03/2011 02:26 PM, Ferenc Kovacs wrote: > > We could set up the email notification any time, we would just have to > > agree where to send (internals, php-qa, creating a dedicated mailing > list?) > > and when to send (as I added to the RFC, having failing tests can be a > > problem from the notification POV). > > We have to find a good middle way, else we will either miss the > > notifications, or we will be overwhelmed, so people will just ignore it. > > Ideally they should go to internals if you can get them to be non-noisy. > true > > As long as it only reports a test status change from the previous run, > hopefully that wouldn't create much noise. yeah, and that is a little bit of a problem, because as I mentioned by default the jenkins way is that your tests passes: http://jenkins.361315.n4.nabble.com/Test-failure-baseline-td386476.html so we have three option as far as I can see: A, it will report every test failure, but we fix all of our tests so we are cool. (my preference) B, same as A, but until we fix all of the tests, we mark the failing test as XFAIL, which won't be counted as failed test (I'm against it, as time taught us that we are lazy, and XFAILs will be kept that way. :/) C, Somebody(me) do some scripting, and our currently failing tests won't fail the build, only make it unstable, but any other test failure will fail the build. I think I didn't explained in the RFC, because currently we don't use it, but in jenkins there is a third result type for a build apart success and failure: unstable. Most plugins support this, as does the email notification plugin also, so this should also work. -- Ferenc Kovács @Tyr43l - http://tyrael.hu
