I think we can even integrate hossman's suggestion and generate a stability report like weekly or something.
I will take a look at this this week but it is definitely something that will require everyone's consensus. Dawid Sent from mobile phone. On Sep 17, 2012 2:42 PM, "Michael McCandless" <[email protected]> wrote: > I agree that a test that frequently fails, and does not get fixed, is > nearly pointless: everybody ignores it so it's as if the test didn't > exist. And so it should be disabled. > > I say *nearly* because the failures are in fact useful to devs who do > have the itch/time to debug/fix them. > > So I think we need some middle ground here, where the tests keep > failing but only those that are interested in the failures see the > notifications. We need to switch from a "push" model (any failure is > broadcast to everybody) to a "pull" model (those devs that want to > debug the failures go and check the logs), for such tests. > > When someone wants to make sure their change didn't break something > (Erick's original use case) then these tests should not run. > > I like Dawid's idea (a separate test plan that Jenkins runs with these > "difficult" tests, and it wouldn't email dev on failure). > > Mike McCandless > > http://blog.mikemccandless.com > > On Mon, Sep 17, 2012 at 7:58 AM, Robert Muir <[email protected]> wrote: > > On Sun, Sep 16, 2012 at 11:10 PM, Mark Miller <[email protected]> > wrote: > >> I get value from this test - if it was disabled, I'd probably re-enable > it. > >> would be great if it didn't fail so much, but the type of fail tells me > >> something. > > > > That means the assert in question isnt important at all. I'll remove it. > > > > Again my problem is the idea that having a failing build is "ok" > > because certain types of failures "don't matter". If they dont matter > > they should be removed. > > > > It causes a ton of noise when people are lazy about tests in this way, > > and it wastes a ton of peoples time. R > > > > Remember every time one of these tests fails it sends an email, that I > > must read (we don't yet have a way to put in the subject header its a > > SOLR test fail versus a LUCENE one, or i'd filter the solr ones and > > not be complaining as much). > > > > -- > > lucidworks.com > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
