I agree that a test that frequently fails, and does not get fixed, is nearly pointless: everybody ignores it so it's as if the test didn't exist. And so it should be disabled.
I say *nearly* because the failures are in fact useful to devs who do have the itch/time to debug/fix them. So I think we need some middle ground here, where the tests keep failing but only those that are interested in the failures see the notifications. We need to switch from a "push" model (any failure is broadcast to everybody) to a "pull" model (those devs that want to debug the failures go and check the logs), for such tests. When someone wants to make sure their change didn't break something (Erick's original use case) then these tests should not run. I like Dawid's idea (a separate test plan that Jenkins runs with these "difficult" tests, and it wouldn't email dev on failure). Mike McCandless http://blog.mikemccandless.com On Mon, Sep 17, 2012 at 7:58 AM, Robert Muir <[email protected]> wrote: > On Sun, Sep 16, 2012 at 11:10 PM, Mark Miller <[email protected]> wrote: >> I get value from this test - if it was disabled, I'd probably re-enable it. >> would be great if it didn't fail so much, but the type of fail tells me >> something. > > That means the assert in question isnt important at all. I'll remove it. > > Again my problem is the idea that having a failing build is "ok" > because certain types of failures "don't matter". If they dont matter > they should be removed. > > It causes a ton of noise when people are lazy about tests in this way, > and it wastes a ton of peoples time. R > > Remember every time one of these tests fails it sends an email, that I > must read (we don't yet have a way to put in the subject header its a > SOLR test fail versus a LUCENE one, or i'd filter the solr ones and > not be complaining as much). > > -- > lucidworks.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
