On Sun, May 6, 2012 at 3:38 PM, Dawid Weiss <dawid.we...@cs.put.poznan.pl> wrote: > I also admit I've never seen anything like > this -- a suite of tests with an allowed failure ratio over time and a > threshold that would trigger a warning...
Not so much an "allowed" failure rate... more of "it fails sometimes and no one has had the time to try to get it to pass with a greater percentage of time". And even when people put effort into get it to pass more often, it's still not 100%. As those tests exist now, there are a few choices a) turn them off (this is bad because it seriously decreases coverage) b) somehow deal with the intermittent failures Given that we're not running on a realtime system, the fact that many higher level tests have timing and scheduling dependencies means that we will never achieve a 100% pass rate on such tests. > These are weird tests if they allow for a (predictable?) failure from > time to time. I don't say it's a bad concept, but I think unit tests > may not be a good framework for handling this. Yeah, these aren't really unit tests. Should we try to move them somewhere else? Or run them separately and email the results to a different list? -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10 --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
