Dawid:
With the new test runner you created, would it be possible to setup an annotation that we could use instead to indicate that a test should in fact be run, and if it fails, include the failure info in the build report, but do not fail the build?
I'm thinking in particular about some of the test that multi-threaded tests that are currently marked @Ignore or @AwaitsFix because they sporadically fail on jenkins in our jail -- but that people haven't been able to reproduce consistently on local dev machines (or that some people have been able to reproduce, but not the people who understand the tests/code well enough to try and fix)
As it stands right now, if somone wants to try and fix a complicated test that's disabled, they have to make a guess at the fix, un-@Ignore, then watch the next few/several builds patiently to see if / how-often it fails, then commit the @Ignore back, and repeat.
If we could leave these tests running on every build, then we could at least monitor the relative frequency of the failures -- ie: "last week testFoo failed in 10% of the builds, this week it fails in every build, so somebody definiteily broke something" or "last week testFoor failed in 10% of the builds, and after my attempted hardening it only fails in 5% of the builds so i may be on to something."
what do folks think? -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
