The tests that Ufuk is referring to are not deterministically failing. This is about hard to debug and hard to fix tests where it is not clear who broke them.
Fixing such a test can take a several days or even more… So locking the master branch is not an option IMO. Deactivating the tests will lower the test coverage such that bugs that would have been caught by an test that infrequently fails for another reason, are not identified. How about opening JIRAs for non-deterministically failing tests and assigning a special label for that. Whenever, a test fails one can check if this is a known issue and act correspondingly. From: Matthias J. Sax Sent: Thursday, 4. June, 2015 09:06 To: dev@flink.apache.org I think, people should be forced to fixed failing tests asap. One way to go, could be to lock the master branch until the test is fixed. If nobody can push to the master, pressure is very high for the responsible developer to get it done asap. Not sure if this is Apache compatible. Just a thought (from industry experience). On 06/04/2015 08:10 AM, Aljoscha Krettek wrote: > I tend to agree with Ufuk, although it would be nice to fix them very quickly. > > On Thu, Jun 4, 2015 at 1:26 AM, Stephan Ewen <se...@apache.org> wrote: >> @matthias: That is the implicit policy right now. Seems not to work... >> >> On Thu, Jun 4, 2015 at 12:40 AM, Matthias J. Sax < >> mj...@informatik.hu-berlin.de> wrote: >> >>> I basically agree that the current policy on not optimal. However, I >>> would rather give failing tests "top priority" to get fixed (if possible >>> within one/a-few days) and not disable them. >>> >>> -Matthias >>> >>> On 06/04/2015 12:32 AM, Ufuk Celebi wrote: >>>> Hey all, >>>> >>>> we have certain test cases, which are failing regularly on Travis. In all >>>> cases I can think of we just keep the test activated. >>>> >>>> I think this makes it very hard for regular contributors to take these >>>> failures seriously. I think the following situation is not unrealistic >>> with >>>> the current policy: I know that test X is failing. I don't know that >>> person >>>> Y fixed this test. I see test X failing (again for a different reason) >>> and >>>> think that it is a "known issue". >>>> >>>> I think a better policy is to just disable the test, assign someone to >>> fix >>>> it, and then only enable it again after someone has fixed it. >>>> >>>> Is this reasonable? Or do we have good reasons to keep such tests (there >>>> are currently one or two) activated? >>>> >>>> – Ufuk >>>> >>> >>> >
