Re: Failing tests policy

2015-06-04 Thread Matthias J. Sax
I agree. It does not help with the current unstable tests. However, I can help to prevent to run into instability issues in the future. On 06/04/2015 11:58 AM, Fabian Hueske wrote: > I think the problem is less with bugs being introduced by new commits but > rather bugs which are already in the co

Re: Failing tests policy

2015-06-04 Thread Fabian Hueske
I think the problem is less with bugs being introduced by new commits but rather bugs which are already in the code base. 2015-06-04 11:52 GMT+02:00 Matthias J. Sax : > I have another idea: the problem is, that some commit might de-stabilize > a former stable test. This in not detected, because t

Re: Failing tests policy

2015-06-04 Thread Matthias J. Sax
I have another idea: the problem is, that some commit might de-stabilize a former stable test. This in not detected, because the build was ("accidentally") green and the code in merged. We could reduce the probability that this happens, if a pull request must pass the test-run multiple times (mayb

Re: Failing tests policy

2015-06-04 Thread Ufuk Celebi
Thanks for the feedback and the suggestions. As Stephan said, the "we have to fix it asap" usually does not work well. I think blocking master is not an option, exactly for the reasons that Fabian and Till outlined. From the comments so far, I don't feel like we are eager to adapt a disable po

Re: Failing tests policy

2015-06-04 Thread fhueske
The tests that Ufuk is referring to are not deterministically failing. This is about hard to debug and hard to fix tests where it is not clear who broke them. Fixing such a test can take a several days or even moreā€¦ So locking the master branch is not an option IMO. Deactivating the tests wi

Re: Failing tests policy

2015-06-04 Thread Till Rohrmann
I'm also in favour of quickly fixing the failing test cases but I think that blocking the master is a kind of drastic measure. IMO this creates a culture of blaming someone whereas I would prefer a more proactive approach. When you see a failing test case and know that someone recently worked on it

Re: Failing tests policy

2015-06-04 Thread Matthias J. Sax
I think, people should be forced to fixed failing tests asap. One way to go, could be to lock the master branch until the test is fixed. If nobody can push to the master, pressure is very high for the responsible developer to get it done asap. Not sure if this is Apache compatible. Just a thought

Re: Failing tests policy

2015-06-03 Thread Aljoscha Krettek
I tend to agree with Ufuk, although it would be nice to fix them very quickly. On Thu, Jun 4, 2015 at 1:26 AM, Stephan Ewen wrote: > @matthias: That is the implicit policy right now. Seems not to work... > > On Thu, Jun 4, 2015 at 12:40 AM, Matthias J. Sax < > mj...@informatik.hu-berlin.de> wrote

Re: Failing tests policy

2015-06-03 Thread Stephan Ewen
@matthias: That is the implicit policy right now. Seems not to work... On Thu, Jun 4, 2015 at 12:40 AM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > I basically agree that the current policy on not optimal. However, I > would rather give failing tests "top priority" to get fixed (if

Re: Failing tests policy

2015-06-03 Thread Matthias J. Sax
I basically agree that the current policy on not optimal. However, I would rather give failing tests "top priority" to get fixed (if possible within one/a-few days) and not disable them. -Matthias On 06/04/2015 12:32 AM, Ufuk Celebi wrote: > Hey all, > > we have certain test cases, which are fai

Failing tests policy

2015-06-03 Thread Ufuk Celebi
Hey all, we have certain test cases, which are failing regularly on Travis. In all cases I can think of we just keep the test activated. I think this makes it very hard for regular contributors to take these failures seriously. I think the following situation is not unrealistic with the current p