Great Idea. In addition, I would like to propose : Strictly tests (especially CI tests) must not be flaky. It really doesn't carry much use. As everyone tends to ignore them. Step-1: Separate them into a different pool, and you won't get a red mark for failing this pool/group. Step-2: Fix each test to be consistent and move it out of the optional pool. Step-3: If there are any tests that can't be consistent but makes sense (???) to keep them part of CI, we can leave them in this flaky/optional pool.
Thanks, JV On Wed, Jan 24, 2018 at 8:56 AM, Sijie Guo <guosi...@gmail.com> wrote: > On Wed, Jan 24, 2018 at 5:20 AM Ivan Kelly <iv...@apache.org> wrote: > > > Hi folks, > > > > Looking at open PRs right now, there's a lot of red Xs > > > > > > https://screenshotscdn.firefoxusercontent.com/images/ > 8df9daf7-d71a-4988-a9cc-69d9c10c2636.png > > > > I see only 3 green. I know it's not all flakes, but a good amount of > > it is. Enough to make a red X meaningless. > > > > So I propose we enable the rerunFailingTest option in surefire. This > > will rerun the flakes. I suggest we set to 2. Anything that it flaking > > more than once per run needs to be looked at. > > > > > > http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html# > rerunFailingTestsCount > > > Good idea. +1 > > <http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html# > rerunFailingTestsCount> > > > > As part of this, I suggest that we also institute a no submission on > > red policy (i.e. tests have to get full green to be merged). > > > FYI. I created one issue for same purpose before: > > https://github.com/apache/bookkeeper/issues/1031 > > > > > > > Regards, > > Ivan > > > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
