There's far too many flaky tests to enforce a clean test run prior to commit. However, there also needs to be a very conscious effort to fix flaky tests - otherwise, we're not going to reach the state where this can be enforced. I'm also not sure whether this is just flaky tests, or whether there are real problems. Maybe a few volunteers could drive the test fixes, and as an artifact of this, come up with guidelines on writing tests (avoid stats comparisons in qfiles, skip lines which may be environment specific, introduce a new compare mechanism for lines which may be in a different order based on internal structures, etc).
On Thu, Mar 9, 2017 at 2:39 PM, Ashutosh Chauhan <hashut...@apache.org> wrote: > I went through all the builds of last 24 hours and though there are couple > of green runs, we still have quite a bit of flakiness in our tests. Most of > that is captured via jiras on : > https://issues.apache.org/jira/browse/HIVE-15058 Till we have that > flakiness I am not sure if we can enforce no commit on test failures > policy. Once flakiness is straightened out then this discussion will become > moot anyways. > Towards that I think this flaky detector job is a step in right direction. > > > On Thu, Mar 9, 2017 at 11:38 AM, Sergio Pena <sergio.p...@cloudera.com> > wrote: > > > - Probably avoiding committing a patch if a flaky test is shown on the > test > > results? > > - Should we add a jenkins job that checks for flaky tests like the hbase > > project did? > > https://builds.apache.org/view/H-L/view/HBase/job/HBase- > Find-Flaky-Tests/ > > > > On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan <hashut...@apache.org> > > wrote: > > > > > Great news! Thanks to everyone who contributed in getting our tests and > > > test infra sorted out. > > > We would definitely want to keep the status either green or blue > > definitely > > > not red :) All our previous efforts in keeping builds green didn't bear > > > fruit. > > > So, I think we need to make some changes here. > > > > > > Any ideas what we can do to ensure green builds going forward? > > > > > > Thanks, > > > Ashutosh > > > > > > On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <sergio.p...@cloudera.com> > > > wrote: > > > > > > > It's actually blue Peter :). > > > > > > > > But good job, I see that the console output is: > > > > > > > > {color:red}ERROR:{color} -1 due to no test(s) being added or > modified. > > > > > > > > {color:green}SUCCESS:{color} +1 due to 10336 tests passed > > > > > > > > > > > > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com> > wrote: > > > > > > > > > Hi, > > > > > > > > > > Congratulations for everyone who have helped taking care of the > unit > > > test > > > > > failures! > > > > > I have got my first green run! :) > > > > > > > > > > If any of you interested in: > > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/ > testReport/ > > < > > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/ > testReport/> > > > :) > > > > > > > > > > Great day, and again thanks everyone! > > > > > > > > > > Peter > > > > > > > > > >
