You all are looking only at the pull request builder. It just does one build to sanity-check a pull request, since that already takes 2 hours and would be prohibitive to build all configurations for every push. There is a different set of Jenkins jobs that periodically tests master against a lot more configurations, including Hadoop 2.4.
On Fri, May 15, 2015 at 6:02 PM, Frederick R Reiss <frre...@us.ibm.com> wrote: > The PR builder seems to be building against Hadoop 2.3. In the log for the > most recent successful build ( > https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32805/consoleFull > ) I see: > > ========================================================================= > Building Spark > ========================================================================= > [info] Compile with Hive 0.13.1 > [info] Building Spark with these arguments: -Pyarn -Phadoop-2.3 > -Dhadoop.version=2.3.0 -Pkinesis-asl -Phive -Phive-thriftserver > ... > ========================================================================= > Running Spark unit tests > ========================================================================= > [info] Running Spark tests with these arguments: -Pyarn -Phadoop-2.3 > -Dhadoop.version=2.3.0 -Pkinesis-asl test > > Is anyone testing individual pull requests against Hadoop 2.4 or 2.6 > before the code is declared "clean"? > > Fred > > [image: Inactive hide details for Ted Yu ---05/15/2015 09:29:09 > AM---Jenkins build against hadoop 2.4 has been unstable recently: https]Ted > Yu ---05/15/2015 09:29:09 AM---Jenkins build against hadoop 2.4 has been > unstable recently: https://amplab.cs.berkeley.edu/jenkins/ > > From: Ted Yu <yuzhih...@gmail.com> > To: Andrew Or <and...@databricks.com> > Cc: "dev@spark.apache.org" <dev@spark.apache.org> > Date: 05/15/2015 09:29 AM > Subject: Re: Recent Spark test failures > ------------------------------ > > > > Jenkins build against hadoop 2.4 has been unstable recently: > > *https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/* > <https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/> > > I haven't found the test which hung / failed in recent Jenkins builds. > > But PR builder has several green builds lately: > *https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/* > <https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/> > > Maybe PR builder doesn't build against hadoop 2.4 ? > > Cheers > > On Mon, May 11, 2015 at 1:11 PM, Ted Yu <*yuzhih...@gmail.com* > <yuzhih...@gmail.com>> wrote: > > Makes sense. > > Having high determinism in these tests would make Jenkins build stable. > > On Mon, May 11, 2015 at 1:08 PM, Andrew Or <*and...@databricks.com* > <and...@databricks.com>> wrote: > Hi Ted, > > Yes, those two options can be useful, but in general I think the > standard to set is that tests should never fail. It's actually the > worst if > tests fail sometimes but not others, because we can't reproduce them > deterministically. Using -M and -A actually tolerates flaky tests to a > certain extent, and I would prefer to instead increase the determinism > in > these tests. > > -Andrew > > 2015-05-08 17:56 GMT-07:00 Ted Yu <*yuzhih...@gmail.com* > <yuzhih...@gmail.com>>: > Andrew: > Do you think the -M and -A options described here can be used in > test runs ? > *http://scalatest.org/user_guide/using_the_runner* > <http://scalatest.org/user_guide/using_the_runner> > > Cheers > > On Wed, May 6, 2015 at 5:41 PM, Andrew Or < > *and...@databricks.com* <and...@databricks.com>> wrote: > Dear all, > > I'm sure you have all noticed that the Spark tests have been > fairly > unstable recently. I wanted to share a tool that I use to > track which tests > have been failing most often in order to prioritize fixing > these flaky > tests. > > Here is an output of the tool. This spreadsheet reports the > top 10 failed > tests this week (ending yesterday 5/5): > > > *https://docs.google.com/spreadsheets/d/1Iv_UDaTFGTMad1sOQ_s4ddWr6KD3PuFIHmTSzL7LSb4* > > <https://docs.google.com/spreadsheets/d/1Iv_UDaTFGTMad1sOQ_s4ddWr6KD3PuFIHmTSzL7LSb4> > > It is produced by a small project: > *https://github.com/andrewor14/spark-test-failures* > <https://github.com/andrewor14/spark-test-failures> > > I have been filing JIRAs on flaky tests based on this tool. > Hopefully we > can collectively stabilize the build a little more as we near > the release > for Spark 1.4. > > -Andrew > > > >
