ok, i did some testing and found out what's happening. https://issues.apache.org/jira/browse/SPARK-4021
here's the TL;DR: jenkins ignores what JDKs are installed via the web interface when there's more than one defined, and falls back to whatever is default on the slave the test is run on. in this case, it's openjdk 7u65... and spark compilation fails. i've removed the 2nd JDK (7u71) from jenkins, and everything is back to normal. On Tue, Oct 21, 2014 at 11:51 AM, shane knapp <skn...@berkeley.edu> wrote: > i'm currently in a meeting and will be starting to do some tests in ~1 > hour or so. > > On Tue, Oct 21, 2014 at 11:07 AM, Nan Zhu <zhunanmcg...@gmail.com> wrote: > >> I agree with Sean >> >> I just compiled spark core successfully with 7u71 in Mac OS X >> >> On Tue, Oct 21, 2014 at 1:11 PM, Josh Rosen <rosenvi...@gmail.com> wrote: >> >>> Ah, that makes sense. I had forgotten that there was a JIRA for this: >>> >>> https://issues.apache.org/jira/browse/SPARK-4021 >>> >>> On October 21, 2014 at 10:08:58 AM, Patrick Wendell (pwend...@gmail.com) >>> wrote: >>> >>> Josh - the errors that broke our build indicated that JDK5 was being >>> used. Somehow the upgrade caused our build to use a much older Java >>> version. See the JIRA for more details. >>> >>> On Tue, Oct 21, 2014 at 10:05 AM, Josh Rosen <rosenvi...@gmail.com> >>> wrote: >>> > I find it concerning that there's a JDK version that breaks out build, >>> since >>> > we're supposed to support Java 7. Is 7u71 an upgrade or downgrade from >>> the >>> > JDK that we used before? Is there an easy way to fix our build so that >>> it >>> > compiles with 7u71's stricter settings? >>> > >>> > I'm not sure why the "New" PRB is failing here. It was originally >>> created >>> > as a clone of the main pull request builder job. I checked the >>> configuration >>> > history and confirmed that there aren't any settings that we've >>> forgotten to >>> > copy over (e.g. their configurations haven't diverged), so I'm not >>> sure >>> > what's causing this. >>> > >>> > - Josh >>> > >>> > On October 21, 2014 at 6:35:39 AM, Nan Zhu (zhunanmcg...@gmail.com) >>> wrote: >>> > >>> > weird.....two buildings (one triggered by New, one triggered by Old) >>> were >>> > executed in the same node, amp-jenkins-slave-01, one compiles, one >>> not... >>> > >>> > Best, >>> > >>> > -- >>> > Nan Zhu >>> > >>> > >>> > On Tuesday, October 21, 2014 at 9:39 AM, Nan Zhu wrote: >>> > >>> >> seems that all PRs built by NewSparkPRBuilder suffers from 7u71, >>> while >>> >> SparkPRBuilder is working fine >>> >> >>> >> Best, >>> >> >>> >> -- >>> >> Nan Zhu >>> >> >>> >> >>> >> On Tuesday, October 21, 2014 at 9:22 AM, Cheng Lian wrote: >>> >> >>> >> > It's a new pull request builder written by Josh, integrated into >>> our >>> >> > state-of-the-art PR dashboard :) >>> >> > >>> >> > On 10/21/14 9:33 PM, Nan Zhu wrote: >>> >> > > just curious...what is this "NewSparkPullRequestBuilder"? >>> >> > > >>> >> > > Best, >>> >> > > >>> >> > > -- >>> >> > > Nan Zhu >>> >> > > >>> >> > > >>> >> > > On Tuesday, October 21, 2014 at 8:30 AM, Cheng Lian wrote: >>> >> > > >>> >> > > > >>> >> > > > Hm, seems that 7u71 comes back again. Observed similar Kinesis >>> >> > > > compilation error just now: >>> >> > > > >>> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/410/consoleFull >>> >> > > > >>> >> > > > >>> >> > > > Checked Jenkins slave nodes, saw /usr/java/latest points to >>> >> > > > jdk1.7.0_71. However, /usr/bin/javac -version says: >>> >> > > > >>> >> > > > > >>> >> > > > > Eclipse Java Compiler 0.894_R34x, 3.4.2 release, Copyright >>> IBM >>> >> > > > > Corp 2000, 2008. All rights reserved. >>> >> > > > > >>> >> > > > >>> >> > > > >>> >> > > > Which JDK is actually used by Jenkins? >>> >> > > > >>> >> > > > >>> >> > > > Cheng >>> >> > > > >>> >> > > > >>> >> > > > On 10/21/14 8:28 AM, shane knapp wrote: >>> >> > > > >>> >> > > > > ok, so earlier today i installed a 2nd JDK within jenkins >>> (7u71), >>> >> > > > > which fixed the SparkR build but apparently made Spark itself >>> quite unhappy. >>> >> > > > > i removed that JDK, triggered a build ( >>> >> > > > > >>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21943/console), >>> >>> >> > > > > and it compiled kinesis w/o dying a fiery death. apparently >>> 7u71 is stricter >>> >> > > > > when compiling. sad times. sorry about that! shane On Mon, >>> Oct 20, 2014 at >>> >> > > > > 5:16 PM, Patrick Wendell <pwend...@gmail.com> (mailto: >>> pwend...@gmail.com) >>> >> > > > > wrote: >>> >> > > > > > The failure is in the Kinesis compoent, can you reproduce >>> this >>> >> > > > > > if you build with -Pkinesis-asl? - Patrick On Mon, Oct 20, >>> 2014 at 5:08 PM, >>> >> > > > > > shane knapp <skn...@berkeley.edu> (mailto: >>> skn...@berkeley.edu) wrote: >>> >> > > > > > > hmm, strange. i'll take a look. On Mon, Oct 20, 2014 at >>> 5:11 >>> >> > > > > > > PM, Nan Zhu <zhunanmcg...@gmail.com> (mailto: >>> zhunanmcg...@gmail.com) wrote: >>> >> > > > > > > > yes, I can compile locally, too but it seems that >>> Jenkins is >>> >> > > > > > > > not happy now... >>> >> > > > > > > > >>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/ All >>> >> > > > > > > > failed to compile Best, -- Nan Zhu On Monday, October >>> 20, 2014 at 7:56 PM, >>> >> > > > > > > > Ted Yu wrote: >>> >> > > > > > > > > I performed build on latest master branch but didn't >>> get >>> >> > > > > > > > > compilation >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > error. >>> >> > > > > > > > > FYI On Mon, Oct 20, 2014 at 3:51 PM, Nan Zhu >>> >> > > > > > > > > <zhunanmcg...@gmail.com (mailto: >>> zhunanmcg...@gmail.com) >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > (mailto:zhunanmcg...@gmail.com)> wrote: >>> >> > > > > > > > > > Hi, I just submitted a patch >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > https://github.com/apache/spark/pull/2864/files >>> >> > > > > > > > > > with one line change but the Jenkins told me it's >>> failed >>> >> > > > > > > > > > to compile on the unrelated >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > > files? >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21935/console >>> >> > > > > > > > > > Best, Nan >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > >>> >> > > >>> >> > >>> >> >>> > >>> >>> >> >
