Re: Test and support only LTS JDK release?

2018-11-07 Thread Tom Graves
+1 seems reasonable at this point. Tom On Tuesday, November 6, 2018, 1:24:16 PM CST, DB Tsai wrote: Given Oracle's new 6-month release model, I feel the only realistic option is to only test and support JDK such as JDK 11 LTS and future LTS release. I would like to have a discussion o

Re: Test and support only LTS JDK release?

2018-11-07 Thread Stavros Kontopoulos
nterprise-java-support-options/ > > Kazuaki Ishizaki, > > > > From:Marcelo Vanzin > To:Felix Cheung > Cc:Ryan Blue , sn...@snazy.de, dev < > dev@spark.apache.org>, Cesar Delgado > Date: 2018/11/07 08:29 > Subject:

Re: Test and support only LTS JDK release?

2018-11-07 Thread Kazuaki Ishizaki
:29 Subject:Re: Test and support only LTS JDK release? https://www.oracle.com/technetwork/java/javase/eol-135779.html On Tue, Nov 6, 2018 at 2:56 PM Felix Cheung wrote: > > Is there a list of LTS release that I can reference? > > > &g

Re: Test and support only LTS JDK release?

2018-11-06 Thread Marcelo Vanzin
To: sn...@snazy.de > Cc: Spark Dev List; cdelg...@apple.com > Subject: Re: Test and support only LTS JDK release? > > +1 for supporting LTS releases. > > On Tue, Nov 6, 2018 at 11:48 AM Robert Stupp wrote: >> >> +1 on supporting LTS releases. >> >> VM di

Re: Test and support only LTS JDK release?

2018-11-06 Thread Felix Cheung
Is there a list of LTS release that I can reference? From: Ryan Blue Sent: Tuesday, November 6, 2018 1:28 PM To: sn...@snazy.de Cc: Spark Dev List; cdelg...@apple.com Subject: Re: Test and support only LTS JDK release? +1 for supporting LTS releases. On Tue

Re: Test and support only LTS JDK release?

2018-11-06 Thread Ryan Blue
+1 for supporting LTS releases. On Tue, Nov 6, 2018 at 11:48 AM Robert Stupp wrote: > +1 on supporting LTS releases. > > VM distributors (RedHat, Azul - to name two) want to provide patches to > LTS versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/). > How that will play out in

Re: Test and support only LTS JDK release?

2018-11-06 Thread Robert Stupp
+1 on supporting LTS releases. VM distributors (RedHat, Azul - to name two) want to provide patches to LTS versions (i.e. into http://hg.openjdk.java.net/jdk-updates/jdk11u/). How that will play out in reality ... I don't know. Whether Oracle will contribute to that repo for 8 after it's EOL a

Re: Test and support only LTS JDK release?

2018-11-06 Thread DB Tsai
OpenJDK will follow Oracle's release cycle, https://openjdk.java.net/projects/jdk/ , a strict six months model. I'm not familiar with other non-Oracle VMs and Redhat support. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc

Re: Test and support only LTS JDK release?

2018-11-06 Thread Marcelo Vanzin
+1, that's always been my view. Although, to be fair, and as Sean mentioned, the jump from jdk8 is probably the harder part. After that it's less likely (hopefully?) that we'll run into issues in non-LTS releases. And even if we don't officially support them, trying to keep up with breaking change

Re: Test and support only LTS JDK release?

2018-11-06 Thread Reynold Xin
What does OpenJDK do and other non-Oracle VMs? I know there was a lot of discussions from Redhat etc to support. On Tue, Nov 6, 2018 at 11:24 AM DB Tsai wrote: > Given Oracle's new 6-month release model, I feel the only realistic option > is to only test and support JDK such as JDK 11 LTS and f

Re: Test fails when compiling spark with tests

2016-09-14 Thread Fred Reiss
Also try doing a fresh clone of the git repository. I've seen some of those rare failure modes corrupt parts of my local copy in the past. FWIW the main branch as of yesterday afternoon is building fine in my environment. Fred On Tue, Sep 13, 2016 at 6:29 PM, Jakob Odersky wrote: > There are s

Re: Test fails when compiling spark with tests

2016-09-13 Thread Jakob Odersky
There are some flaky tests that occasionally fail, my first recommendation would be to re-run the test suite. Another thing to check is if there are any applications listening to spark's default ports. Btw, what is your environment like? In case it is windows, I don't think tests are regularly run

Re: test failed due to OOME

2015-11-02 Thread Ted Yu
I have a PR which tries to address this issue: https://github.com/apache/spark/pull/9384 Comment is welcome. On Mon, Nov 2, 2015 at 9:53 AM, Patrick Wendell wrote: > I believe this is some bug in our tests. For some reason we are using way > more memory than necessary. We'll probably need to lo

Re: test failed due to OOME

2015-11-02 Thread Patrick Wendell
I believe this is some bug in our tests. For some reason we are using way more memory than necessary. We'll probably need to log into Jenkins and heap dump some running tests and figure out what is going on. On Mon, Nov 2, 2015 at 7:42 AM, Ted Yu wrote: > Looks like SparkListenerSuite doesn't OO

Re: test failed due to OOME

2015-11-02 Thread Ted Yu
Looks like SparkListenerSuite doesn't OOM on QA runs compared to Jenkins builds. I wonder if this is due to difference between machines running QA tests vs machines running Jenkins builds. On Fri, Oct 30, 2015 at 1:19 PM, Ted Yu wrote: > I noticed that the SparkContext created in each sub-test

Re: test failed due to OOME

2015-10-30 Thread Ted Yu
I noticed that the SparkContext created in each sub-test is not stopped upon finishing sub-test. Would stopping each SparkContext make a difference in terms of heap memory consumption ? Cheers On Fri, Oct 30, 2015 at 12:04 PM, Mridul Muralidharan wrote: > It is giving OOM at 32GB ? Something l

Re: test failed due to OOME

2015-10-30 Thread Mridul Muralidharan
It is giving OOM at 32GB ? Something looks wrong with that ... that is already on the higher side. Regards, Mridul On Fri, Oct 30, 2015 at 11:28 AM, shane knapp wrote: > here's the current heap settings on our workers: > InitialHeapSize == 2.1G > MaxHeapSize == 32G > > system ram: 128G > > we c

Re: test failed due to OOME

2015-10-30 Thread shane knapp
here's the current heap settings on our workers: InitialHeapSize == 2.1G MaxHeapSize == 32G system ram: 128G we can bump it pretty easily... it's just a matter of deciding if we want to do this globally (super easy, but will affect ALL maven builds on our system -- not just spark) or on a per-j

Re: test failed due to OOME

2015-10-30 Thread Ted Yu
This happened recently on Jenkins: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.3,label=spark-test/3964/console On Sun, Oct 18, 2015 at 7:54 AM, Ted Yu wrote: > From > https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADO

Re: Test workflow - blacklist entire suites and run any independently

2015-09-21 Thread Adam Roberts
with plenty of errors. Must be an easier way... From: Josh Rosen To: Adam Roberts/UK/IBM@IBMGB Cc: dev Date: 21/09/2015 19:19 Subject:Re: Test workflow - blacklist entire suites and run any independently For quickly running individual suites: https://cwiki.apache.or

Re: Test workflow - blacklist entire suites and run any independently

2015-09-21 Thread Josh Rosen
For quickly running individual suites: https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-RunningIndividualTests On Mon, Sep 21, 2015 at 8:21 AM, Adam Roberts wrote: > Hi, is there an existing way to blacklist any test suite? > > Ideally we'd have a tex

Re: Test all the things (Was: Unit test logs in Jenkins?)

2015-04-02 Thread shane knapp
cool. FYI, i'm at databricks today and talked w/patrick, josh and davies about this. we have some great ideas to actually make this happen and will be pushing over the next few weeks to get it done. :) On Thu, Apr 2, 2015 at 9:21 AM, Nicholas Chammas wrote: > (Renaming thread so as to un-hija

Re: Test suites in the python wrapper of kmeans failing

2015-01-21 Thread Meethu Mathew
Hi, Sorry it was my mistake. My code was not properly built. Regards, Meethu __ On Thursday 22 January 2015 10:39 AM, Meethu Mathew wrote: Hi, The test suites in the Kmeans class in clustering.py is not updated to take the seed value and hen

Re: Test failures after Jenkins upgrade

2014-12-15 Thread Josh Rosen
Opened one more PR for this: https://github.com/apache/spark/pull/3704 On December 15, 2014 at 10:59:00 AM, Patrick Wendell (pwend...@gmail.com) wrote: Ah cool Josh - I think for some reason we are hitting this every time now. Since this is holding up a bunch of other patches, I just pushed

Re: Test failures after Jenkins upgrade

2014-12-15 Thread Patrick Wendell
Ah cool Josh - I think for some reason we are hitting this every time now. Since this is holding up a bunch of other patches, I just pushed something ignoring the tests as a hotfix. Even waiting for a couple hours is really expensive productivity-wise given the frequency with which we run tests. We

Re: Test failures after Jenkins upgrade

2014-12-15 Thread Josh Rosen
There’s a JIRA for this: https://issues.apache.org/jira/browse/SPARK-4826 And two open PRs: https://github.com/apache/spark/pull/3695 https://github.com/apache/spark/pull/3701 We might be close to fixing this via one of those PRs, so maybe we should try using one of those instead? On December

Re: (test)

2014-05-16 Thread Aaron Davidson
No. Only 3 of the responses. On Fri, May 16, 2014 at 10:38 AM, Nishkam Ravi wrote: > Yes. > > > On Fri, May 16, 2014 at 8:40 AM, DB Tsai wrote: > > > Yes. > > On May 16, 2014 8:39 AM, "Andrew Or" wrote: > > > > > Apache has been having some problems lately. Do you guys see this > > message? >

Re: (test)

2014-05-16 Thread Reynold Xin
I didn't see the original message, but only a reply. On Fri, May 16, 2014 at 10:38 AM, Nishkam Ravi wrote: > Yes. > > > On Fri, May 16, 2014 at 8:40 AM, DB Tsai wrote: > > > Yes. > > On May 16, 2014 8:39 AM, "Andrew Or" wrote: > > > > > Apache has been having some problems lately. Do you guys

Re: (test)

2014-05-16 Thread Nishkam Ravi
Yes. On Fri, May 16, 2014 at 8:40 AM, DB Tsai wrote: > Yes. > On May 16, 2014 8:39 AM, "Andrew Or" wrote: > > > Apache has been having some problems lately. Do you guys see this > message? > > >

Re: (test)

2014-05-16 Thread DB Tsai
Yes. On May 16, 2014 8:39 AM, "Andrew Or" wrote: > Apache has been having some problems lately. Do you guys see this message? >

Re: (test)

2014-05-16 Thread Ted Yu
Yes. On Thu, May 15, 2014 at 10:34 AM, Andrew Or wrote: > Apache has been having some problems lately. Do you guys see this message? >

Re: test cases stuck on "local-cluster mode" of ReplSuite?

2014-03-14 Thread Nan Zhu
Yeah, I tested that, I had my SPARK_HOME point to a very old location, after I fixed that, everything goes well Thank you so much for pointing this out Best, -- Nan Zhu On Friday, March 14, 2014 at 6:41 PM, Michael Armbrust wrote: > Sorry to revive an old thread, but I just ran into thi

Re: test cases stuck on "local-cluster mode" of ReplSuite?

2014-03-14 Thread Michael Armbrust
Sorry to revive an old thread, but I just ran into this issue myself. It is likely that you do not have the assembly jar built, or that you have SPARK_HOME set incorrectly (it does not need to be set). Michael On Thu, Feb 27, 2014 at 8:13 AM, Nan Zhu wrote: > Hi, all > > Actually this problem