Re: Using local-cluster mode for testing Spark-related projects

2016-04-17 Thread Evan Chan
Jon, Thanks. I think I've figured it out, actually. It's really simple, one needs to simply set spark.executor.extraClassPath to the current value of the java class path (java.class.path system property). Also, to not use HiveContext, which gives errors about initializing a Derby database mu

Re: Possible deadlock in registering applications in the recovery mode

2016-04-17 Thread Reynold Xin
I haven't looked closely at this, but I think your proposal makes sense. On Sun, Apr 17, 2016 at 6:40 PM, Niranda Perera wrote: > Hi guys, > > Any update on this? > > Best > > On Tue, Apr 12, 2016 at 12:46 PM, Niranda Perera > wrote: > >> Hi all, >> >> I have encountered a small issue in the s

Re: Recent Jenkins always fails in specific two tests

2016-04-17 Thread Marcin Tustin
Also hitting this: https://github.com/apache/spark/pull/12455. On Sun, Apr 17, 2016 at 9:22 PM, Hyukjin Kwon wrote: > +1 > > Yea, I am facing this problem as well, > https://github.com/apache/spark/pull/12452 > > I thought they are spurious because the tests are passed in my local. > > > > 201

Re: Possible deadlock in registering applications in the recovery mode

2016-04-17 Thread Niranda Perera
Hi guys, Any update on this? Best On Tue, Apr 12, 2016 at 12:46 PM, Niranda Perera wrote: > Hi all, > > I have encountered a small issue in the standalone recovery mode. > > Let's say there was an application A running in the cluster. Due to some > issue, the entire cluster, together with the

Re: Recent Jenkins always fails in specific two tests

2016-04-17 Thread Hyukjin Kwon
+1 Yea, I am facing this problem as well, https://github.com/apache/spark/pull/12452 I thought they are spurious because the tests are passed in my local. 2016-04-18 3:26 GMT+09:00 Kazuaki Ishizaki : > I realized that recent Jenkins among different pull requests always fails > in the followin

Re: Question about Scala style, explicit typing within transformation functions and anonymous val.

2016-04-17 Thread Koert Kuipers
i find version 3 without the _ also more readable On Sun, Apr 17, 2016 at 3:02 AM, Mark Hamstra wrote: > I actually find my version of 3 more readable than the one with the `_`, > which looks too much like a partially applied function. It's a minor > issue, though. > > On Sat, Apr 16, 2016 at 1

Re: Impact of STW GC events for the driver JVM on overall cluster

2016-04-17 Thread Rahul Tanwani
Does that not mean, GC settings with concurrent collectors should be preferred over parallel collectors atleast on the driver side? If so, why not have concurrent collectors specified by default when the driver JVM is launched without any overriding on this part? -- View this message in context:

Re: Impact of STW GC events for the driver JVM on overall cluster

2016-04-17 Thread Reynold Xin
Your understanding is correct. If the driver is stuck in GC, then during that period it cannot schedule any tasks. On Sun, Apr 17, 2016 at 10:27 AM, Rahul Tanwani wrote: > Hi Devs, > > In case of stop the world GC events on the driver JVM, since all the > application threads will be stopped, th

Recent Jenkins always fails in specific two tests

2016-04-17 Thread Kazuaki Ishizaki
I realized that recent Jenkins among different pull requests always fails in the following two tests "SPARK-8020: set sql conf in spark conf" "SPARK-9757 Persist Parquet relation with decimal column" Here are examples. https://github.com/apache/spark/pull/11956 (consoleFull: https://amplab.cs.be

Impact of STW GC events for the driver JVM on overall cluster

2016-04-17 Thread Rahul Tanwani
Hi Devs, In case of stop the world GC events on the driver JVM, since all the application threads will be stopped, there won't be any new task scheduled / launched on the executors. In cases where the full collection is happening, the applications threads may be stopped for a long time, and if the

Re: Using local-cluster mode for testing Spark-related projects

2016-04-17 Thread Jon Maurer
Take a look at spark testing base. https://github.com/holdenk/spark-testing-base/blob/master/README.md On Apr 17, 2016 10:28 AM, "Evan Chan" wrote: > What I want to find out is how to run tests like Spark's with > local-cluster, just like that suite, but in your own projects. Has > anyone done

Re: Creating Spark Extras project, was Re: SPARK-13843 and future of streaming backends

2016-04-17 Thread Luciano Resende
On Sat, Apr 16, 2016 at 11:12 PM, Reynold Xin wrote: > First, really thank you for leading the discussion. > > I am concerned that it'd hurt Spark more than it helps. As many others > have pointed out, this unnecessarily creates a new tier of connectors or > 3rd party libraries appearing to be en

Re: Using local-cluster mode for testing Spark-related projects

2016-04-17 Thread Evan Chan
What I want to find out is how to run tests like Spark's with local-cluster, just like that suite, but in your own projects. Has anyone done this? On Sun, Apr 17, 2016 at 5:37 AM, Takeshi Yamamuro wrote: > Hi, > Is this a bad idea to create `SparkContext` with a `local-cluster` mode by > yourse

Re: Using local-cluster mode for testing Spark-related projects

2016-04-17 Thread Takeshi Yamamuro
Hi, Is this a bad idea to create `SparkContext` with a `local-cluster` mode by yourself like ' https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/ShuffleSuite.scala#L55 '? // maropu On Sun, Apr 17, 2016 at 9:47 AM, Evan Chan wrote: > Hey folks, > > I'd like to use

Re: Question about Scala style, explicit typing within transformation functions and anonymous val.

2016-04-17 Thread Mark Hamstra
I actually find my version of 3 more readable than the one with the `_`, which looks too much like a partially applied function. It's a minor issue, though. On Sat, Apr 16, 2016 at 11:56 PM, Hyukjin Kwon wrote: > Hi Mark, > > I know but that could harm readability. AFAIK, for this reason, that