Hi Spark users/experts,
In Spark source code (Master.scala & Worker.scala), when registering the
worker with master, I see the usage of *persistenceEngine*. When we don't
specify spark.deploy.recovery mode explicitly, what is the default value
used ? This recovery mode is used to persists and re
When building off of master it looks like the wrong version of the jar is
added to the assembly jar. The fix for SPARK-3039 added the "hadoop2"
classifier at compile time when using the hadoop2 profile, but in the
assembly stage I get:
"[INFO] Including org.apache.avro:avro-mapred:jar:1.7.6 in
Hi everyone,
I’m adding some new message passing between the Master and Worker actors in
order to address https://issues.apache.org/jira/browse/SPARK-3736 .
I was wondering if these kinds of interactions are tested in the automated
Jenkins test suite, and if so, where I could find some examples t
You can use akka testkit
Example:
https://github.com/apache/spark/blob/ef4ff00f87a4e8d38866f163f01741c2673e41da/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala
--
Nan Zhu
On Tuesday, October 14, 2014 at 9:17 PM, Matthew Cheah wrote:
> Hi everyone,
>
> I’m ad
