I wrote a very simple Spark 1.4.1 app that I can run through a local driver
program just fine using setMaster("local[*]"). The app is as follows:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
object Hello {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("Simple
Application").setMaster("local[*]")
val sc = new SparkContext(conf)
val data:RDD[Int] = sc.parallelize(Seq(1,2,12,34,2354,123,100), 2)
println("Max: " + data.max)
println("Min: " + data.min)
}
}
I compile this using the following build.sbt which will pull the needed Spark
libraries for me.
name := """SparkyJeff"""
version := "1.0"
scalaVersion := "2.11.6"
// Change this to another test framework if you prefer
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1",
"org.apache.spark" %% "spark-sql" % "1.4.1")
// Uncomment to use Akka
//libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.11"
fork := true
Now I'm trying to run this against a standalone cluster by changing the
setMaster("local[*]") to setMaster("spark://p3.ourdomain.com:7077"). I
downloaded Spark 1.4.1 for Hadoop 2.6 or greater. Set the
SPARK_MASTER_IP="p3.ourdomain.com",
SPARK_WORKER_CORES="1000",SPARK_WORKER_MEMORY="500g" and then started the
cluster using run-all.sh. The cluster appears to start fine. I can hit cluster
UI at p3.ourdomain.com:8080 and see the same master URL as mentioned above.
Now when I run my little app I get the following client error:
...
[error] 15/08/05 16:03:40 INFO AppClient$ClientActor: Connecting to master
akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:03:40 WARN ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:00 INFO AppClient$ClientActor: Connecting to master
akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:04:00 WARN ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:20 INFO AppClient$ClientActor: Connecting to master
akka.tcp://sparkmas...@p3.ourdomain.com:7077/user/Master...
[error] 15/08/05 16:04:20 WARN ReliableDeliverySupervisor: Association with
remote system [akka.tcp://sparkmas...@p3.ourdomain.com:7077] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
[error] 15/08/05 16:04:40 ERROR SparkDeploySchedulerBackend: Application has
been killed. Reason: All masters are unresponsive! Giving up.
...
Looking into the master logs I find:
15/08/06 22:52:28 INFO Master: akka.tcp://sparkDriver@192.168.137.41:48877 got
disassociated, removing it.
15/08/06 22:52:46 ERROR Remoting: org.apache.spark.deploy.Command; local class
incompatible: stream classdesc serialVersionUID = -7098307370860582211, local
class serialVersionUID = -3335312719467547622
java.io.InvalidClassException: org.apache.spark.deploy.Command; local class
incompatible: stream classdesc serialVersionUID = -7098307370860582211, local
class serialVersionUID = -3335312719467547622
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136)
at
akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at
akka.remote.serialization.MessageContainerSerializer.fromBinary(MessageContainerSerializer.scala:63)
at
akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104)
at scala.util.Try$.apply(Try.scala:161)
at akka.serialization.Serialization.deserialize(Serialization.scala:98)
at
akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23)
at
akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:58)
at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:76)
at
akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:937)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Any help would be greatly appreciated.
Thanks,
Jeff
This message (and any attachments) is intended only for the designated
recipient(s). It
may contain confidential or proprietary information, or have other limitations
on use as
indicated by the sender. If you are not a designated recipient, you may not
review, use,
copy or distribute this message. If you received this in error, please notify
the sender by
reply e-mail and delete this message.
