Hi!
I'm trying to submit Spark applications to Mesos using the 'cluster'
deploy mode. I'm using Marathon as the container orchestration platform
and launching the Mesos Cluster Dispatcher through it. I'm using Spark
1.6.1 with Scala 2.11.
I'm able to successfully communicate with the cluster dispatcher, but
executors that run on a node different from the one where the driver is
started fail to run. I believe that they're assuming the same (local)
classpath as the driver, and thus failing to run. Here's what I see from
the failed tasks on the executor nodes:
sh -c '
"/tmp/mesos/slaves/3b945207-1aac-497d-b2fb-9671d3d0646b-S1/frameworks/b925b271-ce42-48c9-8474-9e19304c2d20-0002/executors/driver-20160418101719-0001/runs/39272646-d305-40a6-9431-5efcacc67b80/spark-1.6.1-bin-2.4.0/bin/spark-class"
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@<driver_ip>:35600 --executor-id
00706e76-f790-465d-b4cd-edd3934ddb6a-S1 --hostname <executor_hostname>
--cores 5 --app-id 00706e76-f790-465d-b4cd-edd3934ddb6a-0048'
Command exited with status 127 (pid: 4118)
When the executors are started in the same node where the driver is, the
application runs ok.
Here's how I'm submitting the applications:
bin/spark-submit --master
mesos://<mesos_cluster_dispatcher_hostname>:7077 --deploy-mode cluster
--conf "spark.executor.uri=<uri_to_spark>" --executor-memory 5G
--total-executor-cores 5 --supervise --class
org.apache.spark.examples.SparkPi <uri_to_spark_example> 10
Does anyone have any ideas on what I might be doing wrong?
Joao
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
