Hi, Not that I might help much with deployment to Mesos, but can you describe your Mesos/Marathon setup? What's Mesos cluster dispatcher?
Jacek 18.04.2016 12:54 PM "Joao Azevedo" <joao.c.azev...@gmail.com> napisał(a): > Hi! > > I'm trying to submit Spark applications to Mesos using the 'cluster' > deploy mode. I'm using Marathon as the container orchestration platform and > launching the Mesos Cluster Dispatcher through it. I'm using Spark 1.6.1 > with Scala 2.11. > > I'm able to successfully communicate with the cluster dispatcher, but > executors that run on a node different from the one where the driver is > started fail to run. I believe that they're assuming the same (local) > classpath as the driver, and thus failing to run. Here's what I see from > the failed tasks on the executor nodes: > > sh -c ' > "/tmp/mesos/slaves/3b945207-1aac-497d-b2fb-9671d3d0646b-S1/frameworks/b925b271-ce42-48c9-8474-9e19304c2d20-0002/executors/driver-20160418101719-0001/runs/39272646-d305-40a6-9431-5efcacc67b80/spark-1.6.1-bin-2.4.0/bin/spark-class" > org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url > spark://CoarseGrainedScheduler@<driver_ip>:35600 --executor-id > 00706e76-f790-465d-b4cd-edd3934ddb6a-S1 --hostname <executor_hostname> > --cores 5 --app-id 00706e76-f790-465d-b4cd-edd3934ddb6a-0048' > Command exited with status 127 (pid: 4118) > > When the executors are started in the same node where the driver is, the > application runs ok. > > Here's how I'm submitting the applications: > > bin/spark-submit --master mesos://<mesos_cluster_dispatcher_hostname>:7077 > --deploy-mode cluster --conf "spark.executor.uri=<uri_to_spark>" > --executor-memory 5G --total-executor-cores 5 --supervise --class > org.apache.spark.examples.SparkPi <uri_to_spark_example> 10 > > Does anyone have any ideas on what I might be doing wrong? > > > Joao > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
