Joe Olson created SPARK-19770:
---------------------------------
Summary: Running Example SparkPi Job on Mesos
Key: SPARK-19770
URL: https://issues.apache.org/jira/browse/SPARK-19770
Project: Spark
Issue Type: Question
Components: Mesos, Spark Core, Spark Submit
Affects Versions: 2.1.0
Environment: spark-2.1.0-bin-hadoop2.3, mesos-1-1
Reporter: Joe Olson
I am trying to submit the example SparkPi job to a 9 node Spark cluster running
on Mesos. My spark-submit statement:
./bin/spark-submit \
--name "Test01:" \
--class org.apache.spark.examples.SparkPi \
--master mesos://<IP Address>:7078 \
--deploy-mode cluster \
--executor-memory 16G \
--executor-cores 1 \
--driver-cores 10 \
--driver-memory 48G \
--num-executors 1 \
file://mnt/ocarchive1/sstables/jars/spark-examples_2.11-2.1.0.jar \
1000
When I do this, the job completes successfully. I can go into the stdout file
on the driver machine, and see the "Pi is roughly...." output.
However, on most of the slave machines, If I go into the stderr file for that
same job, I see the following exceptions:
17/02/28 10:20:59 ERROR RetryingBlockFetcher: Exception while beginning fetch
of 1 outstanding blocks
java.io.IOException: Failed to connect to <machine
name>/fe80:0:0:0:ec4:7aff:fea4:82e1%5:44791
There appears to be a connectivity issue between some of the 9 nodes, some of
the time. It does not seem to be consistent on the routes between machines.
(Sometimes #2 and #7 talk, sometimes they do not).
How can I troubleshoot this? Network behavior seems normal otherwise.
ALSO - sometimes in the logs I'll see
17/02/28 10:20:54 WARN TaskSchedulerImpl: Initial job has not accepted any
resources; check your cluster UI to ensure that workers are registered and have
sufficient resources
However, my resource count in Mesos (via the UI) is accurate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
