Apologies; can you advise as to how I would check that? I can certainly
SSH from master to machine2.
On 6/27/14, 3:22 PM, Sujeet Varakhedi wrote:
Looks like your driver is not able to connect to the remote executor
on machine2/130.49.226.148:60949 <http://130.49.226.148:60949/>. Cn
you check if the master machine can route to 130.49.226.148
Sujeet
On Fri, Jun 27, 2014 at 12:04 PM, Shannon Quinn <squ...@gatech.edu
<mailto:squ...@gatech.edu>> wrote:
For some reason, commenting out spark.driver.host and
spark.driver.port fixed something...and broke something else (or
at least revealed another problem). For reference, the only lines
I have in my spark-defaults.conf now:
spark.app.name <http://spark.app.name> myProg
spark.master spark://192.168.1.101:5060
<http://192.168.1.101:5060>
spark.executor.memory 8g
spark.files.overwrite true
It starts up, but has problems with machine2. For some reason,
machine2 is having trouble communicating with *itself*. Here are
the worker logs of one of the failures (there are 10 before it
quits):
Spark assembly has been built with Hive, including Datanucleus
jars on classpath
14/06/27 14:55:13 INFO ExecutorRunner: Launch command: "java"
"-cp"
"::/home/spark/spark-1.0.0-bin-hadoop2/conf:/home/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/home/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/spark/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar"
"-XX:MaxPermSize=128m" "-Xms8192M" "-Xmx8192M"
"org.apache.spark.executor.CoarseGrainedExecutorBackend"
"akka.tcp://spark@machine1:46378/user/CoarseGrainedScheduler" "7"
"machine2" "8" "akka.tcp://sparkWorker@machine2:48019/user/Worker"
"app-20140627144512-0001"
14/06/27 14:56:54 INFO Worker: Executor app-20140627144512-0001/7
finished with state FAILED message Command exited with code 1
exitStatus 1
14/06/27 14:56:54 INFO LocalActorRef: Message
[akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying]
from Actor[akka://sparkWorker/deadLetters] to
Actor[akka://sparkWorker/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkWorker%40130.49.226.148%3A53561-38#-1924573003]
was not delivered. [10] dead letters encountered. This logging can
be turned off or adjusted with configuration settings
'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/06/27 14:56:54 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkWorker@machine2:48019] ->
[akka.tcp://sparkExecutor@machine2:60949]: Error [Association
failed with [akka.tcp://sparkExecutor@machine2:60949]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkExecutor@machine2:60949]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: machine2/130.49.226.148:60949
<http://130.49.226.148:60949>
]
14/06/27 14:56:54 INFO Worker: Asked to launch executor
app-20140627144512-0001/8 for Funtown, USA
14/06/27 14:56:54 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkWorker@machine2:48019] ->
[akka.tcp://sparkExecutor@machine2:60949]: Error [Association
failed with [akka.tcp://sparkExecutor@machine2:60949]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkExecutor@machine2:60949]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: machine2/130.49.226.148:60949
<http://130.49.226.148:60949>
]
14/06/27 14:56:54 ERROR EndpointWriter: AssociationError
[akka.tcp://sparkWorker@machine2:48019] ->
[akka.tcp://sparkExecutor@machine2:60949]: Error [Association
failed with [akka.tcp://sparkExecutor@machine2:60949]] [
akka.remote.EndpointAssociationException: Association failed with
[akka.tcp://sparkExecutor@machine2:60949]
Caused by:
akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
Connection refused: machine2/130.49.226.148:60949
<http://130.49.226.148:60949>
]
Port 48019 on machine2 is indeed open, connected, and listening.
Any ideas?
Thanks!
Shannon
On 6/27/14, 1:54 AM, sujeetv wrote:
Try to explicitly set set the "spark.driver.host" property to
the master's
IP.
Sujeet
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-standalone-network-configuration-problems-tp8304p8396.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
