Hello,
First of all I m a newbie in Spark ,
I m trying to start the spark-shell with yarn cluster by running:
$ spark-shell --master yarn-client
Sometimes it goes well, but most of the time I got an error:
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1437145851944
final status: FAILED
tracking URL:
http://My-HadoopServer:50080/cluster/app/application_1437033338028_0030
user: hadoop
org.apache.spark.SparkException: Yarn application has already ended! It
might have been killed or unable to launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:115)
....
....
....
searching in the yarn logs I got this log
$ yarn logs -applicationId application_1437033338028_0030
2015-07-17 17:11:03,961 - INFO
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3@74]
- Starting remoting
2015-07-17 17:11:04,200 - ERROR
[sparkYarnAM-akka.actor.default-dispatcher-4:Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1@65]
- failed to bind to My-HadoopServer/10.98.105.11:0, shutting down Netty
transport
2015-07-17 17:11:04,210 - WARN [main:Logging$class@71] - Service
'sparkYarnAM' could not bind on port 0. Attempting port 1.
...
...
...
2015-07-17 17:11:05,123 - ERROR [main:Logging$class@96] - Uncaught
exception:
java.net.BindException: Failed to bind to:
My-HadoopServer/HadoopServerIP:0: Service 'sparkYarnAM' failed after 16
retries!
at
org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393)
at
akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
...
....
....
I m using Spark 1.3, Hadoop 2.6 ,
and in spark-env.sh it points to my hadoop configuration:
export HADOOP_CONF_DIR=/usr/hdp/2.2.4.4-16/hadoop/conf
Is this probleme coming from spark configuration or yarn configuration (or
spark with yarn confs)????
Any Ideas??
Amjad
