Hi everybody,
I am testing zeppelin with dynamic allocation but seem it’s not working.
Logs I received I saw that Spark Context was created successfully and task was
running but after that was terminated.
Any ideas on that?
Thanks.
INFO [2016-07-11 15:03:40,096] ({Thread-0}
RemoteInterpreterServer.java[run]:81) - Starting remote interpreter server on
port 24994
INFO [2016-07-11 15:03:40,471] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter
org.apache.zeppelin.spark.SparkInterpreter
INFO [2016-07-11 15:03:40,521] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter
org.apache.zeppelin.spark.PySparkInterpreter
INFO [2016-07-11 15:03:40,526] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter
org.apache.zeppelin.spark.SparkRInterpreter
INFO [2016-07-11 15:03:40,528] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter
org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2016-07-11 15:03:40,531] ({pool-1-thread-2}
RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter
org.apache.zeppelin.spark.DepInterpreter
INFO [2016-07-11 15:03:40,563] ({pool-2-thread-5}
SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1468224220562
started by scheduler org.apache.zeppelin.spark.SparkInterpreter998491254
WARN [2016-07-11 15:03:41,559] ({pool-2-thread-5}
NativeCodeLoader.java[<clinit>]:62) - Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
INFO [2016-07-11 15:03:41,703] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Changing view acls to: root
INFO [2016-07-11 15:03:41,704] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Changing modify acls to: root
INFO [2016-07-11 15:03:41,708] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
SecurityManager: authentication disabled; ui acls disabled; users with view
permissions: Set(root); users with modify permissions: Set(root)
INFO [2016-07-11 15:03:41,977] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Starting HTTP Server
INFO [2016-07-11 15:03:42,029] ({pool-2-thread-5} Server.java[doStart]:272) -
jetty-8.y.z-SNAPSHOT
INFO [2016-07-11 15:03:42,047] ({pool-2-thread-5}
AbstractConnector.java[doStart]:338) - Started [email protected]:53313
INFO [2016-07-11 15:03:42,048] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service 'HTTP class server' on port 53313.
INFO [2016-07-11 15:03:43,978] ({pool-2-thread-5}
SparkInterpreter.java[createSparkContext]:233) - ------ Create new SparkContext
mesos://zk://master1:2181,master2:2181,master3:2181/mesos -------
INFO [2016-07-11 15:03:44,003] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Running Spark version 1.6.1
INFO [2016-07-11 15:03:44,036] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Changing view acls to: root
INFO [2016-07-11 15:03:44,036] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Changing modify acls to: root
INFO [2016-07-11 15:03:44,037] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
SecurityManager: authentication disabled; ui acls disabled; users with view
permissions: Set(root); users with modify permissions: Set(root)
INFO [2016-07-11 15:03:44,231] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service 'sparkDriver' on port 33913.
INFO [2016-07-11 15:03:44,552]
({sparkDriverActorSystem-akka.actor.default-dispatcher-4}
Slf4jLogger.scala[applyOrElse]:80) - Slf4jLogger started
INFO [2016-07-11 15:03:44,597]
({sparkDriverActorSystem-akka.actor.default-dispatcher-4}
Slf4jLogger.scala[apply$mcV$sp]:74) - Starting remoting
INFO [2016-07-11 15:03:44,754]
({sparkDriverActorSystem-akka.actor.default-dispatcher-4}
Slf4jLogger.scala[apply$mcV$sp]:74) - Remoting started; listening on addresses
:[akka.tcp://[email protected]:55213]
INFO [2016-07-11 15:03:44,760] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service 'sparkDriverActorSystem' on port 55213.
INFO [2016-07-11 15:03:44,771] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Registering MapOutputTracker
INFO [2016-07-11 15:03:44,789] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Registering BlockManagerMaster
INFO [2016-07-11 15:03:44,802] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Created local directory at
/data/tmp/blockmgr-14f6a013-abb7-4b46-adab-07282b03c7e4
INFO [2016-07-11 15:03:44,808] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
MemoryStore started with capacity 511.1 MB
INFO [2016-07-11 15:03:44,871] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Registering OutputCommitCoordinator
INFO [2016-07-11 15:03:45,018] ({pool-2-thread-5} Server.java[doStart]:272) -
jetty-8.y.z-SNAPSHOT
INFO [2016-07-11 15:03:45,031] ({pool-2-thread-5}
AbstractConnector.java[doStart]:338) - Started
[email protected]:4040
INFO [2016-07-11 15:03:45,033] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service 'SparkUI' on port 4040.
INFO [2016-07-11 15:03:45,035] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Started SparkUI at http://10.197.0.3:4040
INFO [2016-07-11 15:03:45,059] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
HTTP File server directory is
/data/tmp/spark-4467170d-a139-4fd4-8628-90820e349760/httpd-b7104d70-f7fb-400c-b1e6-49ec36b7c99e
INFO [2016-07-11 15:03:45,059] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Starting HTTP Server
INFO [2016-07-11 15:03:45,060] ({pool-2-thread-5} Server.java[doStart]:272) -
jetty-8.y.z-SNAPSHOT
INFO [2016-07-11 15:03:45,063] ({pool-2-thread-5}
AbstractConnector.java[doStart]:338) - Started [email protected]:21210
INFO [2016-07-11 15:03:45,064] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service 'HTTP file server' on port 21210.
INFO [2016-07-11 15:03:45,099] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Added JAR
file:/home/spark/log_analyzer/alluxio-core-client-spark-1.1.0-jar-with-dependencies.jar
at
http://10.197.0.3:21210/jars/alluxio-core-client-spark-1.1.0-jar-with-dependencies.jar
with timestamp 1468224225098
INFO [2016-07-11 15:03:45,116] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Added JAR
file:/home/spark/zeppelin-current/interpreter/spark/zeppelin-spark-0.6.0.jar at
http://10.197.0.3:21210/jars/zeppelin-spark-0.6.0.jar with timestamp
1468224225116
INFO [2016-07-11 15:03:45,206] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Created default pool default, schedulingMode: FIFO, minShare: 0, weight: 1
INFO [2016-07-11 15:03:45,286] ({Thread-37} Logging.scala[logInfo]:58) -
Registered as framework ID 90694c50-1759-455b-9034-77a85e3bcab7-0048
INFO [2016-07-11 15:03:45,294] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 37086.
INFO [2016-07-11 15:03:45,294] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Server created on 37086
INFO [2016-07-11 15:03:45,297] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
external shuffle service port = 7337
INFO [2016-07-11 15:03:45,298] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Trying to register BlockManager
INFO [2016-07-11 15:03:45,301] ({dispatcher-event-loop-2}
Logging.scala[logInfo]:58) - Registering block manager 10.197.0.3:37086 with
511.1 MB RAM, BlockManagerId(driver, 10.197.0.3, 37086)
INFO [2016-07-11 15:03:45,303] ({pool-2-thread-5} Logging.scala[logInfo]:58) -
Registered BlockManager
INFO [2016-07-11 15:03:45,414] ({Thread-39} Logging.scala[logInfo]:58) - Mesos
task 3 is now TASK_RUNNING
INFO [2016-07-11 15:03:45,465] ({CoarseMesosSchedulerBackend-mesos-driver}
Logging.scala[logInfo]:58) - driver.run() returned with code DRIVER_ABORTED
INFO [2016-07-11 15:03:45,469] ({Thread-2} Logging.scala[logInfo]:58) -
Shutdown hook called
INFO [2016-07-11 15:03:45,477] ({Thread-2} Logging.scala[logInfo]:58) -
Shutdown hook called
INFO [2016-07-11 15:03:45,478] ({Thread-2} Logging.scala[logInfo]:58) -
Deleting directory /tmp/spark-e23ac5cc-25b0-4a98-ab37-54073ae58a7b
INFO [2016-07-11 15:03:45,479] ({Thread-2} Logging.scala[logInfo]:58) -
Deleting directory /data/tmp/spark-4467170d-a139-4fd4-8628-90820e349760
INFO [2016-07-11 15:03:45,486] ({Thread-2} Logging.scala[logInfo]:58) -
Deleting directory
/data/tmp/spark-4467170d-a139-4fd4-8628-90820e349760/httpd-b7104d70-f7fb-400c-b1e6-49ec36b7c99e
INFO [2016-07-11 15:03:45,486] ({Thread-2} Logging.scala[logInfo]:58) -
Deleting directory
/data/tmp/spark-4467170d-a139-4fd4-8628-90820e349760/userFiles-88d77298-a5c8-4d7f-953c-bfd5b973fb9f
