Hi Amos,
I am using ubuntu docker in a ubuntu system.
Yes, it seems spark is not starting. I don't see the statement "------
Create new SparkContext" in my console as well.
I see a single file in zeppelin\logs zeppelin--7ed5948b0d52.log and
attached the same. It doesn't contain any errors. Do you want me to enable
some logs and rerun the paragraph?
Thanks,
Sam.
On Mon, May 9, 2016 at 8:42 PM, Amos Elberg <amos.elb...@me.com> wrote:
> Sam - it should run fine in docker, but if you are running a Linux docker
> on Windows, I have heard that sometimes the environment can become flaky.
>
> What you're seeing sounds like spark is failing to start.
>
> You should be able to get Zeppelin to output a log for the entire spark
> interpreter group. (What you pasted from is the Zeppelin log.) Can you tell
> us what's in there?
>
> > On May 9, 2016, at 10:22 AM, Samuel Alexander <
> sam...@palmtreeinfotech.com> wrote:
> >
> > Hi,
> >
> > I am running zeppelin in a docker container.
> >
> > I built the zeppelin using `mvn clean package -Pspark-1.6 -Ppyspark
> > -Phadoop-2.4 -Pr -DskipTests`. I am using RRepl Interpreter.
> >
> > I just created a very simple paragraph with the below content.
> >
> > <<<
> > %r
> >
> > print(1+1)
> >
> >
> > It goes to pending state and it is in pending state until I restart
> > zeppelin-server.
> >
> > I am not able to execute any R paragraph. I don't see log statements as
> > well.
> >
> > Here is the log snippet
> >
> > <<<
> >
> > INFO [2016-05-09 14:06:34,311] ({pool-1-thread-2}
> > SchedulerFactory.java[jobStarted]:131) - Job
> > paragraph_1461917041860_145219673 started by scheduler
> >
> org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session499376351
> > INFO [2016-05-09 14:06:34,312] ({pool-1-thread-2}
> > Paragraph.java[jobRun]:226) - run paragraph 20160429-080401_1472155327
> > using r org.apache.zeppelin.interpreter.LazyOpenInterpreter@47debb2d
> > INFO [2016-05-09 14:06:34,321] ({pool-1-thread-2}
> > RemoteInterpreterProcess.java[reference]:119) - Run interpreter process
> > [/work/incubator-zeppelin/bin/interpreter.sh, -d,
> > /work/incubator-zeppelin/interpreter/spark, -p, 48832, -l,
> > /work/incubator-zeppelin/local-repo/2B894THHQ]
> > INFO [2016-05-09 14:06:35,516] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.rinterpreter.RRepl
> > INFO [2016-05-09 14:06:35,547] ({pool-1-thread-2}
> > RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:416) - Push
> local
> > angular object registry from ZeppelinServer to remote interpreter group
> > 2B894THHQ
> > INFO [2016-05-09 14:06:35,578] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.spark.SparkInterpreter
> > INFO [2016-05-09 14:06:35,581] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.spark.PySparkInterpreter
> > INFO [2016-05-09 14:06:35,584] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.rinterpreter.KnitR
> > INFO [2016-05-09 14:06:35,585] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.spark.SparkSqlInterpreter
> > INFO [2016-05-09 14:06:35,586] ({pool-1-thread-2}
> > RemoteInterpreter.java[init]:149) - Create remote interpreter
> > org.apache.zeppelin.spark.DepInterpreter
> >
> >
> > Please note that I've set SPARK_HOME in environment.
> > Also when I did the same in my ubuntu instance, it is working fine.
> >
> > Is there any issue when RInterpreter(RRepl) running in docker container?
> > Or did I miss something?
> >
> > Thanks,
> > Sam.
>
INFO [2016-05-09 14:05:58,831] ({main} ZeppelinConfiguration.java[create]:98) - Load configuration from file:/work/incubator-zeppelin/conf/zeppelin-site.xml
INFO [2016-05-09 14:05:59,210] ({main} ZeppelinServer.java[main]:115) - Starting zeppelin server
INFO [2016-05-09 14:05:59,213] ({main} Server.java[doStart]:272) - jetty-8.1.14.v20131031
INFO [2016-05-09 14:05:59,248] ({main} ContextHandler.java[log]:2040) - Initializing Shiro environment
INFO [2016-05-09 14:05:59,251] ({main} EnvironmentLoader.java[initEnvironment]:128) - Starting Shiro environment initialization.
INFO [2016-05-09 14:05:59,479] ({main} EnvironmentLoader.java[initEnvironment]:141) - Shiro environment initialized in 227 ms.
INFO [2016-05-09 14:05:59,687] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/phoenix
INFO [2016-05-09 14:05:59,778] ({main} InterpreterFactory.java[init]:136) - Interpreter phoenix.sql found. class=org.apache.zeppelin.phoenix.PhoenixInterpreter
INFO [2016-05-09 14:05:59,783] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/alluxio
INFO [2016-05-09 14:05:59,852] ({main} InterpreterFactory.java[init]:136) - Interpreter alluxio.alluxio found. class=org.apache.zeppelin.alluxio.AlluxioInterpreter
INFO [2016-05-09 14:05:59,853] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/angular
INFO [2016-05-09 14:05:59,858] ({main} InterpreterFactory.java[init]:136) - Interpreter angular.angular found. class=org.apache.zeppelin.angular.AngularInterpreter
INFO [2016-05-09 14:05:59,866] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/elasticsearch
INFO [2016-05-09 14:05:59,899] ({main} InterpreterFactory.java[init]:136) - Interpreter elasticsearch.elasticsearch found. class=org.apache.zeppelin.elasticsearch.ElasticsearchInterpreter
INFO [2016-05-09 14:05:59,900] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/md
INFO [2016-05-09 14:05:59,907] ({main} InterpreterFactory.java[init]:136) - Interpreter md.md found. class=org.apache.zeppelin.markdown.Markdown
INFO [2016-05-09 14:05:59,920] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/psql
INFO [2016-05-09 14:05:59,933] ({main} InterpreterFactory.java[init]:136) - Interpreter psql.sql found. class=org.apache.zeppelin.postgresql.PostgreSqlInterpreter
INFO [2016-05-09 14:05:59,937] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/kylin
INFO [2016-05-09 14:05:59,955] ({main} InterpreterFactory.java[init]:136) - Interpreter kylin.kylin found. class=org.apache.zeppelin.kylin.KylinInterpreter
INFO [2016-05-09 14:05:59,962] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/flink
INFO [2016-05-09 14:06:00,018] ({main} InterpreterFactory.java[init]:136) - Interpreter flink.flink found. class=org.apache.zeppelin.flink.FlinkInterpreter
INFO [2016-05-09 14:06:00,024] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/spark
INFO [2016-05-09 14:06:00,092] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.spark found. class=org.apache.zeppelin.spark.SparkInterpreter
INFO [2016-05-09 14:06:00,095] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.pyspark found. class=org.apache.zeppelin.spark.PySparkInterpreter
INFO [2016-05-09 14:06:00,099] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.r found. class=org.apache.zeppelin.rinterpreter.RRepl
INFO [2016-05-09 14:06:00,101] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.knitr found. class=org.apache.zeppelin.rinterpreter.KnitR
INFO [2016-05-09 14:06:00,109] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.sql found. class=org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2016-05-09 14:06:00,111] ({main} InterpreterFactory.java[init]:136) - Interpreter spark.dep found. class=org.apache.zeppelin.spark.DepInterpreter
INFO [2016-05-09 14:06:00,118] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/tajo
INFO [2016-05-09 14:06:00,133] ({main} InterpreterFactory.java[init]:136) - Interpreter tajo.tql found. class=org.apache.zeppelin.tajo.TajoInterpreter
INFO [2016-05-09 14:06:00,140] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/cassandra
INFO [2016-05-09 14:06:00,160] ({main} CassandraInterpreter.java[<clinit>]:154) - Bootstrapping Cassandra Interpreter
INFO [2016-05-09 14:06:00,160] ({main} InterpreterFactory.java[init]:136) - Interpreter cassandra.cassandra found. class=org.apache.zeppelin.cassandra.CassandraInterpreter
INFO [2016-05-09 14:06:00,164] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/jdbc
INFO [2016-05-09 14:06:00,177] ({main} InterpreterFactory.java[init]:136) - Interpreter jdbc.sql found. class=org.apache.zeppelin.jdbc.JDBCInterpreter
INFO [2016-05-09 14:06:00,179] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/sh
INFO [2016-05-09 14:06:00,186] ({main} InterpreterFactory.java[init]:136) - Interpreter sh.sh found. class=org.apache.zeppelin.shell.ShellInterpreter
INFO [2016-05-09 14:06:00,192] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/hive
INFO [2016-05-09 14:06:00,246] ({main} InterpreterFactory.java[init]:136) - Interpreter hive.hql found. class=org.apache.zeppelin.hive.HiveInterpreter
INFO [2016-05-09 14:06:00,254] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/file
INFO [2016-05-09 14:06:00,270] ({main} InterpreterFactory.java[init]:136) - Interpreter file.hdfs found. class=org.apache.zeppelin.file.HDFSFileInterpreter
INFO [2016-05-09 14:06:00,275] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/lens
INFO [2016-05-09 14:06:00,338] ({main} InterpreterFactory.java[init]:136) - Interpreter lens.lens found. class=org.apache.zeppelin.lens.LensInterpreter
INFO [2016-05-09 14:06:00,344] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/ignite
INFO [2016-05-09 14:06:00,381] ({main} InterpreterFactory.java[init]:136) - Interpreter ignite.ignite found. class=org.apache.zeppelin.ignite.IgniteInterpreter
INFO [2016-05-09 14:06:00,382] ({main} InterpreterFactory.java[init]:136) - Interpreter ignite.ignitesql found. class=org.apache.zeppelin.ignite.IgniteSqlInterpreter
INFO [2016-05-09 14:06:00,387] ({main} InterpreterFactory.java[init]:119) - Reading /work/incubator-zeppelin/interpreter/hbase
INFO [2016-05-09 14:06:00,451] ({main} InterpreterFactory.java[init]:136) - Interpreter hbase.hbase found. class=org.apache.zeppelin.hbase.HbaseInterpreter
INFO [2016-05-09 14:06:00,503] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group elasticsearch : id=2BAYG9Y6Z, name=elasticsearch
INFO [2016-05-09 14:06:00,503] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group md : id=2BA76KFDM, name=md
INFO [2016-05-09 14:06:00,503] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group tajo : id=2BBU6YFUE, name=tajo
INFO [2016-05-09 14:06:00,503] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group lens : id=2BB7HZ3UB, name=lens
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group hive : id=2BBK6SB5Q, name=hive
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group r : id=2BFKETRWM, name=r1
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group kylin : id=2B87VYB3F, name=kylin
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group cassandra : id=2B8KU68E5, name=cassandra
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group psql : id=2B9RHRG68, name=psql
INFO [2016-05-09 14:06:00,504] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group spark : id=2B894THHQ, name=spark
INFO [2016-05-09 14:06:00,505] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group phoenix : id=2B94XE4PD, name=phoenix
INFO [2016-05-09 14:06:00,505] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group r : id=2B8282KNZ, name=r
INFO [2016-05-09 14:06:00,505] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group flink : id=2B98DS58V, name=flink
INFO [2016-05-09 14:06:00,505] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group angular : id=2B9GUBHMH, name=angular
INFO [2016-05-09 14:06:00,505] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group sh : id=2BAR178ZJ, name=sh
INFO [2016-05-09 14:06:00,506] ({main} InterpreterFactory.java[init]:198) - Interpreter setting group ignite : id=2B9N9F4PA, name=ignite
INFO [2016-05-09 14:06:00,581] ({main} VfsLog.java[info]:138) - Using "/tmp/vfs_cache" as temporary files store.
INFO [2016-05-09 14:06:00,647] ({main} GitNotebookRepo.java[<init>]:58) - Opening a git repo at '/work/incubator-zeppelin/notebook'
INFO [2016-05-09 14:06:00,945] ({main} NotebookAuthorization.java[loadFromFile]:59) - /work/incubator-zeppelin/conf/notebook-authorization.json
INFO [2016-05-09 14:06:00,986] ({main} StdSchedulerFactory.java[instantiate]:1184) - Using default implementation for ThreadExecutor
INFO [2016-05-09 14:06:00,990] ({main} SimpleThreadPool.java[initialize]:268) - Job execution threads will use class loader of thread: main
INFO [2016-05-09 14:06:01,010] ({main} SchedulerSignalerImpl.java[<init>]:61) - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
INFO [2016-05-09 14:06:01,010] ({main} QuartzScheduler.java[<init>]:240) - Quartz Scheduler v.2.2.1 created.
INFO [2016-05-09 14:06:01,012] ({main} RAMJobStore.java[initialize]:155) - RAMJobStore initialized.
INFO [2016-05-09 14:06:01,013] ({main} QuartzScheduler.java[initialize]:305) - Scheduler meta-data: Quartz Scheduler (v2.2.1) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.
INFO [2016-05-09 14:06:01,013] ({main} StdSchedulerFactory.java[instantiate]:1339) - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
INFO [2016-05-09 14:06:01,013] ({main} StdSchedulerFactory.java[instantiate]:1343) - Quartz scheduler version: 2.2.1
INFO [2016-05-09 14:06:01,014] ({main} QuartzScheduler.java[start]:575) - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
INFO [2016-05-09 14:06:01,884] ({main} Notebook.java[<init>]:116) - Notebook indexing started...
INFO [2016-05-09 14:06:02,293] ({main} LuceneSearch.java[addIndexDocs]:285) - Indexing 72 notebooks took 408ms
INFO [2016-05-09 14:06:02,293] ({main} Notebook.java[<init>]:118) - Notebook indexing finished: 72 indexed in 0s
INFO [2016-05-09 14:06:02,456] ({main} ServerImpl.java[initDestination]:94) - Setting the server's publish address to be /
INFO [2016-05-09 14:06:02,588] ({main} StandardDescriptorProcessor.java[visitServlet]:284) - NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
INFO [2016-05-09 14:06:03,491] ({main} AbstractConnector.java[doStart]:338) - Started SelectChannelConnector@0.0.0.0:8082
INFO [2016-05-09 14:06:03,492] ({main} ZeppelinServer.java[main]:122) - Done, zeppelin server started
INFO [2016-05-09 14:06:34,301] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:491) - Create interpreter instance spark for note shared_session
INFO [2016-05-09 14:06:34,302] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.spark.SparkInterpreter 258982848 created
INFO [2016-05-09 14:06:34,302] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.spark.PySparkInterpreter 386218062 created
INFO [2016-05-09 14:06:34,303] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.rinterpreter.RRepl 1205779245 created
INFO [2016-05-09 14:06:34,303] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.rinterpreter.KnitR 1726580549 created
INFO [2016-05-09 14:06:34,303] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.spark.SparkSqlInterpreter 392771995 created
INFO [2016-05-09 14:06:34,303] ({qtp1213818572-36} InterpreterFactory.java[createInterpretersForNote]:521) - Interpreter org.apache.zeppelin.spark.DepInterpreter 697345136 created
INFO [2016-05-09 14:06:34,311] ({pool-1-thread-2} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1461917041860_145219673 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session499376351
INFO [2016-05-09 14:06:34,312] ({pool-1-thread-2} Paragraph.java[jobRun]:226) - run paragraph 20160429-080401_1472155327 using r org.apache.zeppelin.interpreter.LazyOpenInterpreter@47debb2d
INFO [2016-05-09 14:06:34,321] ({pool-1-thread-2} RemoteInterpreterProcess.java[reference]:119) - Run interpreter process [/work/incubator-zeppelin/bin/interpreter.sh, -d, /work/incubator-zeppelin/interpreter/spark, -p, 48832, -l, /work/incubator-zeppelin/local-repo/2B894THHQ]
INFO [2016-05-09 14:06:35,516] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.rinterpreter.RRepl
INFO [2016-05-09 14:06:35,547] ({pool-1-thread-2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:416) - Push local angular object registry from ZeppelinServer to remote interpreter group 2B894THHQ
INFO [2016-05-09 14:06:35,578] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.spark.SparkInterpreter
INFO [2016-05-09 14:06:35,581] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.spark.PySparkInterpreter
INFO [2016-05-09 14:06:35,584] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.rinterpreter.KnitR
INFO [2016-05-09 14:06:35,585] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2016-05-09 14:06:35,586] ({pool-1-thread-2} RemoteInterpreter.java[init]:149) - Create remote interpreter org.apache.zeppelin.spark.DepInterpreter
