All: I have some code that worked fine in Zeppelin 0.7.4 but I am having issues in 0.8.2 when going from spark master of local to yarn-client. Yarn client worked in 0.7.4.
When my master is set to local[*] it runs just fine. However, as soon as I switch to yarn-client I get the Cannot call methods on a stopped SparkContext error. In looking at my yarn logs everything creates fine and the job finishes without an error. The executors start just fine from what I get out of yarn logs. Any suggestions on where to look? This happens with any note that trys to run spark. If I try this very simple code: // Spark Version spark.version I get this error: java.lang.IllegalStateException: Spark context stopped while waiting for backend at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:614) at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:169) at org.apache.spark.SparkContext.<init>(SparkContext.scala:567) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:259) at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:178) at org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:89) at org.apache.zeppelin.spark.NewSparkInterpreter.open(NewSparkInterpreter.java:102) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:62) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:616) at org.apache.zeppelin.scheduler.Job.run(Job.java:188) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) What am I missing? -- ========= mailto:db...@incadencecorp.com ============ David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ============== http://www.incadencecorp.com/ ============ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org<http://www.iliterobotics.org> Board Member- USSTEM Foundation - www.usstem.org<http://www.usstem.org> The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.
