Did anyone solve that? I'm still getting it on Spark 1.5 On Sat, Jun 27, 2015 at 5:03 AM, moon soo Lee <m...@apache.org> wrote:
> That is really strange. > Could you make sure your core-site.xml does not have anything related > tachyon? > > > On Fri, Jun 26, 2015 at 4:09 PM Udit Mehta <ume...@groupon.com> wrote: > >> could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ? >> Im not sure since im building using spark-1.2 profile. >> >> On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <ume...@groupon.com> wrote: >> >>> Hi, >>> >>> Thanks for the reply moon. I dont see a reason to add TFS based >>> libraries when I am not using them. Do you know of any instance when I need >>> to add them even if I am not using Tachyon? >>> >>> Udit >>> >>> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <m...@apache.org> wrote: >>> >>>> Hi, >>>> >>>> If i guess based on error >>>> >>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by >>>> the TFS FileSystem implementation >>>> >>>> It looks like you need add libraries for the TFS FileSystem into the >>>> classapath of Zeppelin's spark interpreter. >>>> >>>> Best, >>>> moon >>>> >>>> >>>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <ume...@groupon.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am facing some issues while running zeppelin spark interpreter in >>>>> the yarn client mode. I get the following error in the >>>>> zeppelin-interpreter-spark*.log: >>>>> >>>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4} >>>>> ProcessFunction.java[process]:41) - Internal error processing getProgress >>>>> org.apache.zeppelin.interpreter.InterpreterException: >>>>> java.lang.UnsupportedOperationException: Not implemented by the TFS >>>>> FileSystem implementation >>>>> at >>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75) >>>>> at >>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68) >>>>> at >>>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109) >>>>> at >>>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299) >>>>> at >>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938) >>>>> at >>>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923) >>>>> at >>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) >>>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >>>>> at >>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:744) >>>>> >>>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by >>>>> the TFS FileSystem implementation >>>>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) >>>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) >>>>> at >>>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) >>>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) >>>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169) >>>>> at >>>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216) >>>>> at >>>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384) >>>>> at >>>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102) >>>>> at >>>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58) >>>>> at >>>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141) >>>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381) >>>>> at >>>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276) >>>>> at >>>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149) >>>>> at >>>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398) >>>>> at >>>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73) >>>>> ... 11 more >>>>> >>>>> >>>>> Im running the interpreter in yarn-client mode. Here is the command i >>>>> used to build the zeppelin jar: >>>>> >>>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1 >>>>> -Phadoop-2.6 -Pyarn >>>>> >>>>> Here is my conf/interpreter.json (spark section): >>>>> >>>>>> "spark.yarn.access.namenodes": >>>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032", >>>>>> "master": "yarn-client", >>>>>> "zeppelin.spark.maxResult": "1000", >>>>>> "zeppelin.dep.localrepo": "local-repo", >>>>>> "spark.app.name": "Zeppelin-umehta", >>>>>> "spark.yarn.queue": "public", >>>>>> "spark.executor.memory": "512m", >>>>>> "zeppelin.spark.useHiveContext": "true", >>>>>> "zeppelin.spark.concurrentSQL": "false", >>>>>> "spark.home": "/path/to/spark-1.3" >>>>>> >>>>> >>>>> And heres my conf/zeppelin-env >>>>> >>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf >>>>>> export ZEPPELIN_PORT=10008 >>>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041" >>>>>> export >>>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter >>>>>> export >>>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter >>>>>> >>>>>> >>>>> >>>>> Im using hdp 2.2.0.0-2041 on my cluster. >>>>> >>>>> Does anyone have any insights on this? >>>>> >>>>> Thanks in advance, >>>>> Udit >>>>> >>>> >>> >>
