Hi,
Thanks for the reply moon. I dont see a reason to add TFS based libraries
when I am not using them. Do you know of any instance when I need to add
them even if I am not using Tachyon?
Udit
On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <m...@apache.org> wrote:
> Hi,
>
> If i guess based on error
>
> Caused by: java.lang.UnsupportedOperationException: Not implemented by the
> TFS FileSystem implementation
>
> It looks like you need add libraries for the TFS FileSystem into the
> classapath of Zeppelin's spark interpreter.
>
> Best,
> moon
>
>
> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <ume...@groupon.com> wrote:
>
>> Hi,
>>
>> I am facing some issues while running zeppelin spark interpreter in the
>> yarn client mode. I get the following error in the
>> zeppelin-interpreter-spark*.log:
>>
>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>> org.apache.zeppelin.interpreter.InterpreterException:
>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>> FileSystem implementation
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>> at
>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>> at
>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>> at
>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>> at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>> the TFS FileSystem implementation
>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>> at
>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>> at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>> at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>> at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>> at
>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>> at
>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>> at
>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>> at
>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>> at
>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>> ... 11 more
>>
>>
>> Im running the interpreter in yarn-client mode. Here is the command i
>> used to build the zeppelin jar:
>>
>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>> -Phadoop-2.6 -Pyarn
>>
>> Here is my conf/interpreter.json (spark section):
>>
>>> "spark.yarn.access.namenodes":
>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>> "master": "yarn-client",
>>> "zeppelin.spark.maxResult": "1000",
>>> "zeppelin.dep.localrepo": "local-repo",
>>> "spark.app.name": "Zeppelin-umehta",
>>> "spark.yarn.queue": "public",
>>> "spark.executor.memory": "512m",
>>> "zeppelin.spark.useHiveContext": "true",
>>> "zeppelin.spark.concurrentSQL": "false",
>>> "spark.home": "/path/to/spark-1.3"
>>>
>>
>> And heres my conf/zeppelin-env
>>
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>> export ZEPPELIN_PORT=10008
>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>> export
>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>> export
>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>
>>>
>>
>> Im using hdp 2.2.0.0-2041 on my cluster.
>>
>> Does anyone have any insights on this?
>>
>> Thanks in advance,
>> Udit
>>
>
