could this be related: https://issues.apache.org/jira/browse/SPARK-8385 ?
Im not sure since im building using spark-1.2 profile.
On Thu, Jun 25, 2015 at 11:22 PM, Udit Mehta <ume...@groupon.com> wrote:
> Hi,
>
> Thanks for the reply moon. I dont see a reason to add TFS based libraries
> when I am not using them. Do you know of any instance when I need to add
> them even if I am not using Tachyon?
>
> Udit
>
> On Thu, Jun 25, 2015 at 9:13 PM, moon soo Lee <m...@apache.org> wrote:
>
>> Hi,
>>
>> If i guess based on error
>>
>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>> the TFS FileSystem implementation
>>
>> It looks like you need add libraries for the TFS FileSystem into the
>> classapath of Zeppelin's spark interpreter.
>>
>> Best,
>> moon
>>
>>
>> On Thu, Jun 25, 2015 at 3:36 PM Udit Mehta <ume...@groupon.com> wrote:
>>
>>> Hi,
>>>
>>> I am facing some issues while running zeppelin spark interpreter in the
>>> yarn client mode. I get the following error in the
>>> zeppelin-interpreter-spark*.log:
>>>
>>> ERROR [2015-06-25 22:17:56,374] ({pool-1-thread-4}
>>> ProcessFunction.java[process]:41) - Internal error processing getProgress
>>> org.apache.zeppelin.interpreter.InterpreterException:
>>> java.lang.UnsupportedOperationException: Not implemented by the TFS
>>> FileSystem implementation
>>> at
>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:75)
>>> at
>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
>>> at
>>> org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:109)
>>> at
>>> org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:299)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:938)
>>> at
>>> org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:923)
>>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>> at
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> Caused by: java.lang.UnsupportedOperationException: Not implemented by
>>> the TFS FileSystem implementation
>>> at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:216)
>>> at
>>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2564)
>>> at
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2574)
>>> at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
>>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
>>> at
>>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
>>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
>>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
>>> at
>>> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:216)
>>> at
>>> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:384)
>>> at
>>> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:102)
>>> at
>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58)
>>> at
>>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:381)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.createSparkContext(SparkInterpreter.java:276)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.getSparkContext(SparkInterpreter.java:149)
>>> at
>>> org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:398)
>>> at
>>> org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:73)
>>> ... 11 more
>>>
>>>
>>> Im running the interpreter in yarn-client mode. Here is the command i
>>> used to build the zeppelin jar:
>>>
>>> mvn clean install -DskipTests -Pspark-1.3 -Dspark.version=1.3.1
>>> -Phadoop-2.6 -Pyarn
>>>
>>> Here is my conf/interpreter.json (spark section):
>>>
>>>> "spark.yarn.access.namenodes":
>>>> "hdfs://namenode1.snc1:8032,hdfs://namenode2.snc1:8032",
>>>> "master": "yarn-client",
>>>> "zeppelin.spark.maxResult": "1000",
>>>> "zeppelin.dep.localrepo": "local-repo",
>>>> "spark.app.name": "Zeppelin-umehta",
>>>> "spark.yarn.queue": "public",
>>>> "spark.executor.memory": "512m",
>>>> "zeppelin.spark.useHiveContext": "true",
>>>> "zeppelin.spark.concurrentSQL": "false",
>>>> "spark.home": "/path/to/spark-1.3"
>>>>
>>>
>>> And heres my conf/zeppelin-env
>>>
>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>> export ZEPPELIN_PORT=10008
>>>> export ZEPPELIN_JAVA_OPTS="-Dhdp.version=2.2.0.0-2041"
>>>> export
>>>> ZEPPELIN_INTERPRETER_DIR=/home/umehta/zeppelin-0.6.0-incubating-SNAPSHOT/interpreter
>>>> export
>>>> ZEPPELIN_INTERPRETERS=org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter
>>>>
>>>>
>>>
>>> Im using hdp 2.2.0.0-2041 on my cluster.
>>>
>>> Does anyone have any insights on this?
>>>
>>> Thanks in advance,
>>> Udit
>>>
>>
>
