Hyung, thx for your help. I opened these:
https://issues.apache.org/jira/browse/ZEPPELIN-1096 (this scala problem)
https://issues.apache.org/jira/browse/ZEPPELIN-1097 (similar looking python
problem)
LMK if I can provide more info or help in some way.
On Fri, Jul 1, 2016 at 5:08 AM, Hyung Sung Shim <hss...@nflabs.com> wrote:
> Hi Jonathan.
> Unfortunately I got same error in my test bed.
> Do you mind create an jira issue for this?
>
> 2016-07-01 3:19 GMT+09:00 Jonathan Esterhazy <jonathan.esterh...@gmail.com
> >:
>
>> I added this and still get the same exception. The same property is also
>> set in spark-defaults.conf.
>>
>> After that didn't work, I also tried adding --conf
>> spark.authenticate=true and --conf
>> spark.authenticate.enableSaslEncryption=true, to match the other related
>> settings in spark-defaults.conf. Still get the same classnotfoundexception.
>>
>>
>> On Thu, Jun 30, 2016 at 10:45 AM, Hyung Sung Shim <hss...@nflabs.com>
>> wrote:
>>
>>> Please add the *export SPARK_SUBMIT_OPTIONS="--conf
>>> spark.authenticate.secret=secret"* in conf/zeppelin-env.sh and restart
>>> zeppelin and retry your code.
>>>
>>>
>>> 2016-06-30 23:34 GMT+09:00 Jonathan Esterhazy <
>>> jonathan.esterh...@gmail.com>:
>>>
>>>> yes it does. i only see this problem in zeppelin.
>>>>
>>>> On Thu, Jun 30, 2016 at 7:05 AM, Hyung Sung Shim <hss...@nflabs.com>
>>>> wrote:
>>>>
>>>>> Hi Jonathan.
>>>>> It's not easy to build the test environments but I am working on this.
>>>>> I have question for you.
>>>>> Does your code working well on spark-shell in the spark.authenticate
>>>>> mode?
>>>>>
>>>>> 2016-06-30 22:47 GMT+09:00 Jonathan Esterhazy <
>>>>> jonathan.esterh...@gmail.com>:
>>>>>
>>>>>> Hyung, did you have any luck w/ zeppelin + spark authentication? I'm
>>>>>> quite stumped.
>>>>>>
>>>>>> thx.
>>>>>>
>>>>>> On Tue, Jun 28, 2016 at 9:11 PM, Hyung Sung Shim <hss...@nflabs.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you.
>>>>>>> Let me try.
>>>>>>>
>>>>>>> 2016-06-28 22:18 GMT+09:00 Jonathan Esterhazy <
>>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>>
>>>>>>>> Hyung,
>>>>>>>>
>>>>>>>> Yes, here they are.
>>>>>>>>
>>>>>>>> zeppelin-env.sh:
>>>>>>>>
>>>>>>>> export ZEPPELIN_PORT=8890
>>>>>>>> export ZEPPELIN_CONF_DIR=/etc/zeppelin/conf
>>>>>>>> export ZEPPELIN_LOG_DIR=/var/log/zeppelin
>>>>>>>> export ZEPPELIN_PID_DIR=/var/run/zeppelin
>>>>>>>> export ZEPPELIN_PID=$ZEPPELIN_PID_DIR/zeppelin.pid
>>>>>>>> export ZEPPELIN_NOTEBOOK_DIR=/var/lib/zeppelin/notebook
>>>>>>>> export ZEPPELIN_WAR_TEMPDIR=/var/run/zeppelin/webapps
>>>>>>>> export MASTER=yarn-client
>>>>>>>> export SPARK_HOME=/usr/lib/spark
>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>>>>>>>> export
>>>>>>>> CLASSPATH=":/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*
>>>>>>>> :/usr/share/aws/emr/emrfs/auxlib/*"
>>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.8.0
>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_BUCKET=mybucket
>>>>>>>> export ZEPPELIN_NOTEBOOK_S3_USER=zeppelin
>>>>>>>> export
>>>>>>>> ZEPPELIN_NOTEBOOK_STORAGE=org.apache.zeppelin.notebook.repo.S3NotebookRepo
>>>>>>>>
>>>>>>>> spark-defaults.conf:
>>>>>>>>
>>>>>>>> spark.master yarn
>>>>>>>> spark.driver.extraClassPath
>>>>>>>>
>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>> spark.driver.extraLibraryPath
>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>> spark.executor.extraClassPath
>>>>>>>>
>>>>>>>> /etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf
>>>>>>>> :/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*
>>>>>>>> spark.executor.extraLibraryPath
>>>>>>>> /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
>>>>>>>> spark.eventLog.enabled true
>>>>>>>> spark.eventLog.dir hdfs:///var/log/spark/apps
>>>>>>>> spark.history.fs.logDirectory hdfs:///var/log/spark/apps
>>>>>>>> spark.yarn.historyServer.address ip-172-30-54-30.ec2.internal:18080
>>>>>>>> spark.history.ui.port 18080
>>>>>>>> spark.shuffle.service.enabled true
>>>>>>>> spark.driver.extraJavaOptions
>>>>>>>> -Dlog4j.configuration=file:///etc/spark/conf/log4j.properties
>>>>>>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70
>>>>>>>> -XX:MaxHeapFreeRatio=70
>>>>>>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=512M
>>>>>>>> -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>> spark.dynamicAllocation.enabled true
>>>>>>>> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
>>>>>>>> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
>>>>>>>> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CM
>>>>>>>> SClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
>>>>>>>> spark.executor.memory 8640m
>>>>>>>> spark.executor.cores 7
>>>>>>>> spark.authenticate.enableSaslEncryption true
>>>>>>>> spark.driver.memory 1g
>>>>>>>> spark.network.sasl.serverAlwaysEncrypt true
>>>>>>>> spark.driver.cores 1
>>>>>>>> spark.ssl.protocol TLSv1.2
>>>>>>>> spark.ssl.keyStorePassword password
>>>>>>>> spark.yarn.maxAppAttempts 1
>>>>>>>> spark.ssl.keyStore /etc/emr/security/keystore.jks
>>>>>>>> spark.authenticate true
>>>>>>>> spark.ssl.keyPassword password
>>>>>>>> spark.ssl.enabled true
>>>>>>>> spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_CBC_SHA
>>>>>>>> spark.ssl.trustStore /etc/emr/security/truststore.jks
>>>>>>>> spark.authenticate.secret secret
>>>>>>>> spark.ssl.trustStorePassword password
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 27, 2016 at 7:33 PM, Hyung Sung Shim <hss...@nflabs.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>> Could you share your conf/zeppelin-env.sh and spark-defaults.conf ?
>>>>>>>>>
>>>>>>>>> 2016-06-28 8:52 GMT+09:00 Jonathan Esterhazy <
>>>>>>>>> jonathan.esterh...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> I am having trouble using zeppelin in a spark cluster that has
>>>>>>>>>> spark node authentication turned on (e.g. with
>>>>>>>>>> spark.authenticate=true,
>>>>>>>>>> spark.authenticate.secret=...)
>>>>>>>>>>
>>>>>>>>>> Notebook code that calls built-in spark functions (or other
>>>>>>>>>> things on executor classpath) work fine, but functions defined in the
>>>>>>>>>> notebook (anonymous or named) throw ClassNotFoundExceptions when
>>>>>>>>>> called
>>>>>>>>>> from an executor.
>>>>>>>>>>
>>>>>>>>>> For example, this code works:
>>>>>>>>>>
>>>>>>>>>> val rdd = sc.textFile("hdfs://my-text-file")
>>>>>>>>>> rdd.take(1).foreach(println)
>>>>>>>>>>
>>>>>>>>>> rdd.saveAsTextFile("hdfs:///my-other-text-file")
>>>>>>>>>>
>>>>>>>>>> but code like this...
>>>>>>>>>>
>>>>>>>>>> rdd.filter(_.contains("my data"))
>>>>>>>>>>
>>>>>>>>>> fails with
>>>>>>>>>>
>>>>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>>>>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:84)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I get the same kind of error if the filter function is defined as
>>>>>>>>>> a named function in the notebook, or as a member of singleton object
>>>>>>>>>> defined in the notebook.
>>>>>>>>>>
>>>>>>>>>> When I look at the executor's log output, I see this error:
>>>>>>>>>>
>>>>>>>>>> 16/06/27 21:36:23 ERROR repl.ExecutorClassLoader: Failed to check
>>>>>>>>>> existence of class
>>>>>>>>>> $line31.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1 on
>>>>>>>>>> REPL
>>>>>>>>>> class server at https://172.30.54.30:34980
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.getClassFileInputStreamFromHttpServer(ExecutorClassLoader.scala:113)
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:146)
>>>>>>>>>> at
>>>>>>>>>> org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:76)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>>>>> at java.lang.Class.forName(Class.java:348)
>>>>>>>>>>
>>>>>>>>>> ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If I disable spark authentication, everything works as expected.
>>>>>>>>>> I am running zeppelin 0.5.6 on spark 1.6.1 with yarn.
>>>>>>>>>>
>>>>>>>>>> Has anyone been able to get zeppelin working with spark
>>>>>>>>>> authentication?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
