> the spark interpreter is not showing my tables & databases, may be its
running in an isolated mode... I'm just getting empty list, so I attempted
to do kerberos authentication to workaround that issue, and bumped into
this road block.
kerberos would not help here, actually I think it would make the problem
more complicated. You need to first check the log why you get empty list.
On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <pradeepreddy.a...@gmail.com>
wrote:
> Jeff- I was also successfully able to run spark shell, after running
> kdestroy, with the below command and was able to get to my hive tables.
>
> spark-shell --conf spark.yarn.keytab=$HOME/pradeep.x.alla.keytab --conf
> spark.yarn.principal=pradeep.x.alla --deploy-mode client --master yarn
> --queue <QUEUE_NAME>
>
> On Tue, Aug 30, 2016 at 7:34 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Thanks Jeff..I have always used zeppelin in local mode, but when I
>> migrated from 0.5.6 to this version, the spark interpreter is not showing
>> my tables & databases, may be its running in an isolated mode... I'm just
>> getting empty list, so I attempted to do kerberos authentication to
>> workaround that issue, and bumped into this road block.
>>
>> Below is the configuration, I also tested my keytab file and its working
>> fine.
>>
>> *Kerberos test:*
>> $ kdestroy
>>
>> $ klist
>> *klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_12027)*
>>
>> $ kinit -kt pradeep_x_alla.keytab -V pradeep.x.alla
>> *Using default cache: /tmp/krb5cc_12027*
>> *Using principal: pradeep.x.alla@<DOMAIN1>*
>> *Using keytab: pradeep_x_alla.keytab*
>> *Authenticated to Kerberos v5*
>>
>> $ klist
>> *Ticket cache: FILE:/tmp/krb5cc_12027*
>> *Default principal: pradeep.x.alla@<DOMAIN1>*
>>
>> *Valid starting Expires Service principal*
>> *08/30/16 20:25:19 08/31/16 06:25:19 krbtgt/<DOMAIN1>@<DOMAIN1>*
>> * renew until 08/31/16 20:25:19*
>>
>> *zeppelin-env.sh*
>>
>> export HADOOP_CONF_DIR=/etc/hadoop/conf:/etc/hive/conf
>> export SPARK_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/
>> lib/spark
>> export SPARK_SUBMIT_OPTIONS="--deploy-mode client --master yarn
>> --num-executors 2 --executor-memory 2g --queue <QUEUE_NAME>"
>>
>>
>> *Interpreter.json (Spark interpreter config)*
>> "2BUTFVN89": {
>> "id": "2BUTFVN89",
>> "name": "spark",
>> "group": "spark",
>> "properties": {
>> "spark.cores.max": "",
>> "zeppelin.spark.printREPLOutput": "true",
>> "master": "yarn-client",
>> "zeppelin.spark.maxResult": "1000",
>> "zeppelin.dep.localrepo": "local-repo",
>> "spark.app.name": "Zeppelin",
>> "spark.executor.memory": "",
>> "zeppelin.spark.importImplicit": "true",
>> "zeppelin.spark.sql.stacktrace": "true",
>> "zeppelin.spark.useHiveContext": "true",
>> "zeppelin.interpreter.localRepo": "/home/pradeep.x.alla/zeppelin
>> /local-repo/2BUTFVN89",
>> "zeppelin.spark.concurrentSQL": "false",
>> "args": "",
>> "zeppelin.pyspark.python": "python",
>> "spark.yarn.keytab": "/home/pradeep.x.alla/pradeep.
>> x.alla.keytab",
>> "spark.yarn.principal": "pradeep.x.alla",
>> "zeppelin.dep.additionalRemoteRepository": "spark-packages,
>> http://dl.bintray.com/spark-packages/maven,false;";
>> },
>> "status": "READY",
>> "interpreterGroup": [
>> {
>> "name": "spark",
>> "class": "org.apache.zeppelin.spark.SparkInterpreter",
>> "defaultInterpreter": true
>> },
>> {
>> "name": "sql",
>> "class": "org.apache.zeppelin.spark.SparkSqlInterpreter",
>> "defaultInterpreter": false
>> },
>> {
>> "name": "dep",
>> "class": "org.apache.zeppelin.spark.DepInterpreter",
>> "defaultInterpreter": false
>> },
>> {
>> "name": "pyspark",
>> "class": "org.apache.zeppelin.spark.PySparkInterpreter",
>> "defaultInterpreter": false
>> }
>> ],
>> "dependencies": [],
>> "option": {
>> "remote": true,
>> "port": -1,
>> "perNoteSession": false,
>> "perNoteProcess": false,
>> "isExistingProcess": false,
>> "setPermission": false,
>> "users": []
>> }
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 6:52 PM, Jeff Zhang <zjf...@gmail.com> wrote:
>>
>>> It looks like your kerberos configuration issue. Do you mind to share
>>> your configuration ? Or you can first try to run spark-shell using
>>> spark.yarn.keytab & spark.yarn.principle to verify them.
>>>
>>> On Wed, Aug 31, 2016 at 6:12 AM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
>>>> Hi- I recently built zeppelin from source and configured kerberos
>>>> authentication. For Kerberos I added "spark.yarn.keytab" &
>>>> "spark.yarn.principal" and also set master to "yarn-client". But I keep
>>>> getting this error whenever I use spark interpreter in the notebook
>>>>
>>>> 3536728 started by scheduler org.apache.zeppelin.spark.Spar
>>>> kInterpreter335845091
>>>> ERROR [2016-08-30 17:45:37,237] ({pool-2-thread-2} Job.java[run]:189) -
>>>> Job failed
>>>> java.lang.IllegalArgumentException: Invalid rule: L
>>>> RULE:[2:$1@$0](.*@\Q<DOMAIN1>.COM\E$)s/@\Q<DOMAIN1>\E$//L
>>>> RULE:[1:$1@$0](.*@\Q<DOMAIN2>\E$)s/@\Q<DOMAIN2>\E$//L
>>>> RULE:[2:$1@$0](.*@\Q<DOMAIN2>\E$)s/@\Q<DOMAIN2>\E$//L
>>>> DEFAULT
>>>> at org.apache.hadoop.security.authentication.util.KerberosName.
>>>> parseRules(KerberosName.java:321)
>>>> at org.apache.hadoop.security.authentication.util.KerberosName.
>>>> setRules(KerberosName.java:386)
>>>> at org.apache.hadoop.security.HadoopKerberosName.setConfigurati
>>>> on(HadoopKerberosName.java:75)
>>>> at org.apache.hadoop.security.UserGroupInformation.initialize(U
>>>> serGroupInformation.java:227)
>>>> at org.apache.hadoop.security.UserGroupInformation.ensureInitia
>>>> lized(UserGroupInformation.java:214)
>>>> at org.apache.hadoop.security.UserGroupInformation.isAuthentica
>>>> tionMethodEnabled(UserGroupInformation.java:275)
>>>> at org.apache.hadoop.security.UserGroupInformation.isSecurityEn
>>>> abled(UserGroupInformation.java:269)
>>>> at org.apache.hadoop.security.UserGroupInformation.loginUserFro
>>>> mKeytab(UserGroupInformation.java:820)
>>>> at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpr
>>>> eter.java:539)
>>>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(Laz
>>>> yOpenInterpreter.java:69)
>>>> at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpre
>>>> t(LazyOpenInterpreter.java:93)
>>>> at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServ
>>>> er$InterpretJob.jobRun(RemoteInterpreterServer.java:383)
>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>>>> at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOSchedu
>>>> ler.java:139)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor
>>>> s.java:511)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1142)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> INFO [2016-08-30 17:45:37,247] ({pool-2-thread-2}
>>>> SchedulerFactory.java[jobFinished]:137) - Job
>>>> remoteInterpretJob_1472593536728 finished by scheduler
>>>> org.apache.zeppelin.spark.SparkInterpreter335845091
>>>>
>>>> Thanks,
>>>> Pradeep
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>
--
Best Regards
Jeff Zhang
