nothing obvious. I will stick to 0.5.6 build, until the latest builds
stabilize.
On Wed, Aug 31, 2016 at 1:39 AM, Jeff Zhang <zjf...@gmail.com> wrote:
> Then I guess maybe you are connecting to different database. Why not using
> 'z.show(sql("databases"))' to display the databases ? Then you will get
> a hint what's going on.
>
> On Wed, Aug 31, 2016 at 2:36 PM, Pradeep Reddy <
> pradeepreddy.a...@gmail.com> wrote:
>
>> Yes...I didn't wish to show the names of the databases that we have in
>> our data lake on that screen shot. so thats why I chose to display the
>> count. The latest zeppelin build just shows 1 count which is "default"
>> database.
>>
>> Thanks,
>> Pradeep
>>
>> On Wed, Aug 31, 2016 at 1:33 AM, Jeff Zhang <zjf...@gmail.com> wrote:
>>
>>> 116 is the databases count number. Do you expect a list of database ?
>>> then you need to use 'z.show(sql("databases"))'
>>>
>>> On Wed, Aug 31, 2016 at 2:26 PM, Pradeep Reddy <
>>> pradeepreddy.a...@gmail.com> wrote:
>>>
>>>> Here it is Jeff
>>>>
>>>> [image: Inline image 1]
>>>>
>>>> On Wed, Aug 31, 2016 at 1:24 AM, Jeff Zhang <zjf...@gmail.com> wrote:
>>>>
>>>>> Hi Pradeep,
>>>>>
>>>>> I don't see the databases on your screenshot (second one for 0.5.6). I
>>>>> think the output is correct.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 31, 2016 at 12:55 PM, Pradeep Reddy <
>>>>> pradeepreddy.a...@gmail.com> wrote:
>>>>>
>>>>>> Hi Jeff- I was able to make Kerberos work in 0.5.6 zeppelin build. It
>>>>>> seems like Kerberos not working & spark not able to talk to the shared
>>>>>> hive
>>>>>> meta store are defects in the current build.
>>>>>>
>>>>>> On Tue, Aug 30, 2016 at 11:09 PM, Pradeep Reddy <
>>>>>> pradeepreddy.a...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Jeff-
>>>>>>>
>>>>>>> I switched to local mode now, I'm able to summon the implicit
>>>>>>> objects like sc, sqlContext etc., but it doesn't show my databases &
>>>>>>> tables, just shows 1 database "default".
>>>>>>>
>>>>>>> Zeppelin Latest Build
>>>>>>>
>>>>>>> [image: Inline image 3]
>>>>>>>
>>>>>>> Zeppelin 0.5.6, running on the same machine, is able to show my
>>>>>>> databases and tables.
>>>>>>>
>>>>>>> [image: Inline image 4]
>>>>>>>
>>>>>>> On Tue, Aug 30, 2016 at 8:20 PM, Jeff Zhang <zjf...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> > the spark interpreter is not showing my tables & databases, may
>>>>>>>> be its running in an isolated mode... I'm just getting empty list, so I
>>>>>>>> attempted to do kerberos authentication to workaround that issue, and
>>>>>>>> bumped into this road block.
>>>>>>>>
>>>>>>>> kerberos would not help here, actually I think it would make the
>>>>>>>> problem more complicated. You need to first check the log why you get
>>>>>>>> empty list.
>>>>>>>>
>>>>>>>> On Wed, Aug 31, 2016 at 8:56 AM, Pradeep Reddy <
>>>>>>>> pradeepreddy.a...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Jeff- I was also successfully able to run spark shell, after
>>>>>>>>> running kdestroy, with the below command and was able to get to my
>>>>>>>>> hive
>>>>>>>>> tables.
>>>>>>>>>
>>>>>>>>> spark-shell --conf spark.yarn.keytab=$HOME/pradeep.x.alla.keytab
>>>>>>>>> --conf spark.yarn.principal=pradeep.x.alla --deploy-mode client
>>>>>>>>> --master yarn --queue <QUEUE_NAME>
>>>>>>>>>
>>>>>>>>> On Tue, Aug 30, 2016 at 7:34 PM, Pradeep Reddy <
>>>>>>>>> pradeepreddy.a...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Jeff..I have always used zeppelin in local mode, but when
>>>>>>>>>> I migrated from 0.5.6 to this version, the spark interpreter is not
>>>>>>>>>> showing
>>>>>>>>>> my tables & databases, may be its running in an isolated mode... I'm
>>>>>>>>>> just
>>>>>>>>>> getting empty list, so I attempted to do kerberos authentication to
>>>>>>>>>> workaround that issue, and bumped into this road block.
>>>>>>>>>>
>>>>>>>>>> Below is the configuration, I also tested my keytab file and its
>>>>>>>>>> working fine.
>>>>>>>>>>
>>>>>>>>>> *Kerberos test:*
>>>>>>>>>> $ kdestroy
>>>>>>>>>>
>>>>>>>>>> $ klist
>>>>>>>>>> *klist: No credentials cache found (ticket cache
>>>>>>>>>> FILE:/tmp/krb5cc_12027)*
>>>>>>>>>>
>>>>>>>>>> $ kinit -kt pradeep_x_alla.keytab -V pradeep.x.alla
>>>>>>>>>> *Using default cache: /tmp/krb5cc_12027*
>>>>>>>>>> *Using principal: pradeep.x.alla@<DOMAIN1>*
>>>>>>>>>> *Using keytab: pradeep_x_alla.keytab*
>>>>>>>>>> *Authenticated to Kerberos v5*
>>>>>>>>>>
>>>>>>>>>> $ klist
>>>>>>>>>> *Ticket cache: FILE:/tmp/krb5cc_12027*
>>>>>>>>>> *Default principal: pradeep.x.alla@<DOMAIN1>*
>>>>>>>>>>
>>>>>>>>>> *Valid starting Expires Service principal*
>>>>>>>>>> *08/30/16 20:25:19 08/31/16 06:25:19 krbtgt/<DOMAIN1>@<DOMAIN1>*
>>>>>>>>>> * renew until 08/31/16 20:25:19*
>>>>>>>>>>
>>>>>>>>>> *zeppelin-env.sh*
>>>>>>>>>>
>>>>>>>>>> export HADOOP_CONF_DIR=/etc/hadoop/conf:/etc/hive/conf
>>>>>>>>>> export SPARK_HOME=/opt/cloudera/parce
>>>>>>>>>> ls/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/spark
>>>>>>>>>> export SPARK_SUBMIT_OPTIONS="--deploy-mode client --master yarn
>>>>>>>>>> --num-executors 2 --executor-memory 2g --queue <QUEUE_NAME>"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Interpreter.json (Spark interpreter config)*
>>>>>>>>>> "2BUTFVN89": {
>>>>>>>>>> "id": "2BUTFVN89",
>>>>>>>>>> "name": "spark",
>>>>>>>>>> "group": "spark",
>>>>>>>>>> "properties": {
>>>>>>>>>> "spark.cores.max": "",
>>>>>>>>>> "zeppelin.spark.printREPLOutput": "true",
>>>>>>>>>> "master": "yarn-client",
>>>>>>>>>> "zeppelin.spark.maxResult": "1000",
>>>>>>>>>> "zeppelin.dep.localrepo": "local-repo",
>>>>>>>>>> "spark.app.name": "Zeppelin",
>>>>>>>>>> "spark.executor.memory": "",
>>>>>>>>>> "zeppelin.spark.importImplicit": "true",
>>>>>>>>>> "zeppelin.spark.sql.stacktrace": "true",
>>>>>>>>>> "zeppelin.spark.useHiveContext": "true",
>>>>>>>>>> "zeppelin.interpreter.localRepo":
>>>>>>>>>> "/home/pradeep.x.alla/zeppelin/local-repo/2BUTFVN89",
>>>>>>>>>> "zeppelin.spark.concurrentSQL": "false",
>>>>>>>>>> "args": "",
>>>>>>>>>> "zeppelin.pyspark.python": "python",
>>>>>>>>>> "spark.yarn.keytab": "/home/pradeep.x.alla/pradeep.
>>>>>>>>>> x.alla.keytab",
>>>>>>>>>> "spark.yarn.principal": "pradeep.x.alla",
>>>>>>>>>> "zeppelin.dep.additionalRemoteRepository":
>>>>>>>>>> "spark-packages,http://dl.bintray.com/spark-packages/maven,false
>>>>>>>>>> ;"
>>>>>>>>>> },
>>>>>>>>>> "status": "READY",
>>>>>>>>>> "interpreterGroup": [
>>>>>>>>>> {
>>>>>>>>>> "name": "spark",
>>>>>>>>>> "class": "org.apache.zeppelin.spark.SparkInterpreter",
>>>>>>>>>> "defaultInterpreter": true
>>>>>>>>>> },
>>>>>>>>>> {
>>>>>>>>>> "name": "sql",
>>>>>>>>>> "class": "org.apache.zeppelin.spark.Spa
>>>>>>>>>> rkSqlInterpreter",
>>>>>>>>>> "defaultInterpreter": false
>>>>>>>>>> },
>>>>>>>>>> {
>>>>>>>>>> "name": "dep",
>>>>>>>>>> "class": "org.apache.zeppelin.spark.DepInterpreter",
>>>>>>>>>> "defaultInterpreter": false
>>>>>>>>>> },
>>>>>>>>>> {
>>>>>>>>>> "name": "pyspark",
>>>>>>>>>> "class": "org.apache.zeppelin.spark.PyS
>>>>>>>>>> parkInterpreter",
>>>>>>>>>> "defaultInterpreter": false
>>>>>>>>>> }
>>>>>>>>>> ],
>>>>>>>>>> "dependencies": [],
>>>>>>>>>> "option": {
>>>>>>>>>> "remote": true,
>>>>>>>>>> "port": -1,
>>>>>>>>>> "perNoteSession": false,
>>>>>>>>>> "perNoteProcess": false,
>>>>>>>>>> "isExistingProcess": false,
>>>>>>>>>> "setPermission": false,
>>>>>>>>>> "users": []
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 30, 2016 at 6:52 PM, Jeff Zhang <zjf...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> It looks like your kerberos configuration issue. Do you mind to
>>>>>>>>>>> share your configuration ? Or you can first try to run spark-shell
>>>>>>>>>>> using
>>>>>>>>>>> spark.yarn.keytab & spark.yarn.principle to verify them.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 31, 2016 at 6:12 AM, Pradeep Reddy <
>>>>>>>>>>> pradeepreddy.a...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi- I recently built zeppelin from source and configured
>>>>>>>>>>>> kerberos authentication. For Kerberos I added "spark.yarn.keytab" &
>>>>>>>>>>>> "spark.yarn.principal" and also set master to "yarn-client". But
>>>>>>>>>>>> I keep
>>>>>>>>>>>> getting this error whenever I use spark interpreter in the notebook
>>>>>>>>>>>>
>>>>>>>>>>>> 3536728 started by scheduler org.apache.zeppelin.spark.Spar
>>>>>>>>>>>> kInterpreter335845091
>>>>>>>>>>>> ERROR [2016-08-30 17:45:37,237] ({pool-2-thread-2}
>>>>>>>>>>>> Job.java[run]:189) - Job failed
>>>>>>>>>>>> java.lang.IllegalArgumentException: Invalid rule: L
>>>>>>>>>>>> RULE:[2:$1@$0](.*@\Q<DOMAIN1>.COM\E$)s/@\Q<DOMAIN1>\E$//L
>>>>>>>>>>>> RULE:[1:$1@$0](.*@\Q<DOMAIN2>\E$)s/@\Q<DOMAIN2>\E$//L
>>>>>>>>>>>> RULE:[2:$1@$0](.*@\Q<DOMAIN2>\E$)s/@\Q<DOMAIN2>\E$//L
>>>>>>>>>>>> DEFAULT
>>>>>>>>>>>> at org.apache.hadoop.security.aut
>>>>>>>>>>>> hentication.util.KerberosName.parseRules(KerberosName.java:321)
>>>>>>>>>>>> at org.apache.hadoop.security.aut
>>>>>>>>>>>> hentication.util.KerberosName.setRules(KerberosName.java:386)
>>>>>>>>>>>> at org.apache.hadoop.security.Had
>>>>>>>>>>>> oopKerberosName.setConfiguration(HadoopKerberosName.java:75)
>>>>>>>>>>>> at org.apache.hadoop.security.Use
>>>>>>>>>>>> rGroupInformation.initialize(UserGroupInformation.java:227)
>>>>>>>>>>>> at org.apache.hadoop.security.Use
>>>>>>>>>>>> rGroupInformation.ensureInitialized(UserGroupInformation.jav
>>>>>>>>>>>> a:214)
>>>>>>>>>>>> at org.apache.hadoop.security.Use
>>>>>>>>>>>> rGroupInformation.isAuthenticationMethodEnabled(UserGroupInf
>>>>>>>>>>>> ormation.java:275)
>>>>>>>>>>>> at org.apache.hadoop.security.Use
>>>>>>>>>>>> rGroupInformation.isSecurityEnabled(UserGroupInformation.jav
>>>>>>>>>>>> a:269)
>>>>>>>>>>>> at org.apache.hadoop.security.Use
>>>>>>>>>>>> rGroupInformation.loginUserFromKeytab(UserGroupInformation.j
>>>>>>>>>>>> ava:820)
>>>>>>>>>>>> at org.apache.zeppelin.spark.Spar
>>>>>>>>>>>> kInterpreter.open(SparkInterpreter.java:539)
>>>>>>>>>>>> at org.apache.zeppelin.interprete
>>>>>>>>>>>> r.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
>>>>>>>>>>>> at org.apache.zeppelin.interprete
>>>>>>>>>>>> r.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
>>>>>>>>>>>> at org.apache.zeppelin.interprete
>>>>>>>>>>>> r.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteI
>>>>>>>>>>>> nterpreterServer.java:383)
>>>>>>>>>>>> at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
>>>>>>>>>>>> at org.apache.zeppelin.scheduler.
>>>>>>>>>>>> FIFOScheduler$1.run(FIFOScheduler.java:139)
>>>>>>>>>>>> at java.util.concurrent.Executors
>>>>>>>>>>>> $RunnableAdapter.call(Executors.java:511)
>>>>>>>>>>>> at java.util.concurrent.FutureTas
>>>>>>>>>>>> k.run(FutureTask.java:266)
>>>>>>>>>>>> at java.util.concurrent.Scheduled
>>>>>>>>>>>> ThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledT
>>>>>>>>>>>> hreadPoolExecutor.java:180)
>>>>>>>>>>>> at java.util.concurrent.Scheduled
>>>>>>>>>>>> ThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPo
>>>>>>>>>>>> olExecutor.java:293)
>>>>>>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>>>>>> at java.util.concurrent.ThreadPoo
>>>>>>>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>> INFO [2016-08-30 17:45:37,247] ({pool-2-thread-2}
>>>>>>>>>>>> SchedulerFactory.java[jobFinished]:137) - Job
>>>>>>>>>>>> remoteInterpretJob_1472593536728 finished by scheduler
>>>>>>>>>>>> org.apache.zeppelin.spark.SparkInterpreter335845091
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Pradeep
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Best Regards
>>>>>>>>>>>
>>>>>>>>>>> Jeff Zhang
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>>
>>>>>>>> Jeff Zhang
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>>
>>>>> Jeff Zhang
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
