Re: submitting spark job with kerberized Hadoop issue

Any update on this? On Tuesday, 16 August 2016, Aneela Saleem wrote: > Thanks Steve, > > I went through this but still not able to fix the issue > > On Mon, Aug 15, 2016 at 2:01 AM, Steve Loughran > wrote: > >> Hi, >> >> Just came across this while going through all emails I'd left unread over

Re: submitting spark job with kerberized Hadoop issue

Thanks Steve, I went through this but still not able to fix the issue On Mon, Aug 15, 2016 at 2:01 AM, Steve Loughran wrote: > Hi, > > Just came across this while going through all emails I'd left unread over > my vacation. > > did you manage to fix this? > > 1. There's some notes I've taken on

Re: submitting spark job with kerberized Hadoop issue

Thanks Saisai and Ted, I have already configured HBase security and it's working fine. I have also done kinit before submitting job. Following is the code i'm trying to use System.setProperty("java.security.krb5.conf", "/etc/krb5.conf"); System.setProperty("java.security.auth.login.config", "/

Re: submitting spark job with kerberized Hadoop issue

The link in Jerry's response was quite old. Please see: http://hbase.apache.org/book.html#security Thanks On Sun, Aug 7, 2016 at 6:55 PM, Saisai Shao wrote: > 1. Standalone mode doesn't support accessing kerberized Hadoop, simply > because it lacks the mechanism to distribute delegation tokens

Re: submitting spark job with kerberized Hadoop issue

1. Standalone mode doesn't support accessing kerberized Hadoop, simply because it lacks the mechanism to distribute delegation tokens via cluster manager. 2. For the HBase token fetching failure, I think you have to do kinit to generate tgt before start spark application ( http://hbase.apache.org/0

Re: submitting spark job with kerberized Hadoop issue

Thanks Wojciech and Jacek! I tried with Spark on Yarn with kerberized cluster it works fine now. But now when i try to access Hbase through spark i get the following error: 2016-08-07 20:43:57,617 WARN [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: Exception encountered w

Re: submitting spark job with kerberized Hadoop issue

What I can say, is that we successfully use spark on yarn with kerberized cluster. One of my coworkers also tried using it in the same way as you are(spark standalone with kerberized cluster) but as far as I remember, he didn't succeed. I may be wrong, because I was not personally involved in this

Re: submitting spark job with kerberized Hadoop issue

Hi Aneela, I don't really know. I've never been using (or even toying with) Spark Standalone to access a secured HDFS cluster. I however think the settings won't work since they are for Spark on YARN (I would not be surprised to know that it is not supported outside Spark on YARN). Pozdrawiam, Ja

Re: submitting spark job with kerberized Hadoop issue

Hi Jacek! Thanks for your response. I am using spark standalone. I have secured hadoop cluster, Can you please guide me wha to do if i want to access hadoop in my spark job? Thanks On Sat, Aug 6, 2016 at 12:34 AM, Jacek Laskowski wrote: > Just to make things clear...are you using Spark Standa

Re: submitting spark job with kerberized Hadoop issue

Just to make things clear...are you using Spark Standalone and Spark on YARN-specific settings? I don't think it's gonna work. Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jacek

submitting spark job with kerberized Hadoop issue

Hi all, I'm trying to connect to Kerberized Hadoop cluster using spark job. I have kinit'd from command line. When i run the following job i.e., *./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal spark/hadoop-master@platalyticsrealm --class com.platalytics.example.spark.App --