Kerberos and YARN - functions in spark-shell and spark submit local but not cluster mode

[Gerard Casey]

Hello all,

I am using Spark with Kerberos authentication.

I can run my code using `spark-shell` fine and I can also use `spark-submit` in 
local mode (e.g. —master local[16]). Both function as expected.

local mode -

        spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G 
target/scala-2.10/graphx_sp_2.10-1.0.jar

I am now progressing to run in cluster mode using YARN.

cluster mode with YARN - 

        spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster 
--executor-memory 13G --total-executor-cores 32 
target/scala-2.10/graphx_sp_2.10-1.0.jar

However, this returns:

        diagnostics: User class threw exception: 
org.apache.hadoop.security.AccessControlException: Authentication required

Before I run using spark-shell or on local mode in spark-submit I do the 
following kerberos setup:

        kinit -k -t ~/keytab -r 7d `whoami`

Clearly, this setup is not extending to the YARN setup. How do I fix the 
Kerberos issue with YARN in cluster mode? Is this something which must be in my 
/src/main/scala/graphx_sp.scala file? 

Many thanks

Geroid