Thanks Marcelo,
My understanding from a few pointers is that this may be due to insufficient
read permissions to the key tab or a corrupt key tab. I have checked the read
permissions and they are ok. I can see that it is initially configuring
correctly:
INFO security.UserGroupInformation: Login successful for user
user@login_node using keytab file /path/to/keytab
I’ve added the full trace below.
Gerry
Full trace:
Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
Spark1 will be picked by default
16/12/05 18:23:27 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/12/05 18:23:27 INFO security.UserGroupInformation: Login successful for user
me@login_nodeusing keytab file /path/to/keytab
16/12/05 18:23:27 INFO yarn.Client: Attempting to login to the Kerberos using
principal: me@login_node and keytab: /path/to/keytab
16/12/05 18:23:28 INFO impl.TimelineClientImpl: Timeline service address:
http://login_node1.xcat.cluster:8188/ws/v1/timeline/
16/12/05 18:23:28 INFO client.RMProxy: Connecting to ResourceManager at
login_node1.xcat.cluster/
16/12/05 18:23:28 INFO client.AHSProxy: Connecting to Application History
server at login_node1.xcat.cluster/
16/12/05 18:23:28 WARN shortcircuit.DomainSocketFactory: The short-circuit
local reads feature cannot be used because libhadoop cannot be loaded.
16/12/05 18:23:28 INFO yarn.Client: Requesting a new application from cluster
with 32 NodeManagers
16/12/05 18:23:28 INFO yarn.Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (15360 MB per container)
16/12/05 18:23:28 INFO yarn.Client: Will allocate AM container, with 1408 MB
memory including 384 MB overhead
16/12/05 18:23:28 INFO yarn.Client: Setting up container launch context for our
AM
16/12/05 18:23:28 INFO yarn.Client: Setting up the launch environment for our
AM container
16/12/05 18:23:28 INFO yarn.Client: Using the spark assembly jar on HDFS
because you are using HDP,
defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:28 INFO yarn.Client: Credentials file set to:
16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode:
hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_
16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1856
for me on
16/12/05 18:23:28 INFO yarn.Client: Renewal Interval set to 86400009
16/12/05 18:23:28 INFO yarn.Client: Preparing resources for our AM container
16/12/05 18:23:28 INFO yarn.YarnSparkHadoopUtil: getting token for namenode:
hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_
16/12/05 18:23:28 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 1857
for me on
16/12/05 18:23:29 INFO yarn.YarnSparkHadoopUtil: HBase class not found
java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
16/12/05 18:23:29 INFO yarn.Client: To enable the AM to login from keytab,
credentials are being copied over to the AM via the YARN Secure Distributed
Cache.
16/12/05 18:23:29 INFO yarn.Client: Uploading resource file:/path/to/keytab ->
hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/keytab
16/12/05 18:23:29 INFO yarn.Client: Using the spark assembly jar on HDFS
because you are using HDP,
defaultSparkAssembly:hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:29 INFO yarn.Client: Source and destination file systems are the
same. Not copying
hdfs://login_node1.xcat.cluster:8020/hdp/apps/2.5.0.0-1245/spark/spark-hdp-assembly.jar
16/12/05 18:23:29 INFO yarn.Client: Uploading resource
file:/home/me/Aoife/spark-abm/target/scala-2.10/graphx_sp_2.10-1.0.jar ->
hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/graphx_sp_2.10-1.0.jar
16/12/05 18:23:29 INFO yarn.Client: Uploading resource
file:/tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644/__spark_conf__6538744395325375994.zip
->
hdfs://login_node1.xcat.cluster:8020/user/me/.sparkStaging/application_1479877553404_0041/__spark_conf__6538744395325375994.zip
16/12/05 18:23:29 INFO spark.SecurityManager: Changing view acls to: me
16/12/05 18:23:29 INFO spark.SecurityManager: Changing modify acls to: me
16/12/05 18:23:29 INFO spark.SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(me); users with
modify permissions: Set(me)
16/12/05 18:23:29 INFO yarn.Client: Submitting application 41 to ResourceManager
16/12/05 18:23:30 INFO impl.YarnClientImpl: Submitted application
application_1479877553404_0041
16/12/05 18:23:31 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:31 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to
Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1480962209903
final status: UNDEFINED
tracking URL:
http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
user: me
16/12/05 18:23:32 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:33 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:34 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:35 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:36 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:37 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:38 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:39 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:40 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:40 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host:
ApplicationMaster RPC port: 0
queue: default
start time: 1480962209903
final status: UNDEFINED
tracking URL:
http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
user: me
16/12/05 18:23:41 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:42 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:43 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:44 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:45 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:46 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:47 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:48 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:49 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:50 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:51 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:52 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:53 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:54 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:23:55 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:55 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to
Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1480962209903
final status: UNDEFINED
tracking URL:
http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
user: me
16/12/05 18:23:56 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:57 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:58 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:23:59 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:00 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:01 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:02 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:03 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:04 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: ACCEPTED)
16/12/05 18:24:05 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:05 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host:
ApplicationMaster RPC port: 0
queue: default
start time: 1480962209903
final status: UNDEFINED
tracking URL:
http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
user: me
16/12/05 18:24:06 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:07 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:08 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:09 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:10 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:11 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:12 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:13 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:14 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:15 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:16 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:17 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: RUNNING)
16/12/05 18:24:18 INFO yarn.Client: Application report for
application_1479877553404_0041 (state: FINISHED)
16/12/05 18:24:18 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: User class threw exception:
org.apache.hadoop.security.AccessControlException: Authentication required
ApplicationMaster host:
ApplicationMaster RPC port: 0
queue: default
start time: 1480962209903
final status: FAILED
tracking URL:
http://login_node1.xcat.cluster:8088/proxy/application_1479877553404_0041/
user: me
Exception in thread "main" org.apache.spark.SparkException: Application
application_1479877553404_0041 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1122)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1169)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/12/05 18:24:18 INFO util.ShutdownHookManager: Shutdown hook called
16/12/05 18:24:18 INFO util.ShutdownHookManager: Deleting directory
/tmp/spark-2e566133-d50a-4904-920e-ab5cec07c644
On Mon, Dec 5, 2016 at 10:30 AM, Gerard Casey <[email protected]> wrote:
>
>> On 5 Dec 2016, at 19:26, Marcelo Vanzin <[email protected]> wrote:
>>
>> There's generally an exception in these cases, and you haven't posted
>> it, so it's hard to tell you what's wrong. The most probable cause,
>> without the extra information the exception provides, is that you're
>> using the wrong Hadoop configuration when submitting the job to YARN.
>>
>> On Mon, Dec 5, 2016 at 4:35 AM, Gerard Casey <[email protected]>
>> wrote:
>>> Hello all,
>>>
>>> I am using Spark with Kerberos authentication.
>>>
>>> I can run my code using `spark-shell` fine and I can also use `spark-submit`
>>> in local mode (e.g. —master local[16]). Both function as expected.
>>>
>>> local mode -
>>>
>>> spark-submit --class "graphx_sp" --master local[16] --driver-memory 20G
>>> target/scala-2.10/graphx_sp_2.10-1.0.jar
>>>
>>> I am now progressing to run in cluster mode using YARN.
>>>
>>> cluster mode with YARN -
>>>
>>> spark-submit --class "graphx_sp" --master yarn --deploy-mode cluster
>>> --executor-memory 13G --total-executor-cores 32
>>> target/scala-2.10/graphx_sp_2.10-1.0.jar
>>>
>>> However, this returns:
>>>
>>> diagnostics: User class threw exception:
>>> org.apache.hadoop.security.AccessControlException: Authentication required
>>>
>>> Before I run using spark-shell or on local mode in spark-submit I do the
>>> following kerberos setup:
>>>
>>> kinit -k -t ~/keytab -r 7d `whoami`
>>>
>>> Clearly, this setup is not extending to the YARN setup. How do I fix the
>>> Kerberos issue with YARN in cluster mode? Is this something which must be in
>>> my /src/main/scala/graphx_sp.scala file?
>>>
>>> Many thanks
>>>
>>> Geroid
>>
>>
>>
>> --
>> Marcelo
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
